diff --git a/slides/evaluation.html b/slides/evaluation.html
index 8e86e5dfa38d1e9a0e6c3c13d4529e9e1642f403..21dc51a2a024fb7a1b4f7c599ed036397b432f7a 100644
--- a/slides/evaluation.html
+++ b/slides/evaluation.html
@@ -1,13 +1,13 @@
---
layout: presentation
-title: Evaluation --Week 4--
-description: Accessibility Need finding and Evaluation
+title: Evaluation --Week 5--
+description: Accessibility Needfinding and Evaluation
class: middle, center, inverse
---
background-image: url(img/people.png)
.left-column50[
-# Week 4: Accessible Evaluation
+# Week 5: Accessible Evaluation
{{site.classnum}}, {{site.quarter}}
]
@@ -28,7 +28,7 @@ class:
[//]: # (Outline Slide)
# Learning Goals for Today
-- Presenting Accessibly
+- Presenting Acessibly
- How to build inclusive experiment designs
- Running an inclusive summative study
@@ -38,7 +38,7 @@ class:
So you have an app and you think it's accessible. How do you check that?
-.quote[...getting the big picture and assessing the overall experience of a finished product...] - Nielsen Norman Group
+.quote[Unlike formative evaluations, whose goals is to inform the design process, summative evaluations involve getting the big picture and assessing the overall experience of a finished product...] - Nielsen Norman Group
Lots of variables here, but
- typically checks that an app works as expected on standard tasks when used by the target population
@@ -47,18 +47,6 @@ Lots of variables here, but
---
-# Why not Start with Formative User Testing?
-
-- Many of the apps we deploy are designed for people in general
-- Disabled people need to be able to use those apps too
-- Summative testing is the gold standard in assessing accessibility
-
-Of course the entire design process *should not just include majority class people* and much of this presentation applies to formative testing took
-
-We'll also cover that later this week when we talk about designing apps *for* and *with* people with disabilities
-
----
-
# *Accessible* Summative User Testing
- Accessibility doesn't come by accident when planning studies
@@ -71,48 +59,7 @@ Best guidance: [Nielsen-Norman Group](http://www.nngroup.com/reports/accessibili
---
# Accessible Study Planning Workflow
-
-<div class="mermaid">
-%%{init: {'theme':'base', 'themeVariables': { 'primaryColor': '#4CAF50', 'tertiaryColor': '#009688', 'fontSize': '16px', 'textMargin': '0px', 'text-align': 'left' }}%%
-
-flowchart LR
-
- A(Identify stakeholders)
- B(Identify tasks)
-
- A <--> C
- B <--> C
- C <--> D
- D <--> E
-
- subgraph Assign tasks
- C(Access needs
- Effects of familiarity
- Personal preferences
- Institutional constraints)
- end
-
- subgraph Plan Accomm.
- D(Communication
- Materials
- Time
- Space)
- end
-
- subgraph Reflect
- E(Access Synergies
- Access conflicts
- Power Dynamics)
- end
-
-
-classDef default fill:#009688,stroke:#333,stroke-width:2px, color:white;
-classDef reflect fill:#4CAF50,stroke:#333,stroke-width:2px, color:white;
-class A,B,C,D default
-class A,B,C,D,E reflect
-linkStyle default stroke: black,stroke-width:8px
-
-</div>
+
---
@@ -125,55 +72,59 @@ Supposed you are hired to perform usability testing of new food-delivery app wit
# Identify Stakeholders
-.left-column[you are hired to perform usability testing of new food-delivery app with people with sensory disabilities.]
+Supposed you are hired to perform usability testing of new food-delivery app with people with sensory disabilities.
+- Who are the stakeholders who you need to take into account for this study?
+
+---
-.right-column[
-Who are the stakeholders who you need to take into account for this study?
+# Identify Stakeholders
- - Disabled participants (BVI, DHH)
+Supposed you are hired to perform usability testing of new food-delivery app with people with sensory disabilities.
+- Who are the stakeholders who you need to take into account for this study?
+
+ - Disabled participants
+ - Blind people
+ - Deaf people
+ - Hard-of-hearing people
- Research team
- - Lua -- lead researcher, has dyslexia
- - Jay -- project team member, Deaf person who uses sign language.
- - El -- project team member, non-disabled
-]
+ - Lua -- lead researcher, has dyslexia
+ - Jay -- project team member, Deaf person who uses sign language.
+ - El -- project team member, non-disabled
+
---
# Design Study
-.left-column[you are hired to perform usability testing of new food-delivery app with people with sensory disabilities.]
-.right-column[
+Supposed you are hired to perform usability testing of new food-delivery app with people with sensory disabilities.
- Who are the stakeholders who you need to take into account for this study?
- What is the right structure for the study?
-]
+
---
-# Design Study (1 of 5)
+# Design Study (1 of 4)
This is a usability study, so it should include the same tasks as tested for users without disabilities
Metrics should be similar as well -- for example the [System Usability Scale](https://www.usability.gov/how-to-and-tools/methods/system-usability-scale.html) might be useful at the end
---
-# Design Study (2 of 5)
+# Design Study (2 of 4)
This is also a study of accessibility, with disabled users. The study design needs to consider
-
+.left-column50[
- Choice of method
- e.g. think aloud may be more difficult for a nonspeaking individual, or someone with fatigue
- Check in on study measures and statistical design. Does user heterogeneity impact this?
<!-- - Study venue (It may be difficult to duplicate participant hardware/software) -->
-
----
-# Design Study (3 of 5)
-
-This is also a study of accessibility, with disabled users. The study design needs to consider
-
+]
+.right-column50[
- Ensure that participants' hardware/software also addresses study needs (e.g. do participants have a working monitor?)
- Study length (estimate before and after piloting)
- - Study length may increase for some disabled participants
+ - Study length may increase for some disabled participnats
- Study length may negatively impact participants with fatigue-related concerns
+]
---
-# Design Study (4 of 5)
+# Design Study (3 of 4)
Study metrics may also need revision.
.left-column50[
@@ -184,25 +135,25 @@ Consider adding
]
.left-column50[
Consider changing
-- Any metrics that include ableist assumptions
+- Any metrics that include abelist assumptions
- Approach to time based measurements
]
---
-# Design Study (5 of 5)
+# Design Study (4 of 4)
It is also important to consider ethical issues
-
+.left-column60[
- What responsibility do you have to participants in terms of providing skills and help in & out of study?
-- What responsibility do you have to give participants a participatory role in design?
- How do you ensure the integrity of your record of their experience?
- Are you compensating them appropriately and addressing costly transportation needs?
-
-???
+]
+.right-column40[
These changes may in turn impact
- Expense
- Scalability
+]
---
@@ -218,17 +169,20 @@ Supposed you are hired to perform usability testing of new food-delivery app wit
# Tasks List
To perform standard usability testing on the app based on disabilities represented in sample
-1. Assign Tasks
-2. Train study team
-3. Handle recruitment
-4. Pilot with people with disabilities
-5. Run the Study
-6. Analyze the data
+1. Train study team
+2. Handle recruitment
+3. Pilot with people with disabilities
+4. Run the Study
+ - Prepare study materials (e.g., print out protocols)
+ - Hire access support staff
+ - Participate in the study
+ - Compensate participants
+5. Analyze the data
---
---
-# 1. Assign Tasks
+# Assign Tasks
- Consider:
- access needs
- effects of familiarity with the task
@@ -242,20 +196,19 @@ To perform standard usability testing on the app based on disabilities represent
---
# Task Assignment List
-1. ~~Assign Tasks~~
-2. **Lua**: Train study team
-3. **Jay**: Handle recruitment
-4. **Jay**: Pilot with people with disabilities
-5. **Lua**: Run the Study
+1. **Lua**: Train study team
+2. **Jay**: Handle recruitment
+3. **Jay**: Pilot with people with disabilities
+4. **Lua**: Run the Study
- **Lua**: Prepare study materials (e.g., print out protocols)
- **Jay**: Hire access support staff
- **Participants**: Participate in the study
- **El**: Compensate participants
-6. **All**: Analyze the data
+5. **All**: Analyze the data
---
-# 2. Train study team (1 of 2)
+# 1. Train study team (1 of 2)
As study lead, and a person with both disability and study related experience, Lua will train the study team.
@@ -263,7 +216,7 @@ Her goal is to ensure that El and Jay are aware of best practices and considerat
---
-# 2. Train study team (2 of 2)
+# 1. Train study team (2 of 2)
Some things they might discuss include
- Ensuring that they address access needs and communication support from recruitment onward
@@ -274,19 +227,11 @@ Some things they might discuss include
- When to help and when to end a task if participants have trouble
---
-# 3. Handle Recruitment (1/2)
-As a person already connected to the disability community, the team determines that Jay will be most likely to be trusted, and to identify opportunities to build trust throughout the recruitment process.
+# 2. Handle Recruitment
+As a person already connected to the disability community, the team determins that Jay will be most likely to be trusted, and to identify opportunities to build trust throughout the recruitment process
-Recruitment raises a number of access issues
- Are your recruitment materials accessible
-- How will you find participants (next slide)
-- How will you compensate participants?
-
----
-
-# 3. Handle Recruitment (2/2)
-
-Finding participants may require careful work
+- How will you find participants?
- How much do you know about the group your are targeting and what to expect from/of them?
- Will they see you as trustworthy?
- Are you giving them enough agency in the study process?
@@ -295,7 +240,7 @@ Finding participants may require careful work
- Participants may see researchers as “just another professional, conducting professional surveillanceâ€
---
-# 4. Pilot study
+# 3. Pilot study
First, revisit accommodations plan
@@ -312,7 +257,6 @@ Prepare:
- Hire interpreters (at least 2 weeks in advance)
- Print out accessible study materials, consent form, etc
- Set up the room
-- Have compensation ready
]
---
@@ -321,7 +265,7 @@ Prepare:
- Lua gets tired from reading overtime
- Jay knows ASL and Deaf norms
- Study length was problematic for one Blind participant who also had a chronic illness
-- Sometimes sight lines made it hard for Jay to see the interpreter
+- Sometimes sightlines made it hard for Jay to see the interpreter
---
@@ -348,29 +292,16 @@ Prepare:
---
-# 5. Run the Study
+# 4. Run the Study
-At this point the team should be well prepared. However, it is still important to monitor both the team's and the participants' needs and adjust as accessibility concerns arise.
-
-Similar to piloting a study
+At this point the team should be well prepared. However, it is still important to monitor both the team's and the participants' needs and adjust as accessibility concerns arise.
---
-# 6. Analyze the Data
+# 5. Analyze the Data
This is another area where access concerns may come up.
- Data analysis tools may not be accessible
- Team members may have different needs with respect to written versus audio transcripts
- <q>Member Checking</q>
----
-# (If time) Try it
-
-You read about Alexa's use by people with disabilities in class. Form small groups
-- Identify potential tasks to test
-- Write down a sampling goal (who is included)
-- Write down a list of accessibility assessment goals (metrics)
-- Identify potential accessibility concerns
-
-Post on [Ed](https://edstem.org/us/courses/31170/discussion/2427390)
-