- Including people with disabilities in data collection in general
- Running an inclusive summative study
- Plain Language Translation
---
# Summative User Testing
So you have an app and you think it's accessible. How do you check that?
...
...
@@ -37,9 +41,18 @@ So you have an app and you think it's accessible. How do you check that?
.quote[Summative evaluations describe how well a design performs, often compared to a benchmark such as a prior version of the design or a competitor. Unlike formative evaluations, whose goals is to inform the design process, summative evaluations involve getting the big picture and assessing the overall experience of a finished product. Summative evaluations occur less frequently than formative evaluations, usually right before or right after a redesign.] - Nielsen Norman Group
---
# Summative User Testing
Why are we talking about them first?
# What is Summative User Testing
Lots of variables here, but
- typically checks that an app works as expected on standard tasks when used by the target population
- may compare features or apps quantitatively
- may involve think aloud or other qualitative data
---
# Why not Start with Formative User Testing?
- Many of the apps we deploy are designed for people in general
- Disabled people need to be able to use those apps too
- Summative testing is the gold standard in assessing accessibility
...
...
@@ -49,53 +62,37 @@ Of course the entire design process *should not just include majority class peop
We'll also cover that later today when we talk about designing apps *for* and *with* people with disabilities
---
# Summative User Testing
Best Practices
- Eliminate obvious accessibility & usability problems ahead of time
- Consider a remote study, or go to their home (familiar spaces & tech)
- Don’t be overly sensitive (don’t overcompensate)
- Don’t rely on useless cues (audio/visual) to convey encouragement
- Monitor participant fatigue carefully
---
# *Accessible* Summative User Testing
- Accessibility doesn't come by accident when planning studies
- All research should be accessible research (regardless of if it is accessibility research)
- You have to make your system accessible (using inspection techniques) *before* doing this
- We will discuss accessibility *for evaluators* and *for participants* today
---
#Lab Testing (often must take place in the home).
Best guidance: [Nielsen-Norman Group](http://www.nngroup.com/reports/accessibility/testing/); [Anticipate and Adjust](https://a11ykelly.medium.com/anticipate-and-adjust-cultivating-access-in-human-centered-methods-1e46c6845e34)
Little details matter
- How will you find participants?
- How much do you know about the group your are targeting and what to expect from/of them?
- How will you communicate?
- Can you duplicate the custom hardware/software they use, or must you go to them?
- Do they have hardware you need (e.g. working monitor?)
Anticipate and Adjust presents a workflow to help you in answering these questions.
---
# Accessible Study Planning Workflow
---
# An Example
- Supposed you are hired to perform usability testing of new food-delivery app with people with sensory disabilities.
Supposed you are hired to perform usability testing of new food-delivery app with people with sensory disabilities.
---
# Identify Stakeholders
- Supposed you are hired to perform usability testing of new food-delivery app with people with sensory disabilities.
Supposed you are hired to perform usability testing of new food-delivery app with people with sensory disabilities.
- Who are the stakeholders who you need to take into account for this study?
---
# Stakeholders List
- Disabled participants
- Blind people
...
...
@@ -107,125 +104,229 @@ Anticipate and Adjust presents a workflow to help you in answering these questio
- El -- project team member, non-disabled
---
# Identify Tasks
- Supposed you are hired to perform usability testing of new food-delivery app with people with sensory disabilities.
- What tasks should this team plan for?
# Design Study
Supposed you are hired to perform usability testing of new food-delivery app with people with sensory disabilities.
- Who are the stakeholders who you need to take into account for this study?
- What is the right structure for the study?
---
# Design Study (1 of 4)
This is a usability study, so it should include the same tasks as tested for users without disabilities
Metrics should be similar as well -- for example the [System Usability Scale](https://www.usability.gov/how-to-and-tools/methods/system-usability-scale.html) might be useful at the end
---
# Design Study (2 of 4)
This is also a study of accessibility, with disabled users. The study design needs to consider
.left-column50[
- Choice of method
- e.g. think aloud may be more difficult for a nonspeaking individual, or someone with fatigue
- Check in on study measures and statistical design. Does user heterogeneity impact this?
- Study venue (It may be difficult to duplicate participant hardware/software)
]
.right-column50[
- Ensure that participants' hardware/software also addresses study needs (e.g. do participants have a working monitor?)
- Study length (estimate before and after piloting)
- Study length may increase for some disabled participnats
- Study length may negatively impact participants with fatigue-related concerns
]
---
# Design Study (3 of 4)
Study metrics may also need revision.
.left-column50[
Consider adding
- Click errors
- Success criteria & ratings
- Accessibility errors
]
.left-column50[
Consider changing
- Any metrics that include abelist assumptions
- Approach to time based measurements
]
---
# Design Study (4 of 4)
It is also important to consider ethical issues
.left-column60[
- What responsibility do you have to participants in terms of providing skills and help in & out of study?
- What responsibility do you have to give participants a participatory role in design?
- How do you ensure the integrity of your record of their experience?
- Are you compensating them appropriately and addressing costly transportation needs?
]
.right-column40[
These changes may in turn impact
- Expense
- Scalability
]
---
# Tasks List
- Perform standard usability testing on the app based on disabilities represented in sample
- Pilot with people with disabilities
- Book the space
- Prepare interview materials (e.g., print out protocols)
- Hire access support staff
- Perform the interviews
- Participate in interviews
- Compensate participants
# Plan Study Tasks
Supposed you are hired to perform usability testing of new food-delivery app with people with sensory disabilities.
- Who are the stakeholders who you need to take into account for this study?
- What is the right structure for the study?
- What study tasks should this team prepare for?
---
# Tasks List
To perform standard usability testing on the app based on disabilities represented in sample
0. Assign Tasks
1. Train study team
2. Handle recruitment
3. Pilot with people with disabilities
4. Run the Study
- Prepare study materials (e.g., print out protocols)
- Hire access support staff
- Participate in the study
- Compensate participants
5. Analyze the data
---
---
# Assign Tasks
# 0. Assign Tasks
- Consider:
- access needs
- effects of familiarity with the task
- personal preferences
- other institutional constraints
- For this scenario:
- Lua has significant experience interviewing; Lua needs interview materials in large print with a specific font
- El wants to gain more experience interviewing
- Lua has significant experience running studies; Lua needs a study script in large print with a specific font
- El wants to gain more experience running studies
- Jay has significant experience with ASL and Deaf Space; Jay prefers to have interpreters
---
# Task Assignment List
- All: Perform standard usability testing on the app based on disabilities represented in sample
- Lua and El: Pilot with people with disabilities
- Jay: Book the space
- Lua: Prepare interview materials (e.g., print out protocols)
- Jay: Hire access support staff
- Lua: Perform the interviews
- Participants: Participate in interviews
- El: Compensate participants
1. **Lua**: Train study team
2. **Jay**: Handle recruitment
3. **Jay**: Pilot with people with disabilities
4. **Lua**: Run the Study
- **Lua**: Prepare study materials (e.g., print out protocols)
- **Jay**: Hire access support staff
- **Participants**: Participate in the study
- **El**: Compensate participants
5. **All**: Analyze the data
---
# Plan Accommodations
- Consider:
- Communication
- Materials
- Time
- Space
# 1. Train study team (1 of 2)
As study lead, and a person with both disability and study related experience, Lua will train the study team.
Her goal is to ensure that El and Jay are aware of best practices and considerations that might impact study design before they begin interacting with participants.
---
# Accommodations List
# 1. Train study team (2 of 2)
Some things they might discuss include
- Ensuring that they address access needs and communication support from recruitment onward
- Providing basic DEIA training as needed so that all those with participant contact know basics.
- Don’t be overly sensitive (don’t overcompensate)
- Don’t rely on useless cues (audio/visual) to convey encouragement
- Monitor participant fatigue carefully
- When to help and when to end a task if participants have trouble
---
# 2. Handle Recruitment
As a person already connected to the disability community, the team determins that Jay will be most likely to be trusted, and to identify opportunities to build trust throughout the recruitment process
Lots of details here
- Are your recruitment materials accessible
- How will you find participants?
- How much do you know about the group your are targeting and what to expect from/of them?
- Will they see you as trustworthy?
- Are you giving them enough agency in the study process?
---
# 3. Pilot study
First, plan accommodations
.left-column50[
Consider:
- Communication
- Materials
- Time
- Space
]
--
.right-column50[
Prepare:
- Hire interpreters (at least 2 weeks in advance)
- Print out accessible interview materials
- Print out accessible study materials, consent form, etc
- Set up the room
]
---
# Reflect
- Lua gets tired from reading overtime
- Jay knows ASL and Deaf norms
- Study length was problematic for one Blind participant who also had a chronic illness
- Sometimes sightlines made it hard for Jay to see the interpreter
---
# Update Based on Reflection
- All: Perform standard usability testing on the app based on disabilities represented in sample
- Lua and El: Pilot with people with disabilities
- Jay: Book the space
- Lua: Prepare interview materials (e.g., print out protocols)
- Jay: Hire access support staff
- *All*: Perform the interviews
- Participants: Participate in interviews
- El: Compensate participants
# Update Study Design Based on Reflection
- Institute a protocol for monitoring fatigue
- Add a cutoff for tasks that take a long time
- Rearrange the room to better support communication with interpreters
- ...
---
# Consider Accessibility in These Common Tasks
- Picking a method
- Recruiting
- Have a conversation with participants about access needs
- Transportation
- Preparing the space
- Obtaining consent
- Running the Study
- Writing up results from the study
- Member checking
# Update Tasks List Based on Reflection
1. ~~Lua: Train study team~~ (done)
2. **Jay**: Handle recruitment (ongoing)
3. ~~Jay: Pilot with people with disabilities~~ (done)
4. **Jay & El**: Run the Study
- **El**: Prepare study materials (e.g., print out protocols)
- **Jay**: Hire access support staff
- **Participants**: Participate in the study
- **El**: Compensate participants
5. **All**: Analyze the data
---
# Issues
Heterogeneous Users
Accountability
- What responsibility do you have to participants in terms of providing skills and help in & out of study?
- What responsibility do you have to give participants a participatory role in design?
- How do you ensure the integrity of your record of their experience?
Expense
Scalability
# 4. Run the Study
Hard to do with early prototypes
At this point the team should be well prepared. However, it is still important to monitor both the team's and the participants' needs and adjust as accessibility concerns arise.
---
# Possible Metrics
Click errors
Success criteria & ratings
# 5. Analyze the Data
Satisfaction
This is another area where access concerns may come up. Discuss with a partner and post your ideas for problems and solutions on [Ed](https://edstem.org/us/courses/31170/discussion/2427390)
Impact when users fail to complete tasks
- Help them?
- Quantitative study impact
- Qualitative study impact
---
[//]: # (Outline Slide)
# Learning Goals for Today
- How to build inclusive experiment designs
- Running an inclusive summative study
- **Plain Language Translation**
---
# Plain Language Translation
- Guidelines for how to write clearly, with a goal of supporting people with a variety of cognitive abilities and reading comprehension levels
---
# Guidelines We Ask You to Practice
.left-column50[
- Use simple words
...
...
@@ -243,54 +344,65 @@ Impact when users fail to complete tasks
]
---
# Use Simple Words
- "as a means of" ---> "to"
- "at the present time" ---> "now"
- <q>as a means of</q>→<q>to</q>
- <q>at the present time</q>→<q>now</q>
---
# Use Positive Language
- Avoid double negatives
- "no fewer than" ---> "at least"
- <q>no fewer than</q>→<q>at least</q>
---
# Use Short Paragraphs and Sentences
- Consider if a clause should become a new sentence
---
# Avoid Jargon
- Sometimes the complex word isn't necessary
- If you must use jargon, define it
---
# Use Active Voice
- "The lake was polluted by the company" ---> "The company polluted the lake"
- <q>The lake was polluted by the company</q>→<q>The company polluted the lake</q>
- A good check to see if you're using passive voice: are you using
- A form of "to be" (e.g., am, is, are, was, be, being, been)
- A verb ending in "-ed"
- A form of <q>to be</q> (e.g., am, is, are, was, be, being, been)
- A verb ending in <q>-ed</q>
---
# Use Present Tense
- Don't: "These sections describe types of information that would satisfy the application requirements of Circular A-110 as it would apply to this grant program."
- Do: "These sections tell you how to meet the requirements of Circular A-110 for this grant program."
- Don't: <q>These sections describe types of information that would satisfy the application requirements of Circular A-110 as it would apply to this grant program.</q>
- Do: <q>These sections tell you how to meet the requirements of Circular A-110 for this grant program.</q>
---
# Use Examples
- Especially for more complicated topics
---
# Use Transition Words
- Use words: "for instance," "also," "however," "to summarize"
- Use words: <q>for instance,</q><q>also,</q><q>however,</q><q>to summarize</q>
- Echo previous ideas
- Avoid ambiguous "this"
- Avoid ambiguous <q>this</q>
---
# Use Headers
- Preferably, use headers that are built into platforms (e.g., Google Docs, Microsoft Word, EdStem)
- If those are not available, styling text in bold or underlined can still help visual readers
- Live Demo!
---
# Plain Language Resources
- If you have any questions or want examples about any of these concepts, you can look at the [PlainLanguage.gov website](https://www.plainlanguage.gov/guidelines/).
\ No newline at end of file
- If you have any questions or want examples about any of these concepts, you can look at the [PlainLanguage.gov website](https://www.plainlanguage.gov/guidelines/).
