Forked from
teaching-materials / website
462 commits behind, 2 commits ahead of the upstream repository.
-
Jennifer Mankoff authoredJennifer Mankoff authored
evaluation.html 10.57 KiB
---
layout: presentation
title: Evaluation --Week 4--
description: Accessibility Needfinding and Evaluation
class: middle, center, inverse
---
background-image: url(img/people.png)
.left-column50[
# Week 4: Accessible Evaluation
{{site.classnum}}, {{site.quarter}}
]
---
name: normal
layout: true
class:
---
# Important Reminder
## This is an important reminder
## Make sure zoom is running and recording!!!
---
[//]: # (Outline Slide)
# Learning Goals for Today
- How to build inclusive experiment designs
- Running an inclusive summative study
- Plain Language Translation
---
# Summative User Testing
So you have an app and you think it's accessible. How do you check that?
.quote[Unlike formative evaluations, whose goals is to inform the design process, summative evaluations involve getting the big picture and assessing the overall experience of a finished product...] - Nielsen Norman Group
Lots of variables here, but
- typically checks that an app works as expected on standard tasks when used by the target population
- may compare features or apps quantitatively
- may involve think aloud or other qualitative data
---
# Why not Start with Formative User Testing?
- Many of the apps we deploy are designed for people in general
- Disabled people need to be able to use those apps too
- Summative testing is the gold standard in assessing accessibility
Of course the entire design process *should not just include majority class people* and much of this presentation applies to formative testing took
We'll also cover that later today when we talk about designing apps *for* and *with* people with disabilities
---
# *Accessible* Summative User Testing
- Accessibility doesn't come by accident when planning studies
- All research should be accessible research (regardless of if it is accessibility research)
- You have to make your system accessible (using inspection techniques) *before* doing this
- We will discuss accessibility *for evaluators* and *for participants* today
Best guidance: [Nielsen-Norman Group](http://www.nngroup.com/reports/accessibility/testing/); [Anticipate and Adjust](https://a11ykelly.medium.com/anticipate-and-adjust-cultivating-access-in-human-centered-methods-1e46c6845e34)
---
# Accessible Study Planning Workflow
.red[*needs arrows going back]
---
# An Example
Supposed you are hired to perform usability testing of new food-delivery app with people with sensory disabilities.
---
# Identify Stakeholders
Supposed you are hired to perform usability testing of new food-delivery app with people with sensory disabilities.
- Who are the stakeholders who you need to take into account for this study?
- Disabled participants
- Blind people
- Deaf people
- Hard-of-hearing people
- Research team
- Lua -- lead researcher, has dyslexia
- Jay -- project team member, Deaf person who uses sign language.
- El -- project team member, non-disabled
---
# Design Study
Supposed you are hired to perform usability testing of new food-delivery app with people with sensory disabilities.
- Who are the stakeholders who you need to take into account for this study?
- What is the right structure for the study?
---
# Design Study (1 of 4)
This is a usability study, so it should include the same tasks as tested for users without disabilities
Metrics should be similar as well -- for example the [System Usability Scale](https://www.usability.gov/how-to-and-tools/methods/system-usability-scale.html) might be useful at the end
---
# Design Study (2 of 4)
This is also a study of accessibility, with disabled users. The study design needs to consider
.left-column50[
- Choice of method
- e.g. think aloud may be more difficult for a nonspeaking individual, or someone with fatigue
- Check in on study measures and statistical design. Does user heterogeneity impact this?
<!-- - Study venue (It may be difficult to duplicate participant hardware/software) -->
]
.right-column50[
- Ensure that participants' hardware/software also addresses study needs (e.g. do participants have a working monitor?)
- Study length (estimate before and after piloting)
- Study length may increase for some disabled participnats
- Study length may negatively impact participants with fatigue-related concerns
]
---
# Design Study (3 of 4)
Study metrics may also need revision.
.left-column50[
Consider adding
- Click errors
- Success criteria & ratings
- Accessibility errors]
.left-column50[
Consider changing
- Any metrics that include abelist assumptions
- Approach to time based measurements
]
---
# Design Study (4 of 4)
It is also important to consider ethical issues
.left-column60[
- What responsibility do you have to participants in terms of providing skills and help in & out of study?
- What responsibility do you have to give participants a participatory role in design?
- How do you ensure the integrity of your record of their experience?
- Are you compensating them appropriately and addressing costly transportation needs?
]
.right-column40[
These changes may in turn impact
- Expense
- Scalability
]
---
# Plan Study Tasks
Supposed you are hired to perform usability testing of new food-delivery app with people with sensory disabilities.
- Who are the stakeholders who you need to take into account for this study?
- What is the right structure for the study?
- What study tasks should this team prepare for?
---
# Tasks List
To perform standard usability testing on the app based on disabilities represented in sample
0. Assign Tasks
1. Train study team
2. Handle recruitment
3. Pilot with people with disabilities
4. Run the Study
- Prepare study materials (e.g., print out protocols)
- Hire access support staff
- Participate in the study
- Compensate participants
5. Analyze the data
---
---
# 1. Assign Tasks
- Consider:
- access needs
- effects of familiarity with the task
- personal preferences
- other institutional constraints
- For this scenario:
- Lua has significant experience running studies; Lua needs a study script in large print with a specific font
- El wants to gain more experience running studies
- Jay has significant experience with ASL and Deaf Space; Jay prefers to have interpreters
---
# Task Assignment List
1. ~~Assign Tasks~~
2. **Lua**: Train study team
3. **Jay**: Handle recruitment
4. **Jay**: Pilot with people with disabilities5. **Lua**: Run the Study
- **Lua**: Prepare study materials (e.g., print out protocols)
- **Jay**: Hire access support staff
- **Participants**: Participate in the study
- **El**: Compensate participants
6. **All**: Analyze the data
---
# 1. Train study team (1 of 2)
As study lead, and a person with both disability and study related experience, Lua will train the study team.
Her goal is to ensure that El and Jay are aware of best practices and considerations that might impact study design before they begin interacting with participants.
---
# 1. Train study team (2 of 2)
Some things they might discuss include
- Ensuring that they address access needs and communication support from recruitment onward
- Providing basic DEIA training as needed so that all those with participant contact know basics.
- Don’t be overly sensitive (don’t overcompensate)
- Don’t rely on useless cues (audio/visual) to convey encouragement
- Monitor participant fatigue carefully
- When to help and when to end a task if participants have trouble
---
# 2. Handle Recruitment
As a person already connected to the disability community, the team determins that Jay will be most likely to be trusted, and to identify opportunities to build trust throughout the recruitment process
- Are your recruitment materials accessible
- How will you find participants?
- How much do you know about the group your are targeting and what to expect from/of them?
- Will they see you as trustworthy?
- Are you giving them enough agency in the study process?
- Maybe start with gatekeepers
- Negotiating access often requires working with gatekeepers
- Participants may see researchers as “just another professional, conducting professional surveillance”
---
# 3. Pilot study
First, revisit accommodations plan
.left-column50[
Consider:
- Communication
- Materials
- Time
- Space
]
--
.right-column50[
Prepare:
- Hire interpreters (at least 2 weeks in advance)
- Print out accessible study materials, consent form, etc
- Set up the room
]
---
# Reflect
- Lua gets tired from reading overtime
- Jay knows ASL and Deaf norms
- Study length was problematic for one Blind participant who also had a chronic illness
- Sometimes sightlines made it hard for Jay to see the interpreter
---
# Update Study Design Based on Reflection
- Institute a protocol for monitoring fatigue
- Add a cutoff for tasks that take a long time
- Rearrange the room to better support communication with interpreters
- ...
---
# Update Tasks List Based on Reflection
1. ~~Assign Tasks~~
2. ~~Lua: Train study team~~ (done)
3. **Jay**: Handle recruitment (ongoing)
4. ~~Jay: Pilot with people with disabilities~~ (done)
5. ~~Lua~~ **Jay & El**: Run the Study
- **Lua**: Prepare study materials (e.g., print out protocols)
- **Jay**: Hire access support staff
- **Participants**: Participate in the study
- **El**: Compensate participants
6. **All**: Analyze the data
---
# 4. Run the Study
At this point the team should be well prepared. However, it is still important to monitor both the team's and the participants' needs and adjust as accessibility concerns arise.
---
# 5. Analyze the Data
This is another area where access concerns may come up.
- Data analysis tools may not be accessible
- Team members may have different needs with respect to written versus audio transcripts
- <q>Member Checking</q>
---
# Try it
You read about Alexa's use by people with disabilities in class. Form small groups
- Identify potential tasks to test
- Write down a sampling goal (who is included)
- Write down a list of accessibility assessment goals (metrics)
- Identify potential accessibility concerns
Post on [Ed](https://edstem.org/us/courses/31170/discussion/2427390)