Skip to content
Snippets Groups Projects
Commit d3bea1dc authored by Jennifer Mankoff's avatar Jennifer Mankoff
Browse files

Merge branch 'jen-wk6' into 'UACCESS-23fa' 

Jen wk6

See merge request undergrad-accessibility-website!56
parents 23ed7a83 881f5ad4
No related branches found
No related tags found
3 merge requests!12Accessibility website jen summer work,!9Q access 24 sp,!8Q access 24 sp
Hide whitespace changes
Inline Side-by-side
schedule.md
+ 4
− 3
View file @ d3bea1dc
...@@ -256,10 +256,10 @@ Check in to make sure you have a plan and it is do-able ...@@ -256,10 +256,10 @@ Check in to make sure you have a plan and it is do-able
: :
- **Required: Respond to the Reading Questions and Preparation Requirements.** - **Required: Respond to the Reading Questions and Preparation Requirements.**
- [Consumer-Grade Fabrication and Its Potential to Revolutionize Accessibility](https://dl.acm.org/doi/10.1145/3339824), CACM 2019 and [**Respond**](TBD) - Read [Lasercutting with Tinkercad](https://www.tinkercad.com/blog/laser-cutting-with-Tinkercad) and join our (Tinkercard Classroom)[https://www.tinkercad.com/joinclass/IEPJ5JFZ3] and complete the tutorial in our [Laser Cut Accessibility Aid](https://www.tinkercad.com/classrooms/gVpVydMts11/activities/e98dyTpqqmK) activity.
- [Areas of Strategic Visibility: Disability Bias in Biometrics](https://arxiv.org/abs/2208.04712) and [**Respond**](TBD) - Read [Empowering individuals with do-it-yourself assistive technology](https://drive.google.com/open?id=17M_-NY35P0wnXTD1kacD33Tt43lqwMsu&usp=drive_fs) and respond to the reading question on Ed.
- **If you want to go deeper** - **If you want to go deeper**
- [Explaining Explanations: An Approach to Evaluating Interpretability of Machine Learning](https://www.cs.purdue.edu/homes/bb/nit/Lalana-Explainations%20of%20IAS.pdf) - [Consumer-Grade Fabrication and Its Potential to Revolutionize Accessibility](https://dl.acm.org/doi/10.1145/3339824), CACM 2019
- [Fairness issues in AI Systems that augment sensory abilities](https://arxiv.org/pdf/1908.07333.pdf) - [Fairness issues in AI Systems that augment sensory abilities](https://arxiv.org/pdf/1908.07333.pdf)
{: .homework} **Assigned:** [Technology Review](assignments/technology-review.html) {: .homework} **Assigned:** [Technology Review](assignments/technology-review.html)
...@@ -297,6 +297,7 @@ Learn about how to make AR/VR interfaces and games accessible ...@@ -297,6 +297,7 @@ Learn about how to make AR/VR interfaces and games accessible
: :
- **Required: Respond to the Reading Questions and Preparation Requirements.** - **Required: Respond to the Reading Questions and Preparation Requirements.**
- [Areas of Strategic Visibility: Disability Bias in Biometrics](https://arxiv.org/abs/2208.04712 - [Explaining Explanations: An Approach to Evaluating Interpretability of Machine Learning](https://www.cs.purdue.edu/homes/bb/nit/Lalana-Explainations%20of%20IAS.pdf)
- [Increasing Data Equity in Accessibility](https://arxiv.org/abs/2210.01902) - [Increasing Data Equity in Accessibility](https://arxiv.org/abs/2210.01902)
- **If you want to go deeper** - **If you want to go deeper**
- [A systematic literature review of handheld augmented reality solutions for people with disabilities](https://www.mdpi.com/1424-8220/22/20/7719) - [A systematic literature review of handheld augmented reality solutions for people with disabilities](https://www.mdpi.com/1424-8220/22/20/7719)
......
slides/evaluation.html
+ 128
− 59
View file @ d3bea1dc
...@@ -28,7 +28,7 @@ class: ...@@ -28,7 +28,7 @@ class:
[//]: # (Outline Slide) [//]: # (Outline Slide)
# Learning Goals for Today # Learning Goals for Today
- Presenting Acessibly - Presenting Accessibly
- How to build inclusive experiment designs - How to build inclusive experiment designs
- Running an inclusive summative study - Running an inclusive summative study
...@@ -38,7 +38,7 @@ class: ...@@ -38,7 +38,7 @@ class:
So you have an app and you think it's accessible. How do you check that? So you have an app and you think it's accessible. How do you check that?
.quote[Unlike formative evaluations, whose goals is to inform the design process, summative evaluations involve getting the big picture and assessing the overall experience of a finished product...] - Nielsen Norman Group .quote[...getting the big picture and assessing the overall experience of a finished product...] - Nielsen Norman Group
Lots of variables here, but Lots of variables here, but
- typically checks that an app works as expected on standard tasks when used by the target population - typically checks that an app works as expected on standard tasks when used by the target population
...@@ -47,6 +47,18 @@ Lots of variables here, but ...@@ -47,6 +47,18 @@ Lots of variables here, but
--- ---
# Why not Start with Formative User Testing?
- Many of the apps we deploy are designed for people in general
- Disabled people need to be able to use those apps too
- Summative testing is the gold standard in assessing accessibility
Of course the entire design process *should not just include majority class people* and much of this presentation applies to formative testing took
We'll also cover that later this week when we talk about designing apps *for* and *with* people with disabilities
---
# *Accessible* Summative User Testing # *Accessible* Summative User Testing
- Accessibility doesn't come by accident when planning studies - Accessibility doesn't come by accident when planning studies
...@@ -59,7 +71,48 @@ Best guidance: [Nielsen-Norman Group](http://www.nngroup.com/reports/accessibili ...@@ -59,7 +71,48 @@ Best guidance: [Nielsen-Norman Group](http://www.nngroup.com/reports/accessibili
--- ---
# Accessible Study Planning Workflow # Accessible Study Planning Workflow
![:img A flow chart for planning accessible studies. The first two phases occur in parallel and are "identify stakeholders" and "identify tasks". These tasks flow into the "assign tasks" phase where one should consider access needs, effects of familiarity with the task, personal preferences, and other institutional constraints. The next phase is "planning accommodations", where one can consider the dimensions of communication, materials, time and space. And the final phase is reflect on the study plan while considering access synergies and conflicts and power dynamics.](img/wk04/access_workflow.png)
<div class="mermaid">
%%{init: {'theme':'base', 'themeVariables': { 'primaryColor': '#4CAF50', 'tertiaryColor': '#009688', 'fontSize': '16px', 'textMargin': '0px', 'text-align': 'left' }}%%
flowchart LR
A(Identify stakeholders)
B(Identify tasks)
A <--> C
B <--> C
C <--> D
D <--> E
subgraph Assign tasks
C(Access needs
Effects of familiarity
Personal preferences
Institutional constraints)
end
subgraph Plan Accomm.
D(Communication
Materials
Time
Space)
end
subgraph Reflect
E(Access Synergies
Access conflicts
Power Dynamics)
end
classDef default fill:#009688,stroke:#333,stroke-width:2px, color:white;
classDef reflect fill:#4CAF50,stroke:#333,stroke-width:2px, color:white;
class A,B,C,D default
class A,B,C,D,E reflect
linkStyle default stroke: black,stroke-width:8px
</div>
--- ---
...@@ -72,59 +125,55 @@ Supposed you are hired to perform usability testing of new food-delivery app wit ...@@ -72,59 +125,55 @@ Supposed you are hired to perform usability testing of new food-delivery app wit
# Identify Stakeholders # Identify Stakeholders
Supposed you are hired to perform usability testing of new food-delivery app with people with sensory disabilities. .left-column[you are hired to perform usability testing of new food-delivery app with people with sensory disabilities.]
- Who are the stakeholders who you need to take into account for this study?
---
# Identify Stakeholders .right-column[
Who are the stakeholders who you need to take into account for this study?
Supposed you are hired to perform usability testing of new food-delivery app with people with sensory disabilities. - Disabled participants (BVI, DHH)
- Who are the stakeholders who you need to take into account for this study?
- Disabled participants
- Blind people
- Deaf people
- Hard-of-hearing people
- Research team - Research team
- Lua -- lead researcher, has dyslexia - Lua -- lead researcher, has dyslexia
- Jay -- project team member, Deaf person who uses sign language. - Jay -- project team member, Deaf person who uses sign language.
- El -- project team member, non-disabled - El -- project team member, non-disabled
]
--- ---
# Design Study # Design Study
Supposed you are hired to perform usability testing of new food-delivery app with people with sensory disabilities. .left-column[you are hired to perform usability testing of new food-delivery app with people with sensory disabilities.]
.right-column[
- Who are the stakeholders who you need to take into account for this study? - Who are the stakeholders who you need to take into account for this study?
- What is the right structure for the study? - What is the right structure for the study?
]
--- ---
# Design Study (1 of 4) # Design Study (1 of 5)
This is a usability study, so it should include the same tasks as tested for users without disabilities This is a usability study, so it should include the same tasks as tested for users without disabilities
Metrics should be similar as well -- for example the [System Usability Scale](https://www.usability.gov/how-to-and-tools/methods/system-usability-scale.html) might be useful at the end Metrics should be similar as well -- for example the [System Usability Scale](https://www.usability.gov/how-to-and-tools/methods/system-usability-scale.html) might be useful at the end
--- ---
# Design Study (2 of 4) # Design Study (2 of 5)
This is also a study of accessibility, with disabled users. The study design needs to consider This is also a study of accessibility, with disabled users. The study design needs to consider
.left-column50[
- Choice of method - Choice of method
- e.g. think aloud may be more difficult for a nonspeaking individual, or someone with fatigue - e.g. think aloud may be more difficult for a nonspeaking individual, or someone with fatigue
- Check in on study measures and statistical design. Does user heterogeneity impact this? - Check in on study measures and statistical design. Does user heterogeneity impact this?
<!-- - Study venue (It may be difficult to duplicate participant hardware/software) --> <!-- - Study venue (It may be difficult to duplicate participant hardware/software) -->
]
.right-column50[ ---
# Design Study (3 of 5)
This is also a study of accessibility, with disabled users. The study design needs to consider
- Ensure that participants' hardware/software also addresses study needs (e.g. do participants have a working monitor?) - Ensure that participants' hardware/software also addresses study needs (e.g. do participants have a working monitor?)
- Study length (estimate before and after piloting) - Study length (estimate before and after piloting)
- Study length may increase for some disabled participnats - Study length may increase for some disabled participants
- Study length may negatively impact participants with fatigue-related concerns - Study length may negatively impact participants with fatigue-related concerns
]
--- ---
# Design Study (3 of 4) # Design Study (4 of 5)
Study metrics may also need revision. Study metrics may also need revision.
.left-column50[ .left-column50[
...@@ -135,25 +184,25 @@ Consider adding ...@@ -135,25 +184,25 @@ Consider adding
] ]
.left-column50[ .left-column50[
Consider changing Consider changing
- Any metrics that include abelist assumptions - Any metrics that include ableist assumptions
- Approach to time based measurements - Approach to time based measurements
] ]
--- ---
# Design Study (4 of 4) # Design Study (5 of 5)
It is also important to consider ethical issues It is also important to consider ethical issues
.left-column60[
- What responsibility do you have to participants in terms of providing skills and help in & out of study? - What responsibility do you have to participants in terms of providing skills and help in & out of study?
- What responsibility do you have to give participants a participatory role in design?
- How do you ensure the integrity of your record of their experience? - How do you ensure the integrity of your record of their experience?
- Are you compensating them appropriately and addressing costly transportation needs? - Are you compensating them appropriately and addressing costly transportation needs?
]
.right-column40[ ???
These changes may in turn impact These changes may in turn impact
- Expense - Expense
- Scalability - Scalability
]
--- ---
...@@ -169,20 +218,17 @@ Supposed you are hired to perform usability testing of new food-delivery app wit ...@@ -169,20 +218,17 @@ Supposed you are hired to perform usability testing of new food-delivery app wit
# Tasks List # Tasks List
To perform standard usability testing on the app based on disabilities represented in sample To perform standard usability testing on the app based on disabilities represented in sample
1. Train study team 1. Assign Tasks
2. Handle recruitment 2. Train study team
3. Pilot with people with disabilities 3. Handle recruitment
4. Run the Study 4. Pilot with people with disabilities
- Prepare study materials (e.g., print out protocols) 5. Run the Study
- Hire access support staff 6. Analyze the data
- Participate in the study
- Compensate participants
5. Analyze the data
--- ---
--- ---
# Assign Tasks # 1. Assign Tasks
- Consider: - Consider:
- access needs - access needs
- effects of familiarity with the task - effects of familiarity with the task
...@@ -196,19 +242,20 @@ To perform standard usability testing on the app based on disabilities represent ...@@ -196,19 +242,20 @@ To perform standard usability testing on the app based on disabilities represent
--- ---
# Task Assignment List # Task Assignment List
1. **Lua**: Train study team 1. ~~Assign Tasks~~
2. **Jay**: Handle recruitment 2. **Lua**: Train study team
3. **Jay**: Pilot with people with disabilities 3. **Jay**: Handle recruitment
4. **Lua**: Run the Study 4. **Jay**: Pilot with people with disabilities
5. **Lua**: Run the Study
- **Lua**: Prepare study materials (e.g., print out protocols) - **Lua**: Prepare study materials (e.g., print out protocols)
- **Jay**: Hire access support staff - **Jay**: Hire access support staff
- **Participants**: Participate in the study - **Participants**: Participate in the study
- **El**: Compensate participants - **El**: Compensate participants
5. **All**: Analyze the data 6. **All**: Analyze the data
--- ---
# 1. Train study team (1 of 2) # 2. Train study team (1 of 2)
As study lead, and a person with both disability and study related experience, Lua will train the study team. As study lead, and a person with both disability and study related experience, Lua will train the study team.
...@@ -216,7 +263,7 @@ Her goal is to ensure that El and Jay are aware of best practices and considerat ...@@ -216,7 +263,7 @@ Her goal is to ensure that El and Jay are aware of best practices and considerat
--- ---
# 1. Train study team (2 of 2) # 2. Train study team (2 of 2)
Some things they might discuss include Some things they might discuss include
- Ensuring that they address access needs and communication support from recruitment onward - Ensuring that they address access needs and communication support from recruitment onward
...@@ -227,11 +274,19 @@ Some things they might discuss include ...@@ -227,11 +274,19 @@ Some things they might discuss include
- When to help and when to end a task if participants have trouble - When to help and when to end a task if participants have trouble
--- ---
# 2. Handle Recruitment # 3. Handle Recruitment (1/2)
As a person already connected to the disability community, the team determins that Jay will be most likely to be trusted, and to identify opportunities to build trust throughout the recruitment process As a person already connected to the disability community, the team determines that Jay will be most likely to be trusted, and to identify opportunities to build trust throughout the recruitment process.
Recruitment raises a number of access issues
- Are your recruitment materials accessible - Are your recruitment materials accessible
- How will you find participants? - How will you find participants (next slide)
- How will you compensate participants?
---
# 3. Handle Recruitment (2/2)
Finding participants may require careful work
- How much do you know about the group your are targeting and what to expect from/of them? - How much do you know about the group your are targeting and what to expect from/of them?
- Will they see you as trustworthy? - Will they see you as trustworthy?
- Are you giving them enough agency in the study process? - Are you giving them enough agency in the study process?
...@@ -240,7 +295,7 @@ As a person already connected to the disability community, the team determins th ...@@ -240,7 +295,7 @@ As a person already connected to the disability community, the team determins th
- Participants may see researchers as “just another professional, conducting professional surveillance” - Participants may see researchers as “just another professional, conducting professional surveillance”
--- ---
# 3. Pilot study # 4. Pilot study
First, revisit accommodations plan First, revisit accommodations plan
...@@ -257,6 +312,7 @@ Prepare: ...@@ -257,6 +312,7 @@ Prepare:
- Hire interpreters (at least 2 weeks in advance) - Hire interpreters (at least 2 weeks in advance)
- Print out accessible study materials, consent form, etc - Print out accessible study materials, consent form, etc
- Set up the room - Set up the room
- Have compensation ready
] ]
--- ---
...@@ -292,16 +348,29 @@ Prepare: ...@@ -292,16 +348,29 @@ Prepare:
--- ---
# 4. Run the Study # 5. Run the Study
At this point the team should be well prepared. However, it is still important to monitor both the team's and the participants' needs and adjust as accessibility concerns arise. At this point the team should be well prepared. However, it is still important to monitor both the team's and the participants' needs and adjust as accessibility concerns arise.
Similar to piloting a study
--- ---
# 5. Analyze the Data # 6. Analyze the Data
This is another area where access concerns may come up. This is another area where access concerns may come up.
- Data analysis tools may not be accessible - Data analysis tools may not be accessible
- Team members may have different needs with respect to written versus audio transcripts - Team members may have different needs with respect to written versus audio transcripts
- <q>Member Checking</q> - <q>Member Checking</q>
---
# (If time) Try it
You read about Alexa's use by people with disabilities in class. Form small groups
- Identify potential tasks to test
- Write down a sampling goal (who is included)
- Write down a list of accessibility assessment goals (metrics)
- Identify potential accessibility concerns
Post on [Ed](https://edstem.org/us/courses/31170/discussion/2427390)
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment