From 881f5ad435b947207aad6325a8d8a92a314ba088 Mon Sep 17 00:00:00 2001
From: Jennifer Mankoff <jmankoff@cs.washington.edu>
Date: Sun, 29 Oct 2023 08:32:19 -0700
Subject: [PATCH] Jen wk6
---
schedule.md | 7 +-
slides/evaluation.html | 187 ++++++++++++++++++++++++++++-------------
2 files changed, 132 insertions(+), 62 deletions(-)
diff --git a/schedule.md b/schedule.md
index 3c4f7c45..79eb1666 100644
--- a/schedule.md
+++ b/schedule.md
@@ -256,10 +256,10 @@ Check in to make sure you have a plan and it is do-able
:
- **Required: Respond to the Reading Questions and Preparation Requirements.**
- - [Consumer-Grade Fabrication and Its Potential to Revolutionize Accessibility](https://dl.acm.org/doi/10.1145/3339824), CACM 2019 and [**Respond**](TBD)
- - [Areas of Strategic Visibility: Disability Bias in Biometrics](https://arxiv.org/abs/2208.04712) and [**Respond**](TBD)
+ - Read [Lasercutting with Tinkercad](https://www.tinkercad.com/blog/laser-cutting-with-Tinkercad) and join our (Tinkercard Classroom)[https://www.tinkercad.com/joinclass/IEPJ5JFZ3] and complete the tutorial in our [Laser Cut Accessibility Aid](https://www.tinkercad.com/classrooms/gVpVydMts11/activities/e98dyTpqqmK) activity.
+ - Read [Empowering individuals with do-it-yourself assistive technology](https://drive.google.com/open?id=17M_-NY35P0wnXTD1kacD33Tt43lqwMsu&usp=drive_fs) and respond to the reading question on Ed.
- **If you want to go deeper**
- - [Explaining Explanations: An Approach to Evaluating Interpretability of Machine Learning](https://www.cs.purdue.edu/homes/bb/nit/Lalana-Explainations%20of%20IAS.pdf)
+ - [Consumer-Grade Fabrication and Its Potential to Revolutionize Accessibility](https://dl.acm.org/doi/10.1145/3339824), CACM 2019
- [Fairness issues in AI Systems that augment sensory abilities](https://arxiv.org/pdf/1908.07333.pdf)
{: .homework} **Assigned:** [Technology Review](assignments/technology-review.html)
@@ -297,6 +297,7 @@ Learn about how to make AR/VR interfaces and games accessible
:
- **Required: Respond to the Reading Questions and Preparation Requirements.**
+ - [Areas of Strategic Visibility: Disability Bias in Biometrics](https://arxiv.org/abs/2208.04712 - [Explaining Explanations: An Approach to Evaluating Interpretability of Machine Learning](https://www.cs.purdue.edu/homes/bb/nit/Lalana-Explainations%20of%20IAS.pdf)
- [Increasing Data Equity in Accessibility](https://arxiv.org/abs/2210.01902)
- **If you want to go deeper**
- [A systematic literature review of handheld augmented reality solutions for people with disabilities](https://www.mdpi.com/1424-8220/22/20/7719)
diff --git a/slides/evaluation.html b/slides/evaluation.html
index 21dc51a2..89978ae5 100644
--- a/slides/evaluation.html
+++ b/slides/evaluation.html
@@ -28,7 +28,7 @@ class:
[//]: # (Outline Slide)
# Learning Goals for Today
-- Presenting Acessibly
+- Presenting Accessibly
- How to build inclusive experiment designs
- Running an inclusive summative study
@@ -38,7 +38,7 @@ class:
So you have an app and you think it's accessible. How do you check that?
-.quote[Unlike formative evaluations, whose goals is to inform the design process, summative evaluations involve getting the big picture and assessing the overall experience of a finished product...] - Nielsen Norman Group
+.quote[...getting the big picture and assessing the overall experience of a finished product...] - Nielsen Norman Group
Lots of variables here, but
- typically checks that an app works as expected on standard tasks when used by the target population
@@ -47,6 +47,18 @@ Lots of variables here, but
---
+# Why not Start with Formative User Testing?
+
+- Many of the apps we deploy are designed for people in general
+- Disabled people need to be able to use those apps too
+- Summative testing is the gold standard in assessing accessibility
+
+Of course the entire design process *should not just include majority class people* and much of this presentation applies to formative testing took
+
+We'll also cover that later this week when we talk about designing apps *for* and *with* people with disabilities
+
+---
+
# *Accessible* Summative User Testing
- Accessibility doesn't come by accident when planning studies
@@ -59,7 +71,48 @@ Best guidance: [Nielsen-Norman Group](http://www.nngroup.com/reports/accessibili
---
# Accessible Study Planning Workflow
-
+
+<div class="mermaid">
+%%{init: {'theme':'base', 'themeVariables': { 'primaryColor': '#4CAF50', 'tertiaryColor': '#009688', 'fontSize': '16px', 'textMargin': '0px', 'text-align': 'left' }}%%
+
+flowchart LR
+
+ A(Identify stakeholders)
+ B(Identify tasks)
+
+ A <--> C
+ B <--> C
+ C <--> D
+ D <--> E
+
+ subgraph Assign tasks
+ C(Access needs
+ Effects of familiarity
+ Personal preferences
+ Institutional constraints)
+ end
+
+ subgraph Plan Accomm.
+ D(Communication
+ Materials
+ Time
+ Space)
+ end
+
+ subgraph Reflect
+ E(Access Synergies
+ Access conflicts
+ Power Dynamics)
+ end
+
+
+classDef default fill:#009688,stroke:#333,stroke-width:2px, color:white;
+classDef reflect fill:#4CAF50,stroke:#333,stroke-width:2px, color:white;
+class A,B,C,D default
+class A,B,C,D,E reflect
+linkStyle default stroke: black,stroke-width:8px
+
+</div>
---
@@ -72,59 +125,55 @@ Supposed you are hired to perform usability testing of new food-delivery app wit
# Identify Stakeholders
-Supposed you are hired to perform usability testing of new food-delivery app with people with sensory disabilities.
-- Who are the stakeholders who you need to take into account for this study?
-
----
+.left-column[you are hired to perform usability testing of new food-delivery app with people with sensory disabilities.]
-# Identify Stakeholders
+.right-column[
+Who are the stakeholders who you need to take into account for this study?
-Supposed you are hired to perform usability testing of new food-delivery app with people with sensory disabilities.
-- Who are the stakeholders who you need to take into account for this study?
-
- - Disabled participants
- - Blind people
- - Deaf people
- - Hard-of-hearing people
+ - Disabled participants (BVI, DHH)
- Research team
- - Lua -- lead researcher, has dyslexia
- - Jay -- project team member, Deaf person who uses sign language.
- - El -- project team member, non-disabled
-
+ - Lua -- lead researcher, has dyslexia
+ - Jay -- project team member, Deaf person who uses sign language.
+ - El -- project team member, non-disabled
+]
---
# Design Study
-Supposed you are hired to perform usability testing of new food-delivery app with people with sensory disabilities.
+.left-column[you are hired to perform usability testing of new food-delivery app with people with sensory disabilities.]
+.right-column[
- Who are the stakeholders who you need to take into account for this study?
- What is the right structure for the study?
-
+]
---
-# Design Study (1 of 4)
+# Design Study (1 of 5)
This is a usability study, so it should include the same tasks as tested for users without disabilities
Metrics should be similar as well -- for example the [System Usability Scale](https://www.usability.gov/how-to-and-tools/methods/system-usability-scale.html) might be useful at the end
---
-# Design Study (2 of 4)
+# Design Study (2 of 5)
This is also a study of accessibility, with disabled users. The study design needs to consider
-.left-column50[
+
- Choice of method
- e.g. think aloud may be more difficult for a nonspeaking individual, or someone with fatigue
- Check in on study measures and statistical design. Does user heterogeneity impact this?
<!-- - Study venue (It may be difficult to duplicate participant hardware/software) -->
-]
-.right-column50[
+
+---
+# Design Study (3 of 5)
+
+This is also a study of accessibility, with disabled users. The study design needs to consider
+
- Ensure that participants' hardware/software also addresses study needs (e.g. do participants have a working monitor?)
- Study length (estimate before and after piloting)
- - Study length may increase for some disabled participnats
+ - Study length may increase for some disabled participants
- Study length may negatively impact participants with fatigue-related concerns
-]
---
-# Design Study (3 of 4)
+# Design Study (4 of 5)
Study metrics may also need revision.
.left-column50[
@@ -135,25 +184,25 @@ Consider adding
]
.left-column50[
Consider changing
-- Any metrics that include abelist assumptions
+- Any metrics that include ableist assumptions
- Approach to time based measurements
]
---
-# Design Study (4 of 4)
+# Design Study (5 of 5)
It is also important to consider ethical issues
-.left-column60[
+
- What responsibility do you have to participants in terms of providing skills and help in & out of study?
+- What responsibility do you have to give participants a participatory role in design?
- How do you ensure the integrity of your record of their experience?
- Are you compensating them appropriately and addressing costly transportation needs?
-]
-.right-column40[
+
+???
These changes may in turn impact
- Expense
- Scalability
-]
---
@@ -169,20 +218,17 @@ Supposed you are hired to perform usability testing of new food-delivery app wit
# Tasks List
To perform standard usability testing on the app based on disabilities represented in sample
-1. Train study team
-2. Handle recruitment
-3. Pilot with people with disabilities
-4. Run the Study
- - Prepare study materials (e.g., print out protocols)
- - Hire access support staff
- - Participate in the study
- - Compensate participants
-5. Analyze the data
+1. Assign Tasks
+2. Train study team
+3. Handle recruitment
+4. Pilot with people with disabilities
+5. Run the Study
+6. Analyze the data
---
---
-# Assign Tasks
+# 1. Assign Tasks
- Consider:
- access needs
- effects of familiarity with the task
@@ -196,19 +242,20 @@ To perform standard usability testing on the app based on disabilities represent
---
# Task Assignment List
-1. **Lua**: Train study team
-2. **Jay**: Handle recruitment
-3. **Jay**: Pilot with people with disabilities
-4. **Lua**: Run the Study
+1. ~~Assign Tasks~~
+2. **Lua**: Train study team
+3. **Jay**: Handle recruitment
+4. **Jay**: Pilot with people with disabilities
+5. **Lua**: Run the Study
- **Lua**: Prepare study materials (e.g., print out protocols)
- **Jay**: Hire access support staff
- **Participants**: Participate in the study
- **El**: Compensate participants
-5. **All**: Analyze the data
+6. **All**: Analyze the data
---
-# 1. Train study team (1 of 2)
+# 2. Train study team (1 of 2)
As study lead, and a person with both disability and study related experience, Lua will train the study team.
@@ -216,7 +263,7 @@ Her goal is to ensure that El and Jay are aware of best practices and considerat
---
-# 1. Train study team (2 of 2)
+# 2. Train study team (2 of 2)
Some things they might discuss include
- Ensuring that they address access needs and communication support from recruitment onward
@@ -227,11 +274,19 @@ Some things they might discuss include
- When to help and when to end a task if participants have trouble
---
-# 2. Handle Recruitment
-As a person already connected to the disability community, the team determins that Jay will be most likely to be trusted, and to identify opportunities to build trust throughout the recruitment process
+# 3. Handle Recruitment (1/2)
+As a person already connected to the disability community, the team determines that Jay will be most likely to be trusted, and to identify opportunities to build trust throughout the recruitment process.
+Recruitment raises a number of access issues
- Are your recruitment materials accessible
-- How will you find participants?
+- How will you find participants (next slide)
+- How will you compensate participants?
+
+---
+
+# 3. Handle Recruitment (2/2)
+
+Finding participants may require careful work
- How much do you know about the group your are targeting and what to expect from/of them?
- Will they see you as trustworthy?
- Are you giving them enough agency in the study process?
@@ -240,7 +295,7 @@ As a person already connected to the disability community, the team determins th
- Participants may see researchers as “just another professional, conducting professional surveillanceâ€
---
-# 3. Pilot study
+# 4. Pilot study
First, revisit accommodations plan
@@ -257,6 +312,7 @@ Prepare:
- Hire interpreters (at least 2 weeks in advance)
- Print out accessible study materials, consent form, etc
- Set up the room
+- Have compensation ready
]
---
@@ -292,16 +348,29 @@ Prepare:
---
-# 4. Run the Study
+# 5. Run the Study
-At this point the team should be well prepared. However, it is still important to monitor both the team's and the participants' needs and adjust as accessibility concerns arise.
+At this point the team should be well prepared. However, it is still important to monitor both the team's and the participants' needs and adjust as accessibility concerns arise.
+
+Similar to piloting a study
---
-# 5. Analyze the Data
+# 6. Analyze the Data
This is another area where access concerns may come up.
- Data analysis tools may not be accessible
- Team members may have different needs with respect to written versus audio transcripts
- <q>Member Checking</q>
+---
+# (If time) Try it
+
+You read about Alexa's use by people with disabilities in class. Form small groups
+- Identify potential tasks to test
+- Write down a sampling goal (who is included)
+- Write down a list of accessibility assessment goals (metrics)
+- Identify potential accessibility concerns
+
+Post on [Ed](https://edstem.org/us/courses/31170/discussion/2427390)
+
--
GitLab