diff --git a/assets/css/my-remark.scss b/assets/css/my-remark.scss
index 507b67c31935362bd3c181f1e24d9c22d15e5c48..8d7adf109945fc08c3928035c11b58237322966c 100644
--- a/assets/css/my-remark.scss
+++ b/assets/css/my-remark.scss
@@ -43,12 +43,17 @@ $highlight: #5c8a8a;
}
}
+a {
+ color: $red;
+ text-decoration: underline;
+}
+
.remark-slide-content {
background-size: contain;
}
strong {
- color: $highlight;
+ color: $mediumdark;
}
.comment{
@@ -281,9 +286,6 @@ body.compact {
font-size: $footnote-font !important;
}
-.bold{
- font-weight: bold;
-}
.blue {
color: $blue;
diff --git a/slides/evaluation.html b/slides/evaluation.html
index 33ae0e071b335b701160a265e5b0d3674116e452..d56be3d4a96e5b85c695ac2fe12ce0004ebbee7a 100644
--- a/slides/evaluation.html
+++ b/slides/evaluation.html
@@ -11,12 +11,14 @@ background-image: url(img/people.png)
{{site.classnum}}, {{site.quarter}}
]
+
---
name: normal
layout: true
class:
---
+
# Important Reminder
## This is an important reminder
@@ -24,12 +26,14 @@ class:
---
[//]: # (Outline Slide)
+
# Learning Goals for Today
- How to build inclusive experiment designs
-- Including people with disabilities in data collection in general
- Running an inclusive summative study
+- Plain Language Translation
---
+
# Summative User Testing
So you have an app and you think it's accessible. How do you check that?
@@ -37,9 +41,18 @@ So you have an app and you think it's accessible. How do you check that?
.quote[Summative evaluations describe how well a design performs, often compared to a benchmark such as a prior version of the design or a competitor. Unlike formative evaluations, whose goals is to inform the design process, summative evaluations involve getting the big picture and assessing the overall experience of a finished product. Summative evaluations occur less frequently than formative evaluations, usually right before or right after a redesign.] - Nielsen Norman Group
---
-# Summative User Testing
-Why are we talking about them first?
+# What is Summative User Testing
+
+Lots of variables here, but
+- typically checks that an app works as expected on standard tasks when used by the target population
+- may compare features or apps quantitatively
+- may involve think aloud or other qualitative data
+
+---
+
+# Why not Start with Formative User Testing?
+
- Many of the apps we deploy are designed for people in general
- Disabled people need to be able to use those apps too
- Summative testing is the gold standard in assessing accessibility
@@ -49,53 +62,37 @@ Of course the entire design process *should not just include majority class peop
We'll also cover that later today when we talk about designing apps *for* and *with* people with disabilities
---
-# Summative User Testing
-Best Practices
-
-- Eliminate obvious accessibility & usability problems ahead of time
-- Consider a remote study, or go to their home (familiar spaces & tech)
-- Don’t be overly sensitive (don’t overcompensate)
-- Don’t rely on useless cues (audio/visual) to convey encouragement
-- Monitor participant fatigue carefully
----
# *Accessible* Summative User Testing
- Accessibility doesn't come by accident when planning studies
- All research should be accessible research (regardless of if it is accessibility research)
+- You have to make your system accessible (using inspection techniques) *before* doing this
+- We will discuss accessibility *for evaluators* and *for participants* today
-
----
-#Lab Testing (often must take place in the home).
Best guidance: [Nielsen-Norman Group](http://www.nngroup.com/reports/accessibility/testing/); [Anticipate and Adjust](https://a11ykelly.medium.com/anticipate-and-adjust-cultivating-access-in-human-centered-methods-1e46c6845e34)
-Little details matter
-- How will you find participants?
-- How much do you know about the group your are targeting and what to expect from/of them?
-- How will you communicate?
-- Can you duplicate the custom hardware/software they use, or must you go to them?
-- Do they have hardware you need (e.g. working monitor?)
-
-
-Anticipate and Adjust presents a workflow to help you in answering these questions.
-
---
+
# Accessible Study Planning Workflow
---
+
# An Example
-- Supposed you are hired to perform usability testing of new food-delivery app with people with sensory disabilities.
+Supposed you are hired to perform usability testing of new food-delivery app with people with sensory disabilities.
---
+
# Identify Stakeholders
-- Supposed you are hired to perform usability testing of new food-delivery app with people with sensory disabilities.
+Supposed you are hired to perform usability testing of new food-delivery app with people with sensory disabilities.
- Who are the stakeholders who you need to take into account for this study?
---
+
# Stakeholders List
- Disabled participants
- Blind people
@@ -107,125 +104,229 @@ Anticipate and Adjust presents a workflow to help you in answering these questio
- El -- project team member, non-disabled
---
-# Identify Tasks
-- Supposed you are hired to perform usability testing of new food-delivery app with people with sensory disabilities.
-- What tasks should this team plan for?
+# Design Study
+
+Supposed you are hired to perform usability testing of new food-delivery app with people with sensory disabilities.
+- Who are the stakeholders who you need to take into account for this study?
+- What is the right structure for the study?
+
+---
+# Design Study (1 of 4)
+
+This is a usability study, so it should include the same tasks as tested for users without disabilities
+
+Metrics should be similar as well -- for example the [System Usability Scale](https://www.usability.gov/how-to-and-tools/methods/system-usability-scale.html) might be useful at the end
+
+---
+# Design Study (2 of 4)
+
+This is also a study of accessibility, with disabled users. The study design needs to consider
+.left-column50[
+- Choice of method
+ - e.g. think aloud may be more difficult for a nonspeaking individual, or someone with fatigue
+ - Check in on study measures and statistical design. Does user heterogeneity impact this?
+- Study venue (It may be difficult to duplicate participant hardware/software)
+]
+.right-column50[
+- Ensure that participants' hardware/software also addresses study needs (e.g. do participants have a working monitor?)
+- Study length (estimate before and after piloting)
+ - Study length may increase for some disabled participnats
+ - Study length may negatively impact participants with fatigue-related concerns
+]
+
+---
+# Design Study (3 of 4)
+
+Study metrics may also need revision.
+.left-column50[
+Consider adding
+- Click errors
+- Success criteria & ratings
+- Accessibility errors
+]
+.left-column50[
+Consider changing
+- Any metrics that include abelist assumptions
+- Approach to time based measurements
+]
+
+
+---
+# Design Study (4 of 4)
+
+It is also important to consider ethical issues
+.left-column60[
+- What responsibility do you have to participants in terms of providing skills and help in & out of study?
+- What responsibility do you have to give participants a participatory role in design?
+- How do you ensure the integrity of your record of their experience?
+- Are you compensating them appropriately and addressing costly transportation needs?
+]
+.right-column40[
+These changes may in turn impact
+- Expense
+- Scalability
+]
---
-# Tasks List
-- Perform standard usability testing on the app based on disabilities represented in sample
-- Pilot with people with disabilities
-- Book the space
-- Prepare interview materials (e.g., print out protocols)
-- Hire access support staff
-- Perform the interviews
-- Participate in interviews
-- Compensate participants
+# Plan Study Tasks
+
+Supposed you are hired to perform usability testing of new food-delivery app with people with sensory disabilities.
+- Who are the stakeholders who you need to take into account for this study?
+- What is the right structure for the study?
+- What study tasks should this team prepare for?
+
---
+# Tasks List
+To perform standard usability testing on the app based on disabilities represented in sample
+
+0. Assign Tasks
+1. Train study team
+2. Handle recruitment
+3. Pilot with people with disabilities
+4. Run the Study
+ - Prepare study materials (e.g., print out protocols)
+ - Hire access support staff
+ - Participate in the study
+ - Compensate participants
+5. Analyze the data
+
+---
---
-# Assign Tasks
+
+# 0. Assign Tasks
- Consider:
- access needs
- effects of familiarity with the task
- personal preferences
- other institutional constraints
- For this scenario:
- - Lua has significant experience interviewing; Lua needs interview materials in large print with a specific font
- - El wants to gain more experience interviewing
+ - Lua has significant experience running studies; Lua needs a study script in large print with a specific font
+ - El wants to gain more experience running studies
- Jay has significant experience with ASL and Deaf Space; Jay prefers to have interpreters
---
+
# Task Assignment List
-- All: Perform standard usability testing on the app based on disabilities represented in sample
-- Lua and El: Pilot with people with disabilities
-- Jay: Book the space
-- Lua: Prepare interview materials (e.g., print out protocols)
-- Jay: Hire access support staff
-- Lua: Perform the interviews
-- Participants: Participate in interviews
-- El: Compensate participants
+1. **Lua**: Train study team
+2. **Jay**: Handle recruitment
+3. **Jay**: Pilot with people with disabilities
+4. **Lua**: Run the Study
+ - **Lua**: Prepare study materials (e.g., print out protocols)
+ - **Jay**: Hire access support staff
+ - **Participants**: Participate in the study
+ - **El**: Compensate participants
+5. **All**: Analyze the data
---
-# Plan Accommodations
-- Consider:
- - Communication
- - Materials
- - Time
- - Space
+
+# 1. Train study team (1 of 2)
+
+As study lead, and a person with both disability and study related experience, Lua will train the study team.
+
+Her goal is to ensure that El and Jay are aware of best practices and considerations that might impact study design before they begin interacting with participants.
---
-# Accommodations List
+
+# 1. Train study team (2 of 2)
+
+Some things they might discuss include
+- Ensuring that they address access needs and communication support from recruitment onward
+- Providing basic DEIA training as needed so that all those with participant contact know basics.
+ - Don’t be overly sensitive (don’t overcompensate)
+ - Don’t rely on useless cues (audio/visual) to convey encouragement
+ - Monitor participant fatigue carefully
+ - When to help and when to end a task if participants have trouble
+---
+
+# 2. Handle Recruitment
+As a person already connected to the disability community, the team determins that Jay will be most likely to be trusted, and to identify opportunities to build trust throughout the recruitment process
+
+Lots of details here
+- Are your recruitment materials accessible
+- How will you find participants?
+- How much do you know about the group your are targeting and what to expect from/of them?
+- Will they see you as trustworthy?
+- Are you giving them enough agency in the study process?
+
+---
+# 3. Pilot study
+
+First, plan accommodations
+
+.left-column50[
+Consider:
+- Communication
+- Materials
+- Time
+- Space
+]
+--
+.right-column50[
+Prepare:
- Hire interpreters (at least 2 weeks in advance)
-- Print out accessible interview materials
+- Print out accessible study materials, consent form, etc
- Set up the room
+]
---
+
# Reflect
- Lua gets tired from reading overtime
- Jay knows ASL and Deaf norms
+- Study length was problematic for one Blind participant who also had a chronic illness
+- Sometimes sightlines made it hard for Jay to see the interpreter
---
-# Update Based on Reflection
-- All: Perform standard usability testing on the app based on disabilities represented in sample
-- Lua and El: Pilot with people with disabilities
-- Jay: Book the space
-- Lua: Prepare interview materials (e.g., print out protocols)
-- Jay: Hire access support staff
-- *All*: Perform the interviews
-- Participants: Participate in interviews
-- El: Compensate participants
+# Update Study Design Based on Reflection
+- Institute a protocol for monitoring fatigue
+- Add a cutoff for tasks that take a long time
+- Rearrange the room to better support communication with interpreters
+- ...
---
-# Consider Accessibility in These Common Tasks
-- Picking a method
-- Recruiting
-- Have a conversation with participants about access needs
-- Transportation
-- Preparing the space
-- Obtaining consent
-- Running the Study
-- Writing up results from the study
-- Member checking
-
+# Update Tasks List Based on Reflection
+1. ~~Lua: Train study team~~ (done)
+2. **Jay**: Handle recruitment (ongoing)
+3. ~~Jay: Pilot with people with disabilities~~ (done)
+4. **Jay & El**: Run the Study
+ - **El**: Prepare study materials (e.g., print out protocols)
+ - **Jay**: Hire access support staff
+ - **Participants**: Participate in the study
+ - **El**: Compensate participants
+5. **All**: Analyze the data
---
-# Issues
-Heterogeneous Users
-
-Accountability
-- What responsibility do you have to participants in terms of providing skills and help in & out of study?
-- What responsibility do you have to give participants a participatory role in design?
-- How do you ensure the integrity of your record of their experience?
-
-Expense
-Scalability
+# 4. Run the Study
-Hard to do with early prototypes
+At this point the team should be well prepared. However, it is still important to monitor both the team's and the participants' needs and adjust as accessibility concerns arise.
---
-# Possible Metrics
-Click errors
-Success criteria & ratings
+# 5. Analyze the Data
-Satisfaction
+This is another area where access concerns may come up. Discuss with a partner and post your ideas for problems and solutions on [Ed](https://edstem.org/us/courses/31170/discussion/2427390)
-Impact when users fail to complete tasks
-- Help them?
-- Quantitative study impact
-- Qualitative study impact
+---
+[//]: # (Outline Slide)
+
+# Learning Goals for Today
+- How to build inclusive experiment designs
+- Running an inclusive summative study
+- **Plain Language Translation**
---
+
# Plain Language Translation
- Guidelines for how to write clearly, with a goal of supporting people with a variety of cognitive abilities and reading comprehension levels
---
+
# Guidelines We Ask You to Practice
.left-column50[
- Use simple words
@@ -243,54 +344,65 @@ Impact when users fail to complete tasks
]
---
+
# Use Simple Words
-- "as a means of" ---> "to"
-- "at the present time" ---> "now"
+- <q>as a means of</q> → <q>to</q>
+- <q>at the present time</q> → <q>now</q>
---
+
# Use Positive Language
- Avoid double negatives
-- "no fewer than" ---> "at least"
+- <q>no fewer than</q> → <q>at least</q>
---
+
# Use Short Paragraphs and Sentences
- Consider if a clause should become a new sentence
---
+
# Avoid Jargon
- Sometimes the complex word isn't necessary
- If you must use jargon, define it
---
+
# Use Active Voice
-- "The lake was polluted by the company" ---> "The company polluted the lake"
+- <q>The lake was polluted by the company</q> → <q>The company polluted the lake</q>
- A good check to see if you're using passive voice: are you using
- - A form of "to be" (e.g., am, is, are, was, be, being, been)
- - A verb ending in "-ed"
+ - A form of <q>to be</q> (e.g., am, is, are, was, be, being, been)
+ - A verb ending in <q>-ed</q>
---
+
# Use Present Tense
-- Don't: "These sections describe types of information that would satisfy the application requirements of Circular A-110 as it would apply to this grant program."
-- Do: "These sections tell you how to meet the requirements of Circular A-110 for this grant program."
+- Don't: <q>These sections describe types of information that would satisfy the application requirements of Circular A-110 as it would apply to this grant program.</q>
+- Do: <q>These sections tell you how to meet the requirements of Circular A-110 for this grant program.</q>
---
+
# Use Examples
- Especially for more complicated topics
---
+
# Use Transition Words
-- Use words: "for instance," "also," "however," "to summarize"
+- Use words: <q>for instance,</q> <q>also,</q> <q>however,</q> <q>to summarize</q>
- Echo previous ideas
-- Avoid ambiguous "this"
+- Avoid ambiguous <q>this</q>
---
+
# Use Headers
- Preferably, use headers that are built into platforms (e.g., Google Docs, Microsoft Word, EdStem)
- If those are not available, styling text in bold or underlined can still help visual readers
- Live Demo!
---
+
# Plain Language Resources
-- If you have any questions or want examples about any of these concepts, you can look at the [PlainLanguage.gov website](https://www.plainlanguage.gov/guidelines/).
\ No newline at end of file
+- If you have any questions or want examples about any of these concepts, you can look at the [PlainLanguage.gov website](https://www.plainlanguage.gov/guidelines/).
+