Merged and organized all materials for week 3 and before

assignments/website.md

@@ -18,7 +18,7 @@ revised: December 19, 2022
 The goal of this homework is to learn about basics of website accessibility and how to assess; Learn how to use automated tools (and their limitations); Learn how to address the limitations of automated tools using accessibility tools; Learn how to write up an assessment and prioritize whac problems to fix; Practice identifying paths to fixing problems. You will also have the opportunity to practice UI image description. 
 
 ## 0. Pick a website and/or app and two tasks 
-You may pick your website/app from listing below or ask for instructor approval for one of your own. Note that there is a *task* associated with each website/app (and if you pick your own, you should also have a task in mind). 
+You may submit your top three choices for website/app from listing below and include at most one of your own on the [Ed Discussion thread for selection](https://edstem.org/us/courses/31170/discussion/2351217) for this assignment. Our goal is to have *at least four students** working independently on each website. Note that there is a *task* associated with each website/app (and if you pick your own, you should also have a task in mind).  
 
 {% details Possible websites/apps %}
 {% enddetails %}
@@ -77,9 +77,9 @@ For example, consider this view of the the WebAIM automated accessibility checke
 - **Relationship to other problems** Since we only have one example UAR here we don't have much to say, but this could mention for example other areas of the website which are missing labels such as form entries or buttons. 
 
 # Turnin
-You will turn in your raw notes on the problems you found 
+You will turn in your UARs on the problems you found **in one week** on the Ed discussion thread for your website (To be posted)
 
-You will also write up 5 page report documenting problems and suggesting solutions. Your report should be accessible (including proper use of headings, ALT text, table markup and so on) and have the following structure:
+In two weeks you will write up 5 page report documenting problems **that you and others found** on your website and suggesting solutions. Your report should be accessible (including proper use of headings, ALT text, table markup and so on) and have the following structure:
 - The first page should introduce the site or app, its purpose, and the task you assessed and state which tools you used in your assessment. 
 - The first page should also have an executive summary of the biggest (most frequent, severe) problems, and your recommendations for fixing them. Keep this brief, you will provide more detail in the following pages.
 - The next section of the report should provide an overview, and detail, on the problems found. 
@@ -96,8 +96,6 @@ You will also write up 5 page report documenting problems and suggesting solutio
 	- provide details if there are any special cases
 	- list (briefly) all the other places it happens
 - Finally discuss the remedy that is needed
-- Compiled your UARS in final report’s appendix 
-
 
 # Competencies
 This homework will contribute to your competency grade on 

schedule.md

@@ -59,6 +59,10 @@ Introduction to course syllabus & assignments
 - Accessibility Testing: Read [The importance of Manual Accessibility Testing](https://www.smashingmagazine.com/2018/09/importance-manual-accessibility-testing/) and [**Respond on Ed**](https://edstem.org/us/courses/31170/discussion/2345222)
 - ALT text: Read about [image alt text](https://webaim.org/techniques/alttext/) and watch [It's complicated](https://www.youtube.com/watch?v=By6nzFefcxs) and [**Respond on Ed**](https://edstem.org/us/courses/31170/discussion/2345223)
 - Audio Description: Read [What is audio description?](https://adp.acb.org/articles/adprocess.html) from [The Audio Description Project](https://www.acb.org/adp/ad.html) and [**Respond on Ed**](https://edstem.org/us/courses/31170/discussion/2345225)
+
+**Optional**
+- [Switches](https://axesslab.com/switches/)
+
 {% enddetails %}
 
 {: .week}
@@ -120,19 +124,21 @@ Try to find something you didn't find with an automated tool. ([Post a UAR on Ed
 {% details Preparation for next week  %}
 **Required: Respond to the Reading Questions and Preparation Requirements on Ed**
 - Read: [Comparison of different methods for accessibility testing](https://sol.sbc.org.br/journals/index.php/jis/article/view/1778/1848) Mateus etal, 2021  and [**Respond on Ed**](https://edstem.org/us/courses/31170/discussion/TBD) 
+- Read: About how [web semantics are conveyed to screen readers](https://alistapart.com/article/semantics-to-screen-readers/)   and [**Respond on Ed**](https://edstem.org/us/courses/31170/discussion/TBD) 
 
 **If you want to dig deeper**
 - Watch: [Google Video on Practical Web Accessibility](https://www.youtube.com/watch?v=x18vEEfpK3g) — this video provides a great overview of the Web and how to make web content accessible. Highly recommended as a supplement to what we will cover in class.
 - Watch: [Latte: Use-Case and Assistive-Service Driven Automated Accessibility Testing Framework for Android](https://www.youtube.com/watch?v=Z84jeO0UZFE) and read [Lies, Damned Lies, Overlays, and Widgets](https://www.linkedin.com/pulse/lies-damned-overlays-widgets-timothy-springer/)
+- Read: [Semantics for Eye tracking](https://dl.acm.org/doi/pdf/10.1145/3338844)
+- Watch: [Rescribe: Authoring and Automatically Editing Audio Descriptions](https://www.youtube.com/watch?v=jmyPmT5T0PY)) 
 
 {% enddetails %}
 
 {: .week}
 # Week 3 (1/17): Accessibility Assessment 2: Tradeoffs and Advanced Methods
 {% details Learning Goals %} 
-- What are accessibility standards are missing?
-- What are the limitations of automated tools? 
 - What are the tradeoffs between different tools?
+- Beginning to look at study design by example
 - More advanced accessibility techniques
     - Differences between Image description, Diagram description and UI description
     - Video description & Captioning
@@ -142,21 +148,31 @@ Try to find something you didn't find with an automated tool. ([Post a UAR on Ed
 
 {% details Class Plan %} 
 
-<!-- {: .activity} Skill Goal: Video description
-: **In class Exercise** Break into small groups. Pick a video from [youdescribe.org](https://youdescribe.org/) and work on a description as a group. ([post on Ed](https://edstem.org/us/courses/31170/discussion/2350670))
--->
+{: .topic}  5:30-6:00 Comparison of Accessibility Assesssment Approaches
+: **Slides** {% include slide.html title="Assessing Accessibility: Comparing Techniques" loc="assessment2.html" %}
+
+{: .activity} Discussion: Value of Cumulative Assessment
+: **In class Exercise**  Find the students who assessed the same website as you. Talk about the UARs you generated. Did you find different problems from other students? Why do you think you did, or did not, find different problems? Post your discussion notes to the Ed Thread for your website.
+
+{: .break}  6:20-6:40 
+: Break / Food 
 
-<!-- **Small Group Discussion**
-- Where did each approach (automated/trying things) fall short?
-- Power & downsides of "simulation" 
+{: .topic}  6:40-7:40 Best Practices for Improving Accessibility
+: **Slides** {% include slide.html title="Best Practices: Improving Accessibility" loc="best-practices.html" %}
 
+{: .activity} Skill Goal: Video description
+: **In class Exercise** Break into small groups. Pick a video from [youdescribe.org](https://youdescribe.org/) and work on a description as a group. ([post a link to your video on Ed](https://edstem.org/us/courses/31170/discussion/2350670))
 
-{: .activity} Skill Goal: Fixing Accessibility Issues 
-: **Slides** {% include slide.html title="Making Media Accessible" loc="alt-text/media.html" %}
 
-{: .topic} 7:50-8:20 Video Description 
-: Continuation of work on media accessibility
--->
+{: .break} 7:40-7:50
+: Break 
+
+{: .topic}  7:50-8:20 Best Practices for Improving Accessibility Part 2
+: **Slides** {% include slide.html title="Best Practices: Improving Accessibility" loc="best-practices.html#40" %}
+
+{: .activity} In Class Work Time
+: Finish your video description and post it. If done early, work on your website writeup.
+
 
 {% enddetails %}
 
@@ -166,7 +182,7 @@ Try to find something you didn't find with an automated tool. ([Post a UAR on Ed
 {% details Preparation for next week  %}
 **Required: Respond to the Reading Questions and Preparation Requirements on Ed**
 
-- [Design, Disability and Knowing the 'Other'](https://dl.acm.org/doi/fullHtml/10.1145/3290605.3300528#:~:text=WHAT%20IS%20EMPATHY%3F-,The%20Promise%20of%20Empathy%3A%20Design%2C%20Disability,%2C%20and%20Knowing%20the%20'Other'&text=This%20paper%20examines%20the%20promise,order%20to%20inform%20technology%20development.)
+-  [Design, Disability and Knowing the 'Other'](https://dl.acm.org/doi/fullHtml/10.1145/3290605.3300528#:~:text=WHAT%20IS%20EMPATHY%3F-,The%20Promise%20of%20Empathy%3A%20Design%2C%20Disability,%2C%20and%20Knowing%20the%20'Other'&text=This%20paper%20examines%20the%20promise,order%20to%20inform%20technology%20development.) 
 - [Anticipate and Adjust](https://dl.acm.org/doi/abs/10.1145/3491102.3501882)
 - [Disability Dongles](https://blog.castac.org/2022/04/disability-dongle/) by Liz Jackson
 - [Alexa & Accessibility](https://dl.acm.org/doi/pdf/10.1145/3173574.3174033?casa_token=T1I2RwihIjsAAAAA:QEm3SjurdlcW7oX_1LadxaglZ7oneBX-XLIjMCHbof3gu_IFpDW2OO5tqxZfLIps-94Qik9y5wNw8Q)
@@ -227,6 +243,7 @@ Presentations: Accessibility Assessment Assignment
 {: .homework} [Finding Accessibility](assignments/finding-accessibility.html) Assigned: Summarize a first person account of an access technology
 : 
 
+{: .homework} [Class Participation survey](https://forms.gle/qcjc5fSNSAVeE5pK9)
 {: .homework} Post-module Reflection:  [Accessibility Basics Reflection](https://forms.gle/Wk8C51m5Sbq8RX7W8)
 : 
 
@@ -261,7 +278,7 @@ Presentations: Finding Accessibility Assignment
 {% enddetails %}
 
 
-
+{: .homework} [Class Participation survey](https://forms.gle/qcjc5fSNSAVeE5pK9)
 {: .homework} Readings for next week 
 : 
 
@@ -275,6 +292,7 @@ Presentations: Finding Accessibility Assignment
 {% enddetails %}
 
 
+{: .homework} [Class Participation survey](https://forms.gle/qcjc5fSNSAVeE5pK9)
 {: .homework} Readings for next week 
 : 
 
@@ -288,6 +306,7 @@ Presentations: Finding Accessibility Assignment
 {% enddetails %}
 
 
+{: .homework} [Class Participation survey](https://forms.gle/qcjc5fSNSAVeE5pK9)
 {: .homework} Post-module Reflection
 : 
 
@@ -321,6 +340,7 @@ Presentations: Finding Accessibility Assignment
 {% enddetails %}
 
 
+{: .homework} [Class Participation survey](https://forms.gle/qcjc5fSNSAVeE5pK9)
 {: .homework} Readings for next week 
 : 
 
@@ -340,6 +360,7 @@ Presentations: Finding Accessibility Assignment
 {% enddetails %}
 
 
+{: .homework} [Class Participation survey](https://forms.gle/qcjc5fSNSAVeE5pK9)
 {: .homework} Post-module Reflection
 : 
 
@@ -352,12 +373,3 @@ Presentations: Finding Accessibility Assignment
 {: .lecture} Final project presentations
 : 
 
-
- 
-# Graveyard
-
-- [Semantics to screen readers](https://alistapart.com/article/semantics-to-screen-readers/)
-- [Access Lab - the Switch](https://axesslab.com/switches/)
-- [Improving User Experience of Eye Tracking-Based Interfaces](https://dl.acm.org/doi/pdf/10.1145/3338844)
-- [Rescribe: Authoring and Automatically Editing Audio Descriptions](https://amypavel.com/papers/rescribe.pdf) ([talk](https://www.youtube.com/watch?v=jmyPmT5T0PY)) 
-- [Rethinking web accessibility on Windows](https://www.marcozehe.de/rethinking-web-accessibility-on-windows/)

slides/assessment.html

@@ -55,9 +55,35 @@ Understandable: Information and the operation of user interface must be understa
 
 Robust: Content must be robust enough that it can be interpreted by a wide variety of user agents, including assistive technologies
 
+This is appropriate for *all* disabilities -- don't think access is only an issue for blind and low vision (BLV) people 
+
 ???
 Obviously there is some overlap between these, and they build on each other
 
+---
+# Example Disabilities To Consider
+
+.left-column50[
+Web browsing involves 
+- Color
+- Text of different shapes and sizes 
+- Content (images, words, sounds, animat.)
+- Typing (e.g. input to forms)
+- Mousing (e.g. clicking on links for navigation)
+- Comprehension (e.g. reading level)
+]
+.right-column50[
+Therefore…
+- Visual impairments
+- Difficulty understanding content
+- Difficulty hearing
+- Difficulty manipulating mouse our keyboard
+
+all affect accessibility
+
+]
+
+
 ---
 # What is the goal?
 End users interact with the *same* website or app as anyone else
@@ -258,6 +284,30 @@ Although technology is getting better at recognizing what an image depicts, algo
 - Context matters, as covered in depth by [WebAim](https://webaim.org/) article on [ALT text](techniques/alttext/)
 - Pay attention to identity of people. [Bennett et al](https://dl.acm.org/doi/10.1145/3411764.3445498) interviewed BIPOC, Non-Binary and/or Transgender Blind people. 
 
+---
+# Context
+
+.left-column[
+![:img Lezlie looks at us full front with curls down one size of her head and her long sleeve rolled up just on that side of a cowl neck wild abstract print shirt. Her smiling cheeks greet us., 100%, width](img/assessment/award.jpeg)
+]
+.right-column[
+The Society for Disability Studies is pleased to announce that Dr. Lezlie Frye has won the Irving K. Zola Award for Emerging Scholars in Disability Studies for her paper:
+
+Cripping the “Crack Baby” Epidemic: A Feminist Genealogy.
+
+Dr. Frye is an Assistant Professor of Gender Studies and Disability Studies at the University of Utah in the Division of Gender Studies.
+]
+
+---
+# ALT text 
+
+.left-column[
+![:img Lezlie looks at us full front with curls down one size of her head and her long sleeve rolled up just on that side of a cowl neck wild abstract print shirt. Her smiling cheeks greet us., 100%, width](img/assessment/award.jpeg)
+]
+.right-column[
+"Lezlie, a white woman, looks at us full front with curls down one size of her head and her long sleeve rolled up just on that side of a cowl neck wild abstract print shirt. Her smiling cheeks greet us."
+]
+
 ---
 # ALT Text with Identity Markers?
 
@@ -365,12 +415,29 @@ Distinguishable
 
 - Use of color is not the only way information is conveyed
 - Don't force screen reader users to listen to  audio longer than 3s without providing easily found controls; ideally use low or no background audio
-- Make sure you meet contrast expectations (minimum 4.5:1, ideally 7:1) in color (e.g. text and background; diagrams; controls; etc)
 - Support text resizing (& therefore don't use images of text). Ideally (AAA) also support selection of color; justification, spacing, etc
 - Support users who need a 1 column view of content (AAA)
 - If you must have tooltips and popups, make sure they are: dismissable; hoverable; and persistent
+- Make sure you meet contrast expectations (minimum 4.5:1, ideally 7:1) in color (e.g. text and background; diagrams; controls; etc)
+]
+
+---
+# More on Color contrast
+
+Choose colors that provide enough contrast between content and the background so that anyone with low-vision impairments and color deficiencies can perceive the content.
+
+.left-column50[
+WCAG Level AAA requires a contrast ratio of at least
+- .contrast71[7:1 for normal text] 
+- .contrast41[4.5:1 for large text (14t pt bold or larger)] 
+- .badcontrast[Avoid anything else!]
+]
+.right-column50[
+- [Colorzilla](https://chrome.google.com/webstore/detail/colorzilla/bhlhnicpbhignbdhedgjhgdocnmhomnp?hl=en) is an excellent tool for extracting the color value from any page element; 
+- WebAIM has a [contrast checker](https://webaim.org/resources/contrastchecker/#:~:text=WCAG%20Level%20AAA%20requires%20a,value%20from%20any%20page%20element)
 ]
 
+
 ---
 # POUR: Operable: Guidelines 2.1-2.5
 
@@ -434,6 +501,8 @@ Some common pitfalls:
 - Keyboard Traps. Content should not "trap" keyboard focus within subsections of content on a Web page. This is a common problem when multiple formats are combined within a page and rendered using plug-ins or embedded applications.
 - Invisible Content. Some parts of a web page can never be reached
 - Lack of Control. Users should be able to reconfigure or remove shortcuts
+
+Note: Not in guidelines (that I know of) but a "reverse trap" is whether you can reach text that *doesn't* have links or headers when using switch input. How would you do this?
 ]
 
 ???
@@ -585,6 +654,26 @@ Pointers, just like Keyboards, should be able to access everything
 - Allow users to use multiple possible types of input (keyboard, pointer, on-screen keyboard, stylus, etc)
 ]
 
+---
+# More on Size
+
+Especially hard on mobile devices
+
+.left-column40[
+![:img An arrow vs arrow and label make very different target sizes ,90%, width](img/assessment/targets.png) 
+]
+
+.right-column60[
+Even if the user misses the Text Label on the screen, they will still be able to trigger the desired action because the touch target is larger than what appears, resulting in less user error.
+
+White space around targets also helps
+
+Minimum on mobile: 48x48
+]
+???
+Solve for one, extend to many
+Trying to hit a small button with one hand while standing on a moving, crowded bus
+
 ---
 [//]: # (Outline Slide)
 # Learning Goals for Today
@@ -869,13 +958,6 @@ Simulation (Try it yourself using accessibility technologies or simulators such
 
 User Testing
 ]
----
-# Design Guidelines
-
-We just went through the WCAG guidelines
-
-Being familiar with these is a precursor to any other approach working
-
 
 ---
 # Simulation

slides/assessment2.html

+---
+layout: presentation
+title: Assessment 2  --Week 3--
+description: Comparison of Assessment Approaches
+class: middle, center, inverse
+---
+background-image: url(img/people.png)
+
+.left-column50[
+# Week 3: More Assessment!
+
+{{site.classnum}}, {{site.quarter}}
+]
+---
+name: normal
+layout: true
+class:
+
+---
+# Important Reminder
+
+## This is an important reminder
+## Make sure zoom is running and recording!!!
+
+---
+[//]: # (Outline Slide)
+# Learning Goals for Today
+
+- What are tradeoffs between assessment techniques
+- Example of a carefully designed empirical study
+
+---
+# Many Possible Techniques for Assessment
+
+Learning design guidelines (baseline for any other technique)
+
+Automated Assessment
+
+Simulation/Using accessibility Tools
+
+User Studies (focus of next week)
+
+---
+# Which one Should You Use?
+
+What technique will find the most user accessibility problems?
+
+What technique will find the most guideline violations?
+
+What techniques could someone without accessibility training successfully use?
+
+How do you trade off cost & value?
+
+How do you get a system to the point where user testing is worth doing?
+
+---
+# Study to answer these questions
+
+Gather baseline problem data on 4 sites
+- Usability study
+
+Test same sites with other techniques
+- Expert review with guidelines
+- Screen reader
+- Automated tool
+- Remote
+
+Compare problems found to baseline study
+
+---
+# Baseline Study
+5 blind JAWS users 
+- 2 to 6+ years of experience 
+- Used web for e-mail, shopping, information, and news
+
+4 tasks (each on a different site)
+- Range of difficulty
+- Representative activities: bus trip planner; find names on a page; register for a class; grocery item search
+
+---
+# Think Aloud Protocol
+
+Focused analysis on *accessibility* problems (not *usability* or *technology* problems)
+- [+] “Pop up. Had to ask for sighted help to continue”
+- [+] “Date entered in incorrect format” 
+- [-] “Their slogan is annoying”
+- [-] “Forgot JAWS command for forms mode”
+Ranked problems by severity, found 3-10 per site
+
+Grouped like problems (e.g. all encounters with the pop-up window)
+
+???
+AT the time, WCAG 1; Meeting WCAG priority 1 guidelines did not address all severe problems
+
+---
+# Results -- Grocery
+
+.left-column50[
+![:img Picture of the front page of the albertson's website for ordering groceries online,100%, width](img/assessment/grocery.png) 
+]
+
+.right-column50[
+![:img Picture of the grocery cart for the the albertson's website,100%, width](img/assessment/grocery2.png) 
+]
+---
+# Results -- Grocery
+
+.left-column50[
+Special, accessible website
+
+Easiest site
+
+7 accessibility problems
+
+- Redundant text
+
+- Problems with forms
+…
+]
+
+.right-column50[
+![:img Picture of the grocery cart for the the albertson's website,100%, width](img/assessment/grocery2.png) 
+]
+---
+# Results -- Find Names
+
+.left-column50[
+2 simple HTML pages
+
+3 accessibility problems
+- Front page: Use of headers confusing
+- Front page lacked clear description of contents/purpose
+- Name page: too many links
+]
+
+.right-column50[
+![:img Picture of the list of graduate students in Berkeley's HCI group GUIR at the time of the study,100%, width](img/assessment/findnames.png) 
+]
+
+---
+# Results -- Bus
+
+.left-column50[
+Most difficult site
+
+10 accessibility problems
+- Lack of ALT text
+- Bad form labeling
+- Non-descriptive link names
+- Lack of directions/ description of purpose
+]
+
+.right-column50[
+![:img Picture of the public transportation website for using  busses in Berkeley at the time of the study,100%, width](img/assessment/bus.png) 
+]
+
+---
+# Results -- Bus
+
+.left-column50[
+
+1 page 
+
+9 accessibility problems
+- Required field marked only with “bold” text
+- No directions about syntax requirements
+- Fields not well labeled
+…
+]
+
+.right-column50[
+![:img Picture of a simple fake class registration form we made for the study,100%, width](img/assessment/registration.png) 
+]
+
+---
+# Comparative Study
+
+Piloted techniques to be compared
+
+Gathered data on success of each technique for same tasks as baseline
+
+Analyzed results
+
+---
+# Comparative Study
+
+Four conditions
+- Expert Review (web experts)
+- Screen Reader (web experts w/ screen reader)
+- Automated (Bobby)
+- Remote (expert, remote blind users)
+
+Four tasks/sites (same as baseline)
+- 18 current web developers (2-8 yrs exp)
+- Never developed an accessible website
+- Three had taken a class that covered some issues
+- 9 remote, experienced JAWS users
+
+---
+# Web Developers
+
+All web developers introduced to WCAG Priority 1 guidelines (best available at the time)
+
+8 web developers given a Screen Reader tutorial & practice time (~15mins)
+
+Instructed to only review task pages
+
+Think aloud method
+
+---
+# Remote, BLV users
+
+Given parallel instructions 
+
+Asked to email problems back to us
+
+---
+# Analysis
+
+Coded problems: WCAG & Empirical (baseline)
+
+Hypothesis 1: No technique will be more effective than any other
+
+- Thoroughness (portion of actual problems)
+- Validity (false positives)
+
+Hypothesis 2: The types of accessibility problems found by each technique will be the same
+
+- Categories of problems found by each technique
+
+---
+# General Results
+
+1-2 hours to complete
+
+No correlation between developer severity and WCAG priority or empirical severity
+
+5 of 9 remote participants failed to complete 1 of the 4 tasks
+
+---
+# Thoroughness, Validity and Number
+
+.left-column[
+Thoroughness: portion of actual problems
+
+Validity: false positives
+]
+.right-column[
+![:img Graph showing mean problems found as boxplots for remote; expert review; and screen reader groups. One boxplot per website. Following slides will highlight results in this chart,70%, width](img/assessment/graph1.png) ]
+
+???
+
+Note high variance
+
+---
+# H1: Methods Don't Differ 
+.left-column[FALSE for Number:]
+.right-column[
+![:img same graph highlighting that the average reviewer only found less than 20% of problems,70%, width](img/assessment/graph2.png) 
+]
+???
+
+Note small differences between individual developers in finding problems
+
+Difference between remote and screen reader group is significant
+
+---
+# H1: Methods Don't Differ 
+
+.left-column[Few false positives]
+.right-column[
+
+![:img same graph highlighting that the average validity is 60% for remote BLV users; 20% for expert reviewers; and 40% for non-BLV screen reader users,60%, width](img/assessment/validity.png) 
+]
+???
+
+Note that these differences are significant across groups
+But there is no effect for individual evaluators
+
+---
+# H2: Techniques find Different Problems
+
+.left-column60[
+
+WCAG 1 
+
+|               | G1 | G2 | G3 | G4 | G5 | G9 | G10 | G12 | G13 | G14 |
+|:--------------|:---|:---|:---|:---|:---|:---|:----|:----|:----|:----|
+| Expert Review | Y  | Y  | Y  |    |    | Y  | Y   | Y   | Y   |     |
+| Screen Reader | Y  | Y  | Y  | Y  |    | Y  | Y   | Y   | Y   | Y   |
+| Remote        | Y  | Y  |    |    | Y  | Y  | Y   | Y   | Y   | Y   |
+| Automated     | Y  | Y  |    |    | Y  |    |     |     | Y   |     |
+
+]
+
+.right-column40[
+- G1,2: AV alternatives & not just color
+- G3,4: Good markp and clear language
+- G5: Tables
+- G9,10: Device independence & interem solutions for things like javascript
+- G12,13: Context & Navigation
+- G14; clear and simple text
+]
+
+---
+# H2: Techniques find Different Problems
+
+.left-column60[
+
+Empirical
+
+|               | E1 | E2 | E3 | E4 | E5 | E6 | E7 | E8 | E9 |
+|:--------------|:---|:---|:---|:---|:---|:---|:---|:---|:---|
+| Expert Review | Y  | Y  | Y  | Y  |    | Y  | Y  | Y  | Y  |
+| Screen Reader | Y  |    | Y  | Y  | Y  | Y  | Y  | Y  | Y  |
+| Remote        | Y  |    | Y  | Y  |    | Y  | Y  |    | Y  |
+| Automated     | Y  |    |    |    |    |    |    |    |    |
+
+]
+.right-column40[
+- E1: No alt text
+- E2: Poor defaults
+- E3: Poor formatting
+- E4: Too much data
+- E6: No directions
+- E7: Visual pairing
+- E8: Poor names
+- E9: Popups
+]
+
+---
+# Aside: WCAG has come a long way
+
+Many (perhaps all) of these are part of guidelines now
+
+- E1: Should have no alt text
+- E2: Poor defaults
+- E3: Poor formatting
+- E4: Too much data
+- E6: No directions
+- E7: Visual pairing
+- E8: Poor names
+- E9: Popups
+
+---
+# H2: Techniques find Different Types of Problems
+
+Screen reader novices did best at both major types of problems
+
+![:img Four barcharts each showing the cumulative benefit in terms of percentage of known problems found of adding evaluators. The first barchart shows expert reviewers; who reach 30% of WCAG problems and 60% of empirical problems by the third evaluator. The second barchart shows novice screen reader users who reach 60% of both emirical and WCAG problems by the fifth evaluator. The third barchart shows remote BLV participants who reach 25% of WCAG and just under 20% of empirical problems by the fifth evaluator. The fourth barchart shows an automated tool which finds about 25% of WCAG and 5% of empirical problems,100%, width](img/assessment/cumulative.png) 
+
+???
+Explain chart
+also tracks heuristic eval literature: Five Evaluators find ~50% of Problems
+Individuals don't do well, but they  *differ*  from each other
+
+---
+# Discussion
+
+Hyp 1: Screen reader most consistently effective
+
+Hyp 2: All but automated comparable
+
+- Screen missed only tables (w3); poor defaults (empirical)
+
+Really need multiple evaluators
+
+Remote technique needs improvement, could fare better
+
+Accessibility experience would probably change results
+
+---
+# Discussion
+
+Asymptotic testing needed
+- Can’t be sure we found all empirical problems
+
+Falsification testing needed
+- Are problems not in empirical data set really false positives?
+
+More consistent problem reporting & comparison beneficial
+
+Limitations
+- Web only
+- Design of remote test limited result quality
+- Very old study
+
+---
+# Small Group Discussion
+
+Find the students who assessed the same website as you.
+
+Talk about the UARs you generated. Did you find different problems
+from other students? Why do you think you did, or did not, find different problems?
+
+Post your discussion notes to the Ed Thread for your website.
+
+---
+[//]: # (Outline Slide)
+# Learning Goals for Today
+
+- What are tradeoffs between assessment techniques
+- Example of a carefully designed empirical study
+

slides/evaluation.html

+---
+layout: presentation
+title: Evaluation  --Week 4--
+description: Accessibility Needfinding and Evaluation
+class: middle, center, inverse
+---
+background-image: url(img/people.png)
+
+.left-column50[
+# Week 3: More Assessment!
+
+{{site.classnum}}, {{site.quarter}}
+]
+---
+name: normal
+layout: true
+class:
+
+---
+# Important Reminder
+
+## This is an important reminder
+## Make sure zoom is running and recording!!!
+
+---
+[//]: # (Outline Slide)
+# Learning Goals for Today
+
+- 
+
+---
+# Summative User Testing
+
+Participatory Design & Research
+
+Lab Testing (often must take place in the home). Best guidance: [Nielsen-Norman Group](http://www.nngroup.com/reports/accessibility/testing/); [Anticipate and Adjust](https://a11ykelly.medium.com/anticipate-and-adjust-cultivating-access-in-human-centered-methods-1e46c6845e34)
+
+---
+# Summative User Testing
+
+Little details matter
+- How will you find participants?
+- How much do you know about the group your are targeting and what to expect from/of them?
+- How will you communicate? 
+- Can you duplicate the custom hardware/software they use, or must you go to them?
+- Do they have hardware you need (e.g. working monitor?)
+
+---
+# Summative User Testing 
+Best Practices
+
+- Eliminate obvious accessibility & usability problems ahead of time
+- Consider a remote study, or go to their home (familiar spaces & tech)
+- Don’t be overly sensitive (don’t overcompensate)
+- Don’t rely on useless cues (audio/visual) to convey encouragement
+- Monitor participant fatigue carefully
+
+---
+# Issues
+Heterogeneous Users
+
+Accountability
+- What responsibility do you have to participants in terms of providing skills and help in & out of study?
+- What responsibility do you have to give participants a participatory role in design?
+- How do you ensure the integrity of your record of their experience?
+
+Expense 
+
+Scalability 
+
+Hard to do with early prototypes
+
+---
+# Possible Metrics
+Click errors
+
+Success criteria & ratings
+
+Satisfaction
+
+Impact when users fail to complete tasks 
+- Help them? 
+- Quantitative study impact
+- Qualitative study impact