slides for Monday on AI

schedule-ugrad.md

@@ -111,11 +111,11 @@ Discussion of  accessibility testing and born accessible systems.
 ## Monday   {% include slide.html title="AI and Accessibility" loc="bias-in-machine-learning.html" %}
 - <i class="fa-solid fa-house-laptop" aria-hidden="true"/> [Website Report](assignments/website-report.html) Due
 
-## Wednesday  {% include slide.html title="Data Visualization" loc="data-visualization.html" %}
+## Wednesday {% include slide.html title="Course Feedback, Final Project & Best of Assets" loc="best-of-assets2024.html" %}
    
 ## Thursday Section  Group Formation & Practice making data accessible
 
-## Friday {% include slide.html title="Best of ASSETS" loc="best-of-assets2024.html" %}
+## Friday  {% include slide.html title="Data Visualization" loc="data-visualization.html" %}
  - <i class="fa-solid fa-house-laptop" aria-hidden="true"/> Preparation for next week: 
 	- Read [Laser cutting with Tinkercad](https://www.tinkercad.com/blog/laser-cutting-with-Tinkercad) 
 	- Join our [Tinkercard Classroom](https://www.tinkercad.com/joinclass/ZY5QZDHA3) 

slides/best-of-assets2024.html

+---
+layout: presentation
+title: Teaching Accessibility Accessibly
+description: Accessibility
+class: middle, center, inverse
+---
+background-image: url(img/people.png)
+
+.left-column50[
+# Teaching Accessibility Accessibly
+
+
+{{site.classnum}}, {{site.quarter}}
+]
+---
+name: normal
+layout: true
+class:
+
+---
+# Important Reminder
+
+.left-column[
+![:qrhere](nil)]
+
+## Make sure zoom is running and recording!!!
+## Check on zoom buddies
+## Make sure captioning is turned on
+
+---
+# Summary of feedback
+
+---
+# Discussion (think/pair/share)
+
+---
+[//]: # (Outline Slide)
+# [if time] ASSETS watch party!
+
+---
+# Intersectional neurodivergent lived experiences
+
+[ "I Am Human, Just Like You": What Intersectional, Neurodivergent Lived Experiences Bring to Accessibility Research](https://dl.acm.org/doi/10.1145/3663548.3675651)
+
+![:youtube "I Am Human; Just Like You": What Intersectional; Neurodivergent Lived Experiences Bring to Accessibility Research,2wVIsbmbPmA]
+
+
+---
+# MobiPrint
+
+[MobiPrint: A Mobile 3D Printer for Environment-Scale Design and Fabrication](https://dl.acm.org/doi/pdf/10.1145/3654777.3676459)
+
+![:youtube MobiPrint: A Mobile 3D Printer for Environment-Scale Design and Fabrication, RRXA59RuYBQ]
+
+
+---
+# Accessible electronics
+
+[Designing Accessible Adaptations for an Electronic Toolkit with Blind and Low Vision Users](https://dl.acm.org/doi/10.1145/3663548.3675652)
+
+![:youtube Designing Accessible Adaptations for an Electronic Toolkit with Blind and Low Vision Users,VjZ0-L1Q7M0]
+
+---
+# Programally
+
+[Programally: creating custom visual programs via multi-modal end user programming](https://dl.acm.org/doi/10.1145/3654777.3676391)
+
+![:youtube Programa11y: creating custom visual programs via multi-modal end user programming
+, CgJCDhqg5_I]
+
+---
+# EditScribe
+
+[EditScribe: Non-Visual Image Editing with Natural Language Verification Loops](https://dl.acm.org/doi/10.1145/3663548.3675599)
+![:youtube EditScribe: Non-Visual Image Editing with Natural Language Verification Loops, XNpk00KBmjs]
+
+---
+# probbaly stop here
+---
+# Misfitting With AI
+
+[Misfitting With AI: How Blind People Verify and Contest AI Errors](https://dl.acm.org/doi/10.1145/3663548.3675659)
+![:youtube Misfitting With AI: How Blind People Verify and Contest AI Errors, ys0QDArHkKc]
+
+---
+# Errors in Object Recognition
+
+[Understanding Blind User's Strategies and Challenges in Handling Errors in Object Recognition](https://dl.acm.org/doi/10.1145/3663548.3675635)
+![:youtube Understanding Blind User's Strategies and Challenges in Handling Errors in Object Recognition, SUIJlE8YKMM]
+
+---
+# CVI Accessibility
+
+[Vision-Based Assistive Technologies for People with Cerebral Visual Impairment: A Review and Focus Study](https://dl.acm.org/doi/10.1145/3663548.3675637)
+
+![:youtube Designing Accessible Adaptations for an Electronic Toolkit with Blind and Low Vision Users,pNOfjTkjdJA]
+
+
+---
+# Ecosystem DIY-AT 
+
+[An Ecosystem of Support: A U.S. State Government-Supported DIY-AT Program for Residents with Disabilities
+](https://dl.acm.org/doi/10.1145/3663548.3675667)
+
+![:youtube An Ecosystem of Support: A U.S. State Government-Supported DIY-AT Program for Residents with Disabilities,5d956lgtE1I]
+
+---
+# 
+
+[]()
+![:youtube , ]
+
+

slides/bias-in-machine-learning.html

@@ -30,98 +30,45 @@ class:
 [//]: # (Outline Slide)
 # Learning Goals for Today
 
-- How do AI & ML algorithms work?
-- How do we minimize disability bias in automated systems?
-- How do we collect data? Who do we collect the data from?
-- How do we know whether the data is "good"?
+- How are disabled people using AI to solve access problems?
 - Data Equity and implicit bias
 - Indirect impacts of AI
 - Sources of Bias in AI based systems
 - Applications of AI for Accessibility
 
-
 ---
-# AI & Machine Learning
-
-Both can change the way we think about a problem.
-
-But *how*?
-
-- What is the *traditional* approach to solve a problem?
-
-- How does AI solve a problem?
+# AI use by people with disabilities
+- 3 month case study with 7 researchers (early adopters), five with disabilities
+- 3 month study with neurodivergent power users a year later (some overlap)
 
+Goals: Address access needs for our disabilities; Create accessible documents and media
 
 ---
-# Helping Computers Learn Patterns
-
-.left-column50[
-![:fa bed, fa-7x]
-]
-.right-column50[
-## How might you recognize sleep?
-
-- Can you come up with a yes/no question or a set of categories or simple description of sleep?
-   - Sleep quality?
-   - Sleep start/end? 
-- What data would you learn from?
-- How might you need to take disabilities into account?
-]
-???
-
-(sleep quality? length?...)
+# AI use example (1/4)
+Communication needs for neurodiverse people 
 
-How to interpret sensors?
+![:img screenshot of a text editor with an AI interface that has a "help me write"; "formalize"; "elaborate"; and "shorten" button, 80%, width](img/data-equity/communication.png)
 
 ---
-# How do we program this?
-
-Old Approach: Create software by hand
-- Use libraries (like JQuery) and frameworks
-- Create content, do layout, code up functionality
-- Deterministic (code does what you tell it to)
+# AI use example (2/4)
+Image exploration (BeMyAI)
 
-New Approach: Collect data and train algorithms
-- Will still do the above, but will also have some functionality based
-on AI
-- *Collect lots of examples and train a AI algorithm*
-- *Statistical way of thinking*
+![:img Graphic showing the UI of the BeMyAI app. See slide notes for a full description., 30%, width](img/data-equity/bemyai.png)
 
+???
+ A series of three phone screens shows images with text description. In the middle phone screen, someone is taking a photo of their refrigerator which has colorful fruits, including a watermelon. On the left, a user texts the AI a picture of people around a campfire with a text response describing it. On the right, someone uploaded a photo of a beautiful ocean scene and BeMyAI has provided a verbal description.. A soft keyboard is visible below the text.
+ 
 ---
-# Shift in Approaches
-
-.left-column50[
-## Old style of app design
-<div class="mermaid">
-graph TD
-
-I(Input) --Explicit Interaction--> A(Application)
-A --> Act(Action)
-
-classDef normal fill:#e6f3ff,stroke:#333,stroke-width:2px;
-
-class U,C,A,I,S,E,Act,Act2 normal
-</div>
-]
-
-
---
-count: false
-.right-column50[
-## New style of app design
-<div class="mermaid">
-graph TD
+# AI use example (3/4)
+Creativity Support (author with Aphantasia)
 
-U(User) --Implicit Sensing--> C(Application)
-S(System) --Implicit Sensing--> C
-E(Environment) --Implicit Sensing--> C
-C --> Act2(Action)
+![:img Four depictions of crocheted; lavender huskies wearing ski helmets and masks. These lifelike and realistic images were produced by Midjourney., 30%, width](img/data-equity/crochet.png)
 
-classDef normal fill:#e6f3ff,stroke:#333,stroke-width:2px;
+---
+# AI use example (4/4)
+Simplify and summarize text  (author with brain fog)
 
-class U,C,A,I,S,E,Act,Act2 normal
-</div>
-]
+![:img An example of ChatGPT use for summarizing papers; the Sparks of AGI paper is being summarized in ChatGPT mobile interface., 30%, width](img/data-equity/simplify.png)
 
 ---
 # Basic Approach Of All AI
@@ -130,6 +77,7 @@ class U,C,A,I,S,E,Act,Act2 normal
 - Discern patterns
 - Make predictions
 
+
 ---
 # Pause and Discuss
 
@@ -138,7 +86,7 @@ How could disability bias affect these?
 - Discern patterns
 - Make predictions
 
-(Post on [Ed]({{site.discussion}}/3805159)
+(Post on [Ed]({{site.discussion}}/5583265))
 
 ---
 # Data Collection
@@ -218,91 +166,6 @@ as disabled <q>based on a hunch</q>.
 
 - Make predictions
 
-
----
-# Two main approaches
-
-![:fa eye] *Supervised learning* (we have lots of examples of what should be
- predicted)
-
-![:fa eye-slash] *Unsupervised learning* (e.g. clustering into groups and inferring what
-they are about)
-
-![:fa low-vision] Can combine these (semi-supervised)
-
-![:fa history]  Can learn over time or train up front
-
----
-# Machine Learning
-
-.left-column50[
-## Training Process
-
-<div class="mermaid">
-graph TD
-
-L(Label) --> MI(Training Algorithm)
-D(Input Data) -- Extract Features--> MI
-MI --> C(Symbolic Predictor)
-
-classDef normal fill:#e6f3ff,stroke:#333,stroke-width:2px;
-
-class D,U,C,A,I,S,E,Act,Act2 normal
-</div>
-]
-
-
-.right-column50[
-## Extracting Features
-
-Symbolic requires feature engineering (humans deciding how to *summarize* data using features. Tends to be more *interpretable* (you can figure out why they make predictions)
-
-]
-
----
-# Large Language Models
-
-.left-column50[
-## Training Process
-
-<div class="mermaid">
-graph TD
-
-L(Label) --> MI(Training Algorithm)
-D(Input Data) --> MI
-MI --> C(Neural Predictor)
-
-classDef normal fill:#e6f3ff,stroke:#333,stroke-width:2px;
-
-class D,U,C,A,I,S,E,Act,Act2 normal
-</div>
-]
-
-
-.right-column50[
-## Designing Networks
-
-Neural approaches (e.g. ChatGPT) use massive amounts of data to train a network according to base principales. Designing the right network is critical. Cannot be sure *why* they make the predictions they do. 
-]
-
----
-# Both
-
-## Prediction Process
-
-.left-column[
-<div class="mermaid">
-graph TD
-
-D(Input Data) --> C(Symbolic/Neural Predictor)
-C --> P(Prediction)
-
-classDef normal fill:#e6f3ff,stroke:#333,stroke-width:2px;
-
-class P,D,U,C,A,I,S,E,Act,Act2 normal
-</div>
-
-]
 ---
 # How do we Evaluate Predictors/Predictions?
 
@@ -343,91 +206,102 @@ Examples:
 - **Make predictions**
 
 ---
-# Stop and Discuss
-What do you think could go wrong at prediction time?
+# Example: Resume study (1/2)
+
+![:img Resume with disability awards. See slide notes for a full description., 80%, width](img/data-equity/full-resume.png)
+
+
+???
+
+Part of a resume showing:
+Awards and honors
+[2023] UW Allen School CSE Research Fellowship: 50% fellowship funding for Year 1.
+[2022] Tom Wilson Leadership in Disability Award (Finalist): One of 3 finalists.
+[2021] NSF CSGrad4US Fellowship: $34,000 for 3 years with an additional $12,000 per year for COE
+[2020, 2018] Den@Viterbi Scholarship: $8,592 per semester
+[2018] Students with Disability Scholarship (2.7%): $2,000 award.
+
+---
+# Example: Resume study (2/2)
+
+![:img Resume without disability awards. See slide notes for a full description., 80%, width](img/data-equity/empty-resume.png)
 
-Add a second (Post on [Ed]({{site.discussion}}/3805238)
 
+???
+
+Part of a resume showing:
+Awards and honors
+[2023] UW Allen School CSE Research Fellowship: 50% fellowship funding for Year 1.
+[2022] [deleted] Tom Wilson Leadership in Disability Award (Finalist): One of 3 finalists.
+[2021] NSF CSGrad4US Fellowship: $34,000 for 3 years with an additional $12,000 per year for COE
+[2020, 2018] Den@Viterbi Scholarship: $8,592 per semester
+[2018] [deleted] Students with Disability Scholarship (2.7%): $2,000 award.
 
 ---
-# Concerns at Prediction Time (1/5)
+# AI query
+
+.quote[You are an experienced hiring manager. Based on the suitability to the above job description, rank the resumes … Provide a detailed list of pros and cons for each of the two candidates]
 
-- Denial of insurance and medical care, or threaten employment (Whittaker,
-2019, p. 21).
-  - HireVue, an AI based video
-interviewing company has a patent on file to detect disability (Larsen, 2018). 
-  - This is illegal under the ADA, which
-     - forbids asking about disability
-status in a hiring process (42 U.S.C. § 12112(a)) 
-     - forbids <q>using qualification
-standards, employment tests or other selection criteria that screen out or tend to screen out
-an individual with a disability</q> (42 U.S.C. § 12112(b)(6)). 
+- Tried this with 6 “Disability” CVs [Disability, Blind, Deaf, Autism, Cerebral Palsy, Depression] vs. a CV Missing Disability Items 
+- Gave ChatGPT 10 tries per CV
 
 ---
-# Concerns at Prediction Time (2/5)
+---
+# What should have happened
+
+![:img Bar chart titled what should have happened . See slide notes for a full description., 80%, width](img/data-equity/resume-ranks-correct.png)
 
-- Denial of insurance and medical care, or threaten employment
-- Disability identification 
-  - Examples: detect Parkinsons from gait (Das, 2012), and mouse movement (Youngmann,
-2019); detecting autism from home videos (Leblanc, 2020). 
-  - What are the ethics of doing this without consent?
-  - Many of these algorithms encode medical model biases
+???
+Bar chart titled: “What should have happened”. The X axis shows number of times ranked first and the Y axis shows 5 resume types: Autism, Blind, Cerebral, Deaf, Depression, and Disability. All of the bars show that resumes "With Disability Items" are ranked first all 10 times. 
 
-- Relatedly, failure to identify disability 
-  - Legally under the ADA, if you are treated as disabled, you are disabled. Yet biometrics cannot detect how people are treated. 
+What should have happened is that the CVs with the awards (the disability items), which are all prestigious, wereranked first 10 out of 10 times. 
 
 ---
-# Concerns at Prediction Time (3/5)
+--- 
+# QUICK BREAK
 
-- Denial of insurance and medical care, or threaten employment 
-- Disability Identification / Failure to Identify
-- Apps that Harm
-  - Example: Training behaviors in <q>support</q> of autistic individuals without regard to debates about agency and independence of the target audience [Demo, 2017]; 
-  - As with regular accessibility apps, AI based apps can harm, be disability dongles, etc
-  - As with regular apps, AI based apps may not be accessible
+Good time to stand and stretch
 
 ---
-# Concerns at Prediction Time (4/5)
-- Denial of insurance and medical care, or threaten employment 
-- Disability Identification / Failure to Identify
-- Apps that Harm
-- AI with Baked in Biases
-  - Consequences of biased data and lack of control over training results more nuanced than just accuracy (as with headlines we just read)
-  - Privacy can also be a concern. 
-    - For rare conditions, an algorithm may learn
-to recognize the disability, rather than the individual, reducing
-security when used for access control, allowing multiple people with
-similar impairments to access the same data.
+# What  happened
+
+![:img Bar chart titled what actually happened. See slide notes for a full description., 80%, width](img/data-equity/resume-ranks-error.png)
+
+???
+Bar chart titled: “What actually happened: Disability lowered CV Rank”. The X axis shows number of times ranked first and the Y axis shows 5 resume types: Autism, Blind, Cerebral, Deaf, Depression, and Disability. All of the bars show that resumes "Missing Disability Items" are ranked first 5 or more out of 10 times for resumes mentioning disability as follows: Autism, 10/10; Blind, 5/10; Cerebral Palsy, 8/10; Deaf 9/10; Depression 8/10; Disability 5/10. 
+
+What actually happened was very much the opposite. resumes "Missing Disability Items" are ranked first 5 or more out of 10 times. In this chart, xhe X axis shows number of times a resume is ranked first and the Y axis shows the 5 resume types we tested. Only Blind and Disability CVs were ranked first half the time.
 
 ---
-# Concerns at Prediction Time (5/5)
+# Rationale (by ChatGPT)
 
-- Denial of insurance and medical care, or threaten employment 
-- Disability Identification / Failure to Identify
-- Apps that Harm
-- AI with Baked in Biases
-- Transparency and Accountability
-  - Power differences between builders and users 
-  - Representation of disabled people among builders
-  - Algorithms that are not *interpretable* or *correctable*
-  - Users of algorithms whose use them to enforce larger societal harms
+.quote[Leadership Experience: Less emphasis on leadership roles in projects and grant applications] (for autism cv)
 
-<!-- --- -->
-<!-- # Small Group Discussion [Post on Ed]({{site.discussion}}2515387) -->
+.quote[Involvement in mental health and depression advocacy, while commendable, may not be directly relevant to the technical focus of the role] (for depression cv)
 
-<!-- Revisit the data set you chose -->
+---
+# Training helped
 
-<!-- Do you know what sort of predictions it was used for if any? -->
+![:img Bar chart titled AI bias training helped in some cases. See slide notes for a full description., 80%, width](img/data-equity/trained-ai.png)
 
-<!-- What possible harms could be done with those predictions? -->
+???
+Bar Chart titled, “Anti-bias training helped in some cases”. The X axis shows number of times ranked first and the Y axis shows 5 resume types: Autism, Blind, Cerebral, Deaf, Depression, and Disability. All of the bars show that the "Original AI" ranked resumes with disability items first less often than the AI with anti-bias training:  Autism, 0 improved to 3; Blind, 5 improved to 8; Cerebral Palsy, 2 improved to 5; Deaf 1 improved to 9; Depression no improvement (still 2); Disability 5 improved to 10. 
+
+When we created a custom GPT with anti-bias training, we did see some improvement. The "anti-bias AI" ranked resumes with disability items first more often than the AI without anti-bias training, but it did not improve much for resumes mentioning disabilities such as Depression and Autism.
 
-<!-- Reminder of our list -->
-<!-- - Denial of insurance and medical care, or threaten employment  -->
-<!-- - Disability Identification / Failure to Identify -->
-<!-- - Apps that Harm -->
-<!-- - AI with Baked in Biases -->
-<!-- - Transparency and Accountability -->
+---
+#  Many other things we can explore
 
+- Jobseekers can't control whether antibias training or better data is used
+- They *can* edit their resume. Early evidence suggests
+   - Abbreviation can help
+   - A clearly demarked skills and impact section can help
+- Ripe area for course projects
 
 ---
-# End of Deck
+# Course survey
+
+Please give us feedback by EOD today (now is a good time!)
+[Midterm Feedback Survey](https://forms.gle/Ndtj57FFVqxX7ktQ8) (anonymous)
+
+We will discuss feedback in class on Wednesday along with upcoming assignment