---
layout: presentation
title: Event Handling II--Essential Geometry and View Updates
description:
class: middle, center, inverse
---
layout: false
.title[# Hall of Fame/Shame]
.body[
]
---
name: inverse
layout: true
class: center, middle, inverse
---
# Essential Geometry and View Updates
Jennifer Mankoff
CSE 340 Winter 2020
---
layout:false
[//]: # (Outline Slide)
.title[Today's goals]
.body[
- Input dispatching process (toolkit architecture)
- Picking (alternative: Focus)
- Capture
- Bubble
Introduce how interactors process them
Discuss use of Propositional Production Systems in implementing interactors
Understand Essential Geometry
Try creating a PPS
Understand View Updates
]
---
template: inverse
## Input Dispatch Process
---
layout: false
# Input __Dispatch__ Process
Input thread:
- When a user interacts, __events__ are created
- Events go into a queue
---
# Input __Dispatch__ Process
Input Thread
Dispatch thread:
- Front event comes off queue
- How does a toolkit decide where to send events?
---
# How do we decide where to send events?
---
# Input Process - Picking
.left-column[
<div class="mermaid">
graph TD
V0((V0)) --> V1((V1))
V0 --> V4((V4))
V1 --> V2((V2))
V1 --> V3((V3))
V4 --> V5((V5))
V4 --> V6((V6))
</div>
]
.right-column[
- In what order does dispatch "pick" the `View` objects?
]
--
.right-column[
- Hint: Post order traversal of the tree
]
--
.right-column[
- Picked Views = {`V2`, `V3`, `V1`, `V5`, `V6`, `V4`, `V0`}
- Order matters!
]
---
# Input Process - Picking
.left-column[
<div class="mermaid">
graph TD
V0((V0)) --> V1((V1))
V0 --> V4((V4))
V1 --> V2((V2))
V1 --> V3((V3))
V4 --> V5((V5))
V4 --> V6((V6))
</div>
]
.right-column[
But we can refine this! We can filter!
]
???
What are some reasons to skip views?
- Not inside the view
- View doesn't care about that type of input
--
.right-column[
What are some reasons to skip views?
- Not inside the view
- View doesn't care about that type of input
]
---
# Input Process - Picking only those interested
.left-column[
<div class="mermaid">
graph TD
V0((V0)) --> V1((V1*))
V0 --> V4((V4*))
V1 --> V2((V2))
V1 --> V3((V3*))
V4 --> V5((V5))
V4 --> V6((V6*))
class V0,V2,V5 blue
class V1,V4,V3,V6 bluegreen
</div>
]
.footnote[*: denotes the element responds to the event because of position/type]
---
# Input Example - Capture
.left-column[
<div class="mermaid">
graph TD
V0((V0)) --> V1((V1*))
V0 --> V4((V4*))
V1 --> V2((V2))
V1 --> V3((V3*))
V4 --> V5((V5))
V4 --> V6((V6*))
class V0,V2,V5 blue
class V1,V4,V3,V6 bluegreen
</div>
]
.right-column[
- Picked Views = {`V3`, `V1`, `V6`, `V4`}
- What happens next?
]
--
.right-column[
- Dispatch starts at the end of the picked `View` object list (`V4`)
- Walking down the list, dispatch asks: will you consume this event?
- If `true`: the event is consumed and the event propagation __stops__
- If `false`: Move to the next element in the `View` list
]
---
# Input Example - Capture
.left-column[
<div class="mermaid">
graph TD
V0((V0)) --> V1((V1*))
V0 --> V4((V4*))
V1 --> V2((V2))
V1 --> V3((V3*))
V4 --> V5((V5))
V4 --> V6((V6*))
class V0,V2,V5 blue
class V1,V4,V3,V6 bluegreen
</div>
]
.right-column[
- Picked Views = {`V3`, `V1`, `V6`, `V4`}
- Dispatch starts at the end of the picked `View` object list (`V4`)
- What happens when we reach the last `View` in the list and no one has consumed the event?
]
---
# Input Example - Bubbling
.left-column[
<div class="mermaid">
graph TD
V0((V0)) --> V1((V1*))
V0 --> V4((V4*))
V1 --> V2((V2))
V1 --> V3((V3*))
V4 --> V5((V5))
V4 --> V6((V6*))
class V0,V2,V5 blue
class V1,V4,V3,V6 bluegreen
</div>
]
.right-column[
- Picked Views = {`V3`, `V1`, `V6`, `V4`}
]
--
.right-column[
- Android backtracks through the list, asking if they want to consume the event
- Who gets asked first ?
]
---
# Input Example - Recap
.left-column[
<div class="mermaid">
graph TD
V0((V0)) --> V1((V1*))
V0 --> V4((V4*))
V1 --> V2((V2))
V1 --> V3((V3*))
V4 --> V5((V5))
V4 --> V6((V6*))
class V0,V2,V5 blue
class V1,V4,V3,V6 bluegreen
</div>
]
.right-column[
|State |Ordering |
|:---------------|:----------------|
|*Picking* | V3| V1 | V6 | V4|
|*Capture Order* | 4 | 3 | 2 | 1 |
|*Bubbling Order*| 1 | 2 | 3 | 4 |
]
---
# How would you handle input in a circular component?
???
Consumption is based on the bounding box...
--
Delivery is based on bounding box
Return `false` if input is outside circle even if it's in the bounding box
---
.left-column-half[
## Event Dispatch
]
.right-column-half[
Dispatch Strategies
- Positional (Bottom-first and Top-down)
- Focus-based
]
---
.left-column-half[
## Event Dispatch
]
.right-column-half[
Callbacks handle *application* response to events
- Update Application Model
]
---
.left-column-half[
## Event Dispatch
]
.right-column-half[
Callbacks handle *application* response to events
- Update Application Model
- Best implemented using custom listeners
]
---
# What about focus input?
- Some event types are focus based (e.g. keyboard input)
- Just skip picking: Focus list is globally created, and we walk it the same way
---
# Implementing Drag and Drop
Focus? Or Positional?
???
[Android's tutorial is positional](https://developer.android.com/training/gestures/scale#java)
---
# How does *Android* decide where to send events?
- **Capture** (most things don’t) top to bottom deliver to target
object (bottom)
- example: `onInterceptTouchEvent()` in Android
- **Pick** to identify objects of interest (or just focus())
- `buildTouchDispatchChildList()` in Android; happens only after capture!
- **Bubble** (bottom to top)
- example: `onTouchEvent()` in Android
- Invoke callback (wait until complete)
- we do this by creating a custom listener in Android
.footnote[fascinating to look at [implementation in
ViewGroup](https://android.googlesource.com/platform/frameworks/base/+/master/core/java/android/view/ViewGroup.java)
-- not very general!]
---
# Creating a new type of callback
```java
protected MyView.MyListener mListener;
public void setMyListener(MyListener mListener) {
this.mListener = mListener;
}
// Creates a new named inner interface for listening to color selection events
public interface MyListener {
void onInterestingEvent(); // can include a parameter
}
// somewhere else in your code, when callback time happens
public void someMethod() {
mListener.onInterestingEvent();
}
```
---
name: inverse
layout: true
class: center, middle, inverse
---
# Using Input to Create Interaction Techniques
---
layout:false
# Interaction Technique
A method for carrying out a specific interactive task
Example: enter a number in a range
could use...
???
have the class try to think of examples
--
- (simulated) slider
- (simulated) knob
- type in a number (text edit box)
Each is a different interaction technique
---
# Example: Specify the end points for a line
Could just specify two endpoints – click, click
- not good: no affordance, <br>no feedback (/ feedforward)
Better feedback is to use “rubber banding”
- stretch out the line as you drag
- at all times, shows where you would end up <br> if you “let go”
???
Importance of feedback vs application callback
---
# Implementing rubber banding
```
Accept the press for endpoint P1; P2 = P1;
Draw line P1-P2;
Repeat
Erase line P1-P2;
P2 = current_position(); Draw line P1-P2;
Until release event;
Act on line input;
```
???
Discuss!
Not event based
Not in the basic event/redraw loop
Potentially locks up the system
---
# Implementing rubber banding
Need to get around this loop <br> absolute min of 5 times / sec
– 10 times better
– more would be better
???
aside -- why 5-10 times per second?
---
# Event driven code
Needs to respond to events as they arrive
Needs to maintain state between events
---
# Solution: State Machine
.left-column-half[
Special circle start state <br> (arrow going into it)
Special circle for 'final state' <br> (really means 'reset to start')
Transitions represent actions (callbacks).
]
.rigth-column-half[
<div class="mermaid">
graph TD
S((.)) --> A((A))
A -- "Event/Callback()" --> B((B))
B -- "Event2/Callback2()" --> C[C]
linkStyle 0 stroke-width:4px;
linkStyle 1 stroke-width:4px;
linkStyle 2 stroke-width:4px;
class S invisible
class A start
class C finish
class B normal
</div>
]
???
---
# PPS Example: Rubber Banding
.left-column-half[
Compare to previous implementation:
```
Accept the press for endpoint P1; P2 = P1;
Draw line P1-P2;
Repeat
Erase line P1-P2;
P2 = current_position(); Draw line P1-P2;
Until release event;
Act on line input;
```
]
.right-column-half[
- Determine the Events (triggers)
- Determine the States
- Determine the Actions
- Determine the Queries
]
---
# PPS Example: Rubber Banding
.left-column-half[
Compare to previous implementation:
```
Accept the press for endpoint P1; P2 = P1;
Draw line P1-P2;
Repeat
Erase line P1-P2;
P2 = current_position(); Draw line P1-P2;
Until release event;
Act on line input;
```
]
.right-column-half[
<div class="mermaid">
graph TD
S((.)) --> A((Start))
A -- "Mouse Down:?inView/Start_Line()" --> B((Drawing))
B -- "Mouse_Move:?inView/Update()" --> B
B -- "Mouse_Release:?inView/Finish_Line()" --> C[Finished]
linkStyle 0 stroke-width:4px;
linkStyle 1 stroke-width:4px;
linkStyle 2 stroke-width:4px;
linkStyle 3 stroke-width:4px;
class S invisible
class A start
class C finish
class B normal
</div>
]
---
# PPS Example: Rubber Banding
.left-column-half[
Reading a state machine: translates input sequence into action!
- When you are in Start State, and a Mouse Down event arrives, do
the action ```Start_line()``` and go to Drawing State.
- Update the line end point position every time the mouse moves.
- When it releases (Mouse Release event), finish the line (at this
stage a callback to the application might be appropriate)
]
.right-column-half[
<div class="mermaid">
graph TD
S((.)) --> A((Start))
A -- "Mouse Down:?inView/Start_Line()" --> B((Drawing))
B -- "Mouse_Move:?inView/Update()" --> B
B -- "Mouse_Release:?inView/Finish_Line()" --> C[Finished]
linkStyle 0 stroke-width:4px;
linkStyle 1 stroke-width:4px;
linkStyle 2 stroke-width:4px;
linkStyle 3 stroke-width:4px;
class S invisible
class A start
class C finish
class B normal
</div>
]
???
How could we provide a better affordance?
Does it matter if we are using a mouse or a touch screen?
---
name: inverse
layout: true
class: center, middle, inverse
---
# Using Essential Geometry as the basis for state
---
layout:false
# Using Essential Geometry as the basis for state
What is the essence of this scrollbar?
How do we capture/implement that?
---
# Scrollbar State machine with Essential Geometry
.left-column-half[
Sometimes also use “guards” --> **Propositional Production System**
- predicate (Boolean expr) before event
- adds extra conditions required to fire
- typical notation: event : pred ? action
- e.g. MouseDown : InsideAboveThumb? Scrollup()
Note: FSM augmented with guards is Turing complete
]
.right-column-half[
<div class="mermaid">
graph LR
S((.)) --> START((START))
START -- "MouseDown:InThumb?" --> SCROLLING((SCROLLING))
START -- "MouseClick:InsideAboveThumb?Scrollup()" --> DONE((DONE))
START -- "MouseClick:InsideBelowThumb?Scrolldown()" --> DONE((DONE))
SCROLLING -- "MouseMove:updateThumbDocument()" --> SCROLLING
SCROLLING -- "MouseUp:Keep()" --> DONE
linkStyle 0 stroke-width:4px;
linkStyle 1 stroke-width:4px;
linkStyle 2 stroke-width:4px;
linkStyle 3 stroke-width:4px;
linkStyle 4 stroke-width:4px;
linkStyle 5 stroke-width:4px;
class S invisible
class START start
class SCROLLING normal
class DONE finish
</div>
]
---
# Let's try it for a button
.left-column-half[
Essential geometry is:
- InsideButton
- OutsideButton
and methods for
- `indentButton()` (when button is pressed)
- `normalButton()` (when button is not pressed)
- `invokeAction()` (when the user releases in the button)
- `cancelAction()` (when the user releases outside the button)
]
---
# You should have something like this
.left-column-half[
Essential geometry is:
- InsideButton
- OutsideButton
and methods for
- `indentButton()` (when button is pressed)
- `normalButton()` (when button is not pressed)
- `invokeAction()` (when the user releases in the button)
- `cancelAction()` (when the user releases outside the button)
]
.right-column-half[
<div class="mermaid">
graph TD
S((.)) --> START((START))
START -- "DOWN:Inside?indentButton()" --> PRESSED((PRESSED))
PRESSED -- "MOVE:Outside?normalButton()" --> PRESSED
PRESSED -- "UP:Outside?cancelAction()" --> DONE[DONE]
PRESSED -- "UP:Inside?invokeAction()" --> DONE
PRESSED -- "MOVE:Inside?indentButton()" --> PRESSED
linkStyle 0 stroke-width:4px;
linkStyle 1 stroke-width:4px;
linkStyle 2 stroke-width:4px;
linkStyle 3 stroke-width:4px;
linkStyle 4 stroke-width:4px;
linkStyle 5 stroke-width:4px;
class S invisible
class START start
class PRESSED normal
class DONE finish
</div>
]
---
# How do we implement this?
.left-column-half[
- Implement in `onTouch()` using a switch statement
- Assume there is an
`essentialGeometry(Point p)` method. It returns a struct that tells you what part of the geomery you are in for that point.
- Assume implementations of all the methods
- Assume a field, `state` which is the current state of the state machine
- enums `Geometry` and `State` for comparing against
]
.right-column-half[
```java
public boolean onTouch(MotionEvent e) {
geometry = essentialGeometry(new Point(e.getX(), e.getY());
switch(state) {
case State.START:
if (geometry == Geometry.INSIDE && e.getAction() == MotionEvent.ACTION_DOWN) {
indentButton();
state = State.PRESSED;
}
case PRESSED
if (e.getAction() == MotionEvent.ACTION_MOVE) {
if (geometry == Geometry.INSIDE) {
indentButton()
} else {
normalButton()
}
} else if (e.getAction() == MotionEvent.ACTION_UP) {
state = State.START; // note we don't actually use the DONE state
if (geometry == Geometry.INSIDE) {
invokeAction()
} else {
cancelAction()
}
}
...
```
]
---
# Enums
Group of named constants
- Used for PPS in colorPicker (PPS States; Essential Geometry)
- Used for PPS in Menu assignment *and* for experimental conditions
- Easy to inspect if you want to (we do this in ExperimentSession)
[Documentation](https://docs.oracle.com/javase/tutorial/java/javaOO/enum.html)
---
.left-column[
## Harder Button
]
.right-column[
- Determine the Events (triggers)
- Determine the States
- Determine the Queries (essential geometry, context)
- Determine the Actions
]
???
What constitutes an “event” varies
- may be just low level events, or
- higher level (synthesized) events
- e.g. region-enter, press-inside
What is missing? Query fields
---
.left-column[
## Facebook Button Solution
]
--
.right-column[
Press:?inside => highlight(), start_animation(), small, active<br>
AnimateStep ==> update(), active<br>
AnimateFinish ==> !small, active<br>
Release:inside,small => unhighlight(), exit()<br>
Release:inside,!small => add_to_chat(), small, unhighlight(),
exit()<br>
__rest is unknowable from this animation__
<div class="mermaid">
graph LR
S((.)) --> A((Start))
A -- "Press:?inside/highlight(), start_animation()" --> B((Active))
B -- "AnimateStep,update()" --> B
B -- "AnimateFinish,!small"--> B
B -- "Release,inside:small, unhighlight" -->D(End)
B -- "Release,inside:!small,add_to_chat(),unhighlight()" --> D
classDef finish outline-style:double,fill:#d1e0e0,stroke:#333,stroke-width:2px;
classDef normal fill:#e6f3ff,stroke:#333,stroke-width:2px;
classDef start fill:#d1e0e0,stroke:#333,stroke-width:4px;
classDef invisible fill:#FFFFFF,stroke:#FFFFFF,color:#FFFFFF
linkStyle 0 stroke-width:4px;
linkStyle 1 stroke-width:4px;
linkStyle 2 stroke-width:4px;
linkStyle 3 stroke-width:4px;
linkStyle 4 stroke-width:4px;
linkStyle 5 stroke-width:4px;
class S invisible
class A start
class D finish
class B normal
</div>
]
---
# When to use PPSs
You're probably already using them, just not intentionally (and maybe
less well as a result)
PPSs are a good way to do control flow in event driven systems
Can do (formal or informal) analysis
- are all possible inputs (e.g. errors) handled from each state
- what are next legal inputs: can use to enable / disable
Can be automated based on higher level specification
---
#Summary
State machines are very good (for this job) but do have limits
State machines don't handle independent actions very well (state explosion)
Mostly useful for smaller things
- Great for individual components
- Not so great for whole dialogs
Path of least resistance is rigid sequencing
Ask: is this good for what I am doing?
???
xxx TODO decide whether to keep
xxx TODO decide how to end this deck and/or what other material needs
to be covered
---