Commit 93fbbcf4 authored by James R. Wilcox's avatar James R. Wilcox
Browse files

notes for balanced token fuzzer

parent 08a2c858
......@@ -124,3 +124,67 @@ This fuzzer generates input by generating a random sequence of 100 tokens. Each
token is drawn from a custom distribution that we made up because it seemed
reasonable. See the code documentation for `genTokenStringInto` for more
information, and feel free to play around with tweaking the token distribution!
## Token fuzzer with balanced parentheses
Now we start to diverge from our Trefoil v1 fuzzers as we add more structure. In
Trefoil v1, our next step was to introduce a fuzzer that never underflowed the
stack. That fuzzer's structure reflected the structure of the language being
fuzzed. In Trefoil v2, we don't have a stack, but we do have a similar kind of
common error that we can try to eliminate: unbalanced parentheses.
This fuzzer keeps track of how many open parentheses it has generated so that it
knows to close exactly the right amount.
Let's take a look at the numbers.
```
Paren:0
Abstract:695841
Unbound Variables:285036
Unbound Functions:18075
Other Runtime:1030
StackOverflows:0
Programs:18
Total:1000000
```
The first thing to notice is that there are 0 parenthesized symbol errors. We
succeeded in our goal of structurally eliminating this class of error from our
fuzzers output! This is a very typical workflow for fuzzing: refining the output
by adding structure to eliminate a class of error that we are no longer
interested in testing, so that we can go deeper in other parts of the input
space.
Since parenthesized syntax errors used to make up nearly 30% of the results,
generally we expect all other results to become more frequent.
- 70% of results are now abstract syntax error. In the staff solution, most of
these are from keyword errors, but also a good number from generating the
empty set of parens `()` (never valid in Trefoil v2) or from applying an
operator to the wrong number of arguments.
- 30% of results are unbound variables or functions
- About the same number of results (0.1%) are other runtime errors.
- Perhaps surprisingly, there are many fewer valid programs (essentially zero --
only 18 out of 1 million). This is because this fuzzer does not generate
comment characters, so it is *much* harder for the fuzzer to generate 100
tokens without making a mistake.
Here is one lucky program from our fuzzer:
```
-3 -8 -6 ( * 6 -1632566905 ) -190 -10
```
Mostly just a bunch of literals, but one correct multiplication operation!
This is pretty impressive, because each token of this input was generated
randomly. So it had to get lucky to open the parentheses, close the
parentheses, and generate the correct number of arguments (of the correct
type!). Now you see why this almost never happens.
### The generating procedure
Similar to the previous fuzzer except that it tracks an `int` that counts the
number of open-but-not-yet closed parentheses. In order to make this tracking
easier, this fuzzer does not generate comments (since parentheses inside
comments wouldn't count).
Supports Markdown
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment