README.md 5.65 KB
Newer Older
Ryan Rowe's avatar
Ryan Rowe committed
1
2
# CSE 340 Grading Harness

Jeremy Zhang's avatar
Jeremy Zhang committed
3
This is a Edstem-compatible grading harness.
Ryan Rowe's avatar
Ryan Rowe committed
4
5

## Environment
Jeremy Zhang's avatar
Jeremy Zhang committed
6
As CSE 340 is an Android-based course, any grading harness would require access to the Android SDK as well as a testing emulator.
Ryan Rowe's avatar
Ryan Rowe committed
7

Jeremy Zhang's avatar
Jeremy Zhang committed
8
Such a testing environment is established during the sourcing of `env.sh` in all executables. It establishes the Android SDK location and prepares the files environment.
Ryan Rowe's avatar
Ryan Rowe committed
9

Jeremy Zhang's avatar
Jeremy Zhang committed
10
11
### Workspaces vs Lessons
During development of this harness, we would set the DEBUG bit to 1 in `env.sh`. This would modify the behavior for the environment to source the submission files from `/home/Submission` rather than just `/home`.
Ryan Rowe's avatar
Ryan Rowe committed
12

Jeremy Zhang's avatar
Jeremy Zhang committed
13
14
15
16
17
18
### `/course` Directory
Ed provides a convenient read-only course-specific directory that is shared between all assignment and workspaces. As such we are able to use it to minimize network calls.

- `cache/gradle` - `/home/.gradle` cached after executing each of our assignments to generate a local copy of dependencies.
- `data` - Contains executables such as `aapt2 7.0.2` and `gradle 5.6.4`.

Ryan Rowe's avatar
Ryan Rowe committed
19
20
## Executables

Jeremy Zhang's avatar
Jeremy Zhang committed
21
There three harness executables. The first one is `check.sh` which is executed during check. It is intended to verify comparability of submitted code and provides students their comparison screenshots. The second one is `mark.sh` which compiles and grades the code according to unit and integration test results. Finally a `run.sh` is provided so graders may run submissions on the spot without executing tests. All executables execute in the environment specified above.
Ryan Rowe's avatar
Ryan Rowe committed
22
23
24
25
26
27
28
29
30
31
32
33

### Scripts

Scripts serve to support the main executables. For now, they are only python3 and have their dependencies listed in `scripts/requirements.txt`.

#### `move.py`

This helper script will analyze `assignment.xml` and identify any submission files which must be moved into the solution. To do this, it looks for the `dest` attribute on files in the XML. If such an attribute exists, the script will do its best to move (or unzip) the specified file to its destination. For usage, see `scripts/move.py -h`.

#### `tests.py`

This helper script will output grading commands to stdout based on `critera.xml` in conjunction with XML files output during application testing. Any grading items which contain a `testRegex` attribute will be considered. For each item, a grade between 0 and max will be assigned proportional to the number of tests matching the supplied regex which have passed. The order of items does not matter, with one exception. If the attribute `testSubtracts` is specified, any tests which match that item's regex will not be considered for subsequent items.
Ryan Rowe's avatar
Ryan Rowe committed
34

Jeremy Zhang's avatar
Jeremy Zhang committed
35
### Check
Ryan Rowe's avatar
Ryan Rowe committed
36

Jeremy Zhang's avatar
Jeremy Zhang committed
37
On preparing the environment, a student's code is loaded into the solution by `move.py` and compiled with `gradle`. If compilation is successful, the turn-in succeeds and fails otherwise. A screenshot comparison is provided as HTML file to support the student.
Ryan Rowe's avatar
Ryan Rowe committed
38

Jeremy Zhang's avatar
Jeremy Zhang committed
39
### Mark
Ryan Rowe's avatar
Ryan Rowe committed
40

Jeremy Zhang's avatar
Jeremy Zhang committed
41
At grade time, each student's code is again loaded into the solution by move.py. The emulator is wiped of all previous installations and screenshots. The application is compiled and installed on the target device and integration tests are then run. Finally, `tests.py` is run which reads JUnit test result XML files and assigns points according to GitGrade rubric. After grading is complete the grading commands along with any screenshots files produced are written out of the remote execution environment.
Ryan Rowe's avatar
Ryan Rowe committed
42
43
44

## Support Files

Jeremy Zhang's avatar
Jeremy Zhang committed
45
46
### `criteria.json`
TODO: see `criteria.xml`.
Ryan Rowe's avatar
Ryan Rowe committed
47

Ryan Rowe's avatar
Ryan Rowe committed
48
### `file_mappings.json`
Jeremy Zhang's avatar
Jeremy Zhang committed
49
This file takes the following form and paths support `*` and `**` globbing:
Ryan Rowe's avatar
Ryan Rowe committed
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67

```
[
  {
    "path_to_source": true   # This indicates the file should be moved to the corresponding path in Solution and is optional
    "path_to_source": false  # This indicates the file should be moved to the corresponding path in Solution and is required
    "exclude": [             # This is optional, scoped to this dict
        "paths_to_exclude",
        ...
    ],
    "optional": true         # This is overridden by the more tightly bound optional true/falses above. However, it allows you to do this:
    "path_to_source": "different_dest_path"  # So if you want to do this to one such file, encapsulate it in it's own scope
  }
]
```

### `criteria.xml` (DEPRECATED)
~~This is an XML file used by GradeIt to describe the point distribution for an assignment. We've expanded upon this format by also using the file to store the test case regex which is used by `tests.py` to assign points. Below is an exerpt of Doodle's `criteria.xml` to show the syntax:~~
Ryan Rowe's avatar
Ryan Rowe committed
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93

```xml
<?xml version="1.0" encoding="UTF-8"?>
<criteria course="cse340" assignment="Doodle">
    <student/>
    <grader/>

    <category name="Turn-in">
        <item max="1" id="0">Compiles</item>
    </category>

    <category name="Integration Tests">
        <category name="Part 1">
            <item max="1" id="1" testRegex=".*Part1Tests\..*_(landscape|constraints|contents|bitmap)">Landscape</item>
            <item max="1" id="2" testRegex=".*Part1Tests\..*_(portrait|contents|bitmap)">Portrait</item>
        </category>

        <category name="Part 2">
            <item max="1" id="3" testRegex=".*Part2Tests\..*">Portrait and Landscape</item>
        </category>
        
        <!- ... ->
    </category>
</criteria>
```

Jeremy Zhang's avatar
Jeremy Zhang committed
94
We added the `testRegex` attribute. The order of the `<item>`s does not matter unless the `testSubtracts` attribute is specified on an item. If this attribute is specified, test cases matching the item regex will be excluded from consideration in subsequent items.
95
96
97

### GitGrade Assignment Rubric
Using the Item Specific Tags feature in the rubric (designated by the tag symbol button next to an item), `testRegex` may be added as a key along with the value of a regex for the tests to apply the score automatically.