README.mdwn 3.4 KB
Newer Older
Johannes Bechberger's avatar
Johannes Bechberger committed
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
MJTest
======

A test runner (and suite) for the MiniJava compiler (and its parts) written in the compiler lab of the KIT.

It's heavily inspired by Sisyphus (and uses some of its code)

__Please contribute to the test cases__

Test modes
----------
The test cases are divided in 3 'modes':
- __syntax__: Test cases that just check whether `./run --parsecheck` accepts as correct or rejects
 them.
- __semantic__: Test cases that check semantic checking of MiniJava programs
- __exec__: Test cases that check the correct compilation of MiniJava programs.

_Only the syntax mode is currently usable, but the other three will follow._

The test different test cases for each mode are located in a folder with the same name.
The default directory that contains all test folders is `tests`.

The different types a test cases are differentiated by their file endings.


Test types for the syntax mode
------------------------------

<table>
<tr><th>File ending(s) of test cases</th><th>Expected behaviour to complete a test of this type</th></tr>
<tr>
    <td><code>.valid.mj</code><code>.mj</code>
    <td>Return code is <code>0</code>, i.e. the MiniJava is accepted as syntactically correct</td>
</tr>
<tr>
    <td><code>.invalid.mj</code>
    <td>Return code is <code>&gt; 0</code> and the error output contains the word <code>error</code></td>
</tr>
</table>


Test runner
-----------

### Requirements
The following programs are required (and executable by simply calling their names).
- `python3` (at least Python3.3)
- `javac` and `java`

### Installation

Just clone `mjtest` and install it via `pip3`.

```sh
git clone https://github.com/mj3-16/mjtest
cd mjtest
sudo pip3 install .
```

### Usage
Output of the `mjtest --help`

```
usage: mjtest [-h] [--tmp_dir] [--test_dir] [--only_incorrect_tests]
              [--parallel] [--timeout] [--report_dir] [--log LOG]
              {syntax,semantic,exec} MJ_RUN_CMD

MiniJava test runner

positional arguments:
  {syntax,semantic,exec}
                        What do you want to test?
  MJ_RUN_CMD            Command to run your MiniJava implementation, e.g.
                        `mj/run`

optional arguments:
  -h, --help            show this help message and exit
  --tmp_dir             Used temporary directory
  --test_dir            Directory that contains all test cases, default is the
                        'tests' directory
  --only_incorrect_tests
                        Only run the tests that were incorrect the last run
  --parallel            Run the tests in parallel
  --timeout             Abort a program after TIMEOUT seconds
  --report_dir          Directory to store the reports in, default is
                        'reports'
  --log LOG             Logging level (error, warn, info or debug)
```

### Example usage
Assuming you want to run the syntax tests and your MiniJava base folder is `~/code/mj` then run
```
mjtest syntax `~/code/mj/run --lextest`
```
This will…
- … create reports in a folder named after the current date and time inside the `reports` folder
- … output something like
```
```
- … log that some test cases were executed correctly
- … return with an error code of `0` if all tests executed correct

Contributions
-------------
__Please contribute to this test runner and the accompanied test cases.__

To add test cases just open a pull request. The test cases must have unique names (in each mode folder).

Licence
-------
MIT, see LICENCE file for more information.