README.mdwn 6.83 KB
Newer Older
Johannes Bechberger's avatar
Johannes Bechberger committed
1 2 3 4 5 6 7
MJTest
======

A test runner (and suite) for the MiniJava compiler (and its parts) written in the compiler lab of the KIT.

It's heavily inspired by Sisyphus (and uses some of its code)

8
__Please contribute test cases.__
Johannes Bechberger's avatar
Johannes Bechberger committed
9

Johannes Bechberger's avatar
Johannes Bechberger committed
10 11
__Semantic test cases that are executable (and terminate) should be placed in the `exec` folder.__

12
__Exec test cases can now use the import statement to import other classes. See the exec test case folder and the preproc folder for more information__
Johannes Bechberger's avatar
Johannes Bechberger committed
13 14 15
*There's also some other additional code in this repo, that might provide fuzzing capabilities. Please consider contributing
such code as it probably helps the other teams (and could later be integrated into a bigger test suite).*

Johannes Bechberger's avatar
Johannes Bechberger committed
16 17
Test modes
----------
18 19
The test cases are divided in 5 'modes':
- __lexer__: Test cases that check the lexed token (and their correct output)
Johannes Bechberger's avatar
Johannes Bechberger committed
20 21
- __syntax__: Test cases that just check whether `./run --parsecheck` accepts as correct or rejects
 them.
Johannes Bechberger's avatar
Johannes Bechberger committed
22
- __ast__: Test cases that check the generated ast by using the pretty printing functionality.
Johannes Bechberger's avatar
Johannes Bechberger committed
23 24 25
- __semantic__: Test cases that check semantic checking of MiniJava programs
- __exec__: Test cases that check the correct compilation of MiniJava programs.

26
_Only the lexer, syntax, ast and semantic  mode are currently usable, but the others will follow._
Johannes Bechberger's avatar
Johannes Bechberger committed
27 28 29 30

The test different test cases for each mode are located in a folder with the same name.
The default directory that contains all test folders is `tests`.

31 32
Sub folders in side the test case folders are allowed.

Johannes Bechberger's avatar
Johannes Bechberger committed
33 34
The different types a test cases are differentiated by their file endings.

35
Side note: An error code greater than 0 should result in an error message on error output containing the word `error`.
Johannes Bechberger's avatar
Johannes Bechberger committed
36

Johannes Bechberger's avatar
Johannes Bechberger committed
37

38 39 40 41 42 43
Test types for the lexer mode
------------------------------

<table>
<tr><th>File ending(s) of test cases</th><th>Expected behaviour to complete a test of this type</th></tr>
<tr>
44
    <td><code>.valid.mj</code> <code>.mj</code></td>
45 46 47 48 49 50 51 52 53
    <td>Return code is <code>0</code> and the output matches the expected output (located in the file `[test file].out`</td>
</tr>
<tr>
    <td><code>.invalid.mj</code>
    <td>Return code is <code>&gt; 0</code> and the error output contains the word <code>error</code></td>
</tr>
</table>


Johannes Bechberger's avatar
Johannes Bechberger committed
54 55 56 57 58 59
Test types for the syntax mode
------------------------------

<table>
<tr><th>File ending(s) of test cases</th><th>Expected behaviour to complete a test of this type</th></tr>
<tr>
60
    <td><code>.valid.mj</code> <code>.mj</code> <code>.valid.java</code> <code>.java</code>
Johannes Bechberger's avatar
Johannes Bechberger committed
61 62 63
    <td>Return code is <code>0</code>, i.e. the MiniJava is accepted as syntactically correct</td>
</tr>
<tr>
64
    <td><code>.invalid.mj</code> <code>.invalid.java</code></td>
Johannes Bechberger's avatar
Johannes Bechberger committed
65 66 67 68
    <td>Return code is <code>&gt; 0</code> and the error output contains the word <code>error</code></td>
</tr>
</table>

69
It uses all semantic mode tests implicitly if `--ci_testing` is given.
Johannes Bechberger's avatar
Johannes Bechberger committed
70 71 72 73 74 75 76

Test types for the ast mode
------------------------------

<table>
<tr><th>File ending(s) of test cases</th><th>Expected behaviour to complete a test of this type</th></tr>
<tr>
77
    <td><code>.valid.mj</code> <code>.mj</code> <code>.valid.java</code> <code>.java</code></td>
Johannes Bechberger's avatar
Johannes Bechberger committed
78
    <td>Pretty printing the source file should result in the same output as pretty printing the already pretty printed file.</td>
Johannes Bechberger's avatar
Johannes Bechberger committed
79 80 81
</tr>
</table>

Johannes Bechberger's avatar
Johannes Bechberger committed
82 83
It uses all syntax mode tests implicitly.

Johannes Bechberger's avatar
Johannes Bechberger committed
84

85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102
Test types for the semantic mode
------------------------------

<table>
<tr><th>File ending(s) of test cases</th><th>Expected behaviour to complete a test of this type</th></tr>
<tr>
    <td><code>.valid.mj</code> <code>.mj</code> <code>.valid.java</code> <code>.java</code>
    <td>Return code is <code>0</code>, i.e. the MiniJava is accepted as semantically correct</td>
</tr>
<tr>
    <td><code>.invalid.mj</code> <code>.invalid.java</code></td>
    <td>Return code is <code>&gt; 0</code> and the error output contains the word <code>error</code></td>
</tr>
</table>

__All semantic mode tests have to syntactically correct___


Johannes Bechberger's avatar
Johannes Bechberger committed
103 104 105 106 107
Test runner
-----------

### Requirements
The following programs are required (and executable by simply calling their names).
108 109
- `python3` (at least Python3.4)
- `javac` and `java` (for `.java` test cases)
Johannes Bechberger's avatar
Johannes Bechberger committed
110 111 112

### Installation

113
Just clone this repository and you're done:
Johannes Bechberger's avatar
Johannes Bechberger committed
114 115 116 117 118 119

```sh
git clone https://github.com/mj3-16/mjtest
cd mjtest
```

120 121 122
To get colored output install the python3 module `termcolor`.


Johannes Bechberger's avatar
Johannes Bechberger committed
123
### Usage
124
Output of the `./mjt.py --help`
Johannes Bechberger's avatar
Johannes Bechberger committed
125 126

```
127
usage: mjt.py [-h] [--only_incorrect_tests] [--produce_no_reports]
128
              [--produce_all_reports] [--parallel]
129 130 131
              [--output_no_incorrect_reports] [--ci_testing]
              [--log_level LOG_LEVEL]
              {all,lexer,syntax,ast,semantic,exec} MJ_RUN
Johannes Bechberger's avatar
Johannes Bechberger committed
132 133 134 135

MiniJava test runner

positional arguments:
136
  {all,lexer,syntax,ast,semantic,exec}
Johannes Bechberger's avatar
Johannes Bechberger committed
137
                        What do you want to test?
138 139 140
  MJ_RUN                Command to run your MiniJava implementation, e.g.
                        `mj/run`, can be omitted by assigning the environment
                        variable MJ_RUN
Johannes Bechberger's avatar
Johannes Bechberger committed
141 142 143 144 145

optional arguments:
  -h, --help            show this help message and exit
  --only_incorrect_tests
                        Only run the tests that were incorrect the last run
146 147
  --produce_no_reports  Produce no long reports besides the command line
                        output
148 149
  --produce_all_reports
                        Produce reports also for correct test cases
Johannes Bechberger's avatar
Johannes Bechberger committed
150
  --parallel            Run the tests in parallel
151 152
  --output_no_incorrect_reports
                        Output the long report for every incorrect test case
153 154 155 156
  --ci_testing          In mode X the succeeding test cases of later
                        modes/phases should also succeed in this mode, and
                        failing test cases of prior modes/phases should also
                        fail in this phase.
157 158
  --log_level LOG_LEVEL
                        Logging level (error, warn, info or debug)
Johannes Bechberger's avatar
Johannes Bechberger committed
159 160
```

161 162
For Windows users: Using environment variables doesn't work and you have to use prefix `./mjt.py` with `python`.

Johannes Bechberger's avatar
Johannes Bechberger committed
163 164 165
### Example usage
Assuming you want to run the syntax tests and your MiniJava base folder is `~/code/mj` then run
```
166
MJ_RUN="~/code/mj/run" ./mjt.py syntax
Johannes Bechberger's avatar
Johannes Bechberger committed
167 168 169 170 171
```
This will…
- … create reports in a folder named after the current date and time inside the `reports` folder
- … output something like
```
Johannes Bechberger's avatar
Johannes Bechberger committed
172 173 174
[FAIL   ] syntax:test.invalid                     the error output doesn't contain the word "error"
----------------------------------------
Ran 1 tests, of which 1 failed.
175
A full report for each test can be found at mjtest/reports/30-10-16_08-05-10.failed
Johannes Bechberger's avatar
Johannes Bechberger committed
176 177 178 179 180 181 182 183 184 185 186 187 188
```
- … log that some test cases were executed correctly
- … return with an error code of `0` if all tests executed correct

Contributions
-------------
__Please contribute to this test runner and the accompanied test cases.__

To add test cases just open a pull request. The test cases must have unique names (in each mode folder).

Licence
-------
MIT, see LICENCE file for more information.