README.md 12.5 KB
Newer Older
Johannes Bechberger's avatar
Johannes Bechberger committed
1 2 3 4 5
MJTest
======

A test runner (and suite) for the MiniJava compiler (and its parts) written in the compiler lab of the KIT.

Johannes Bechberger's avatar
Johannes Bechberger committed
6
It's heavily inspired by Sisyphus (and uses some of its code). See the `preproc` folder more information on the included preprocessor.
Johannes Bechberger's avatar
Johannes Bechberger committed
7

Johannes Bechberger's avatar
Johannes Bechberger committed
8
__Please contribute test cases and improve the existing ones.__
Johannes Bechberger's avatar
Johannes Bechberger committed
9

10
__Semantic test cases that are executable (and terminate) should be placed in the `exec` folder or sub folder and should follow the style recommendations from end of this document.__
Johannes Bechberger's avatar
Johannes Bechberger committed
11 12

__Exec test cases can now use the import statement to import other classes. See the exec test case folder and the `preproc` folder for more information__
Johannes Bechberger's avatar
Johannes Bechberger committed
13

Johannes Bechberger's avatar
Johannes Bechberger committed
14 15 16
*There's also some other additional code in this repo, that might provide fuzzing capabilities. Please consider contributing such code as it probably helps the other teams (and could later be integrated into a bigger test suite).*

**The test cases should be committed in the [mjtest-tests](https://git.scc.kit.edu/IPDSnelting/mjtest-tests) repository**
Johannes Bechberger's avatar
Johannes Bechberger committed
17

Johannes Bechberger's avatar
Johannes Bechberger committed
18 19
Test modes
----------
20
The test cases are divided in 6 'modes':
21
- __lexer__: Test cases that check the lexed token (and their correct output)
Johannes Bechberger's avatar
Johannes Bechberger committed
22
- __syntax__: Test cases that just check whether `./run --parsecheck` accepts as correct or rejects
Johannes Bechberger's avatar
Johannes Bechberger committed
23
   them.
24
- __ast__: Test cases that check the generated ast by using the pretty printing functionality.
Johannes Bechberger's avatar
Johannes Bechberger committed
25
- __semantic__: Test cases that check semantic checking of MiniJava programs
26
- __compile-firm-only__: Test cases that check the compilation of MiniJava programs with the libfirm backend.
27
- __compile-firm__ (alias `exec-firm`): Test cases that check the correct compilation and execution of MiniJava programs with the libfirm backend.
28
- __compile-only__: Test cases that check the compilation MiniJava programs with the self implemented backend.
29
- __compile__ (alias `exec`): Test cases that check the correct compilation and execution of MiniJava programs with the self implemented backend.
Johannes Bechberger's avatar
Johannes Bechberger committed
30

Johannes Bechberger's avatar
Johannes Bechberger committed
31 32
The test different test cases for each mode are located in a folder with the same name. 
Except the compile-firm test cases which are located in the `exec` folder.
Johannes Bechberger's avatar
Johannes Bechberger committed
33 34 35

The default directory that contains all test folders is `tests`.

36 37
Sub folders in side the test case folders are allowed.

Johannes Bechberger's avatar
Johannes Bechberger committed
38 39
The different types a test cases are differentiated by their file endings.

40
Side note: An error code greater than 0 should result in an error message on error output containing the word `error`.
Johannes Bechberger's avatar
Johannes Bechberger committed
41

Johannes Bechberger's avatar
Johannes Bechberger committed
42

43 44 45 46 47 48
Test types for the lexer mode
------------------------------

<table>
<tr><th>File ending(s) of test cases</th><th>Expected behaviour to complete a test of this type</th></tr>
<tr>
49
    <td><code>.valid.mj</code> <code>.mj</code></td>
Johannes Bechberger's avatar
Johannes Bechberger committed
50
    <td>Return code is <code>0</code> and the output matches the expected output (located in the file `[test file].out`)</td>
51 52 53 54 55 56 57 58
</tr>
<tr>
    <td><code>.invalid.mj</code>
    <td>Return code is <code>&gt; 0</code> and the error output contains the word <code>error</code></td>
</tr>
</table>


Johannes Bechberger's avatar
Johannes Bechberger committed
59 60 61 62 63 64
Test types for the syntax mode
------------------------------

<table>
<tr><th>File ending(s) of test cases</th><th>Expected behaviour to complete a test of this type</th></tr>
<tr>
65
    <td><code>.valid.mj</code> <code>.mj</code> <code>.valid.java</code> <code>.java</code>
Johannes Bechberger's avatar
Johannes Bechberger committed
66 67 68
    <td>Return code is <code>0</code>, i.e. the MiniJava is accepted as syntactically correct</td>
</tr>
<tr>
69
    <td><code>.invalid.mj</code> <code>.invalid.java</code></td>
Johannes Bechberger's avatar
Johannes Bechberger committed
70 71 72 73
    <td>Return code is <code>&gt; 0</code> and the error output contains the word <code>error</code></td>
</tr>
</table>

74
It uses all semantic mode tests implicitly if `--ci_testing` is given.
Johannes Bechberger's avatar
Johannes Bechberger committed
75 76 77 78 79 80 81

Test types for the ast mode
------------------------------

<table>
<tr><th>File ending(s) of test cases</th><th>Expected behaviour to complete a test of this type</th></tr>
<tr>
82
    <td><code>.valid.mj</code> <code>.mj</code> <code>.valid.java</code> <code>.java</code></td>
Johannes Bechberger's avatar
Johannes Bechberger committed
83
    <td>Pretty printing the source file should result in the same output as pretty printing the already pretty printed file.</td>
Johannes Bechberger's avatar
Johannes Bechberger committed
84 85
</tr>
</table>
Johannes Bechberger's avatar
Johannes Bechberger committed
86
_It uses all syntax mode tests implicitly._
Johannes Bechberger's avatar
Johannes Bechberger committed
87

Johannes Bechberger's avatar
Johannes Bechberger committed
88
Place the test files in the folder `ast`. Might not be all that useful, wasn't used at all in the two years ago.
89

Johannes Bechberger's avatar
Johannes Bechberger committed
90

91 92 93 94 95 96
Test types for the semantic mode
------------------------------

<table>
<tr><th>File ending(s) of test cases</th><th>Expected behaviour to complete a test of this type</th></tr>
<tr>
Johannes Bechberger's avatar
Johannes Bechberger committed
97
    <td><code>.valid.mj</code> <code>.mj</code> <code>.valid.java</code> <code>.java</code></td>
98 99 100 101 102 103 104 105
    <td>Return code is <code>0</code>, i.e. the MiniJava is accepted as semantically correct</td>
</tr>
<tr>
    <td><code>.invalid.mj</code> <code>.invalid.java</code></td>
    <td>Return code is <code>&gt; 0</code> and the error output contains the word <code>error</code></td>
</tr>
</table>

Johannes Bechberger's avatar
Johannes Bechberger committed
106
__All semantic mode tests have to be syntactically correct__
107

108 109 110 111 112 113 114 115 116 117 118 119 120
Test types for the compile-firm-only and compile-only mode
----------------------------------------------------------

<table>
<tr><th>File ending(s) of test cases</th><th>Expected behaviour to complete a test of this type</th></tr>
<tr>
    <td><code>.valid.mj</code> <code>.mj</code> <code>.valid.java</code> <code>.java</code></td>
    <td>Return code is <code>0</code> and the compiler produces an `a.out` file, i.e. the MiniJava code can be compiled</td>
</tr>
</table>

__All valid semantic test-cases are also used, the non-semantic compile-(firm)-only
should be placed into the `compile-only` folder__
121

122 123
Test types for the compile-firm and compile mode
------------------------------------------------
Johannes Bechberger's avatar
Johannes Bechberger committed
124 125 126 127 128 129

<table>
<tr><th>File ending(s) of test cases</th><th>Expected behaviour to complete a test of this type</th></tr>
<tr>
    <td><code>.java</code> <code>.mj</code></td>
    <td>The test file should compile. The resulting binary should be runnable and its output should be:<br/>
uqddy's avatar
uqddy committed
130
    if a <code>[input file].out</code> (for test cases with input) or <code>[test file].out</code> (else) exists: the content of this file<br/>
Johannes Bechberger's avatar
Johannes Bechberger committed
131 132
    if no such file exists: the same as the execution of the file with <code>java</code></td>
</tr>
133 134 135 136 137 138 139 140 141 142 143 144 145 146
<tr>
    <td><code>.[number].inputc</code></td>
    <td>This a test case with an input. The input is interpreted as ASCII chars. It requires a corresponding 
    <code>.input.java</code> or <code>.input.mj</code> file that is executed with this input.
    The rest is equivalent to normal test cases from the row above</td>
</tr>
<tr>
    <td><code>.[number].input</code></td>
    <td>This is an input test case whichs input file contains space seperated decimal numbers that are used as input data.</td>
</tr>
<tr>
    <td><code>.invalid.mj</code> <code>.invalid.java</code></td>
    <td>The test file should compile. The execution of the resulting binary should abort width a return code <code>&gt; 0</code></td>
</tr>
Johannes Bechberger's avatar
Johannes Bechberger committed
147 148 149 150
<tr>
    <td><code>.inf.mj</code> <code>.inf.java</code></td>
    <td>The test file should run longer than the specified timeout. This causes long test run-times.</td>
</tr>
Johannes Bechberger's avatar
Johannes Bechberger committed
151 152
</table>

Johannes Bechberger's avatar
Johannes Bechberger committed
153 154 155
The test cases should be placed into the `exec` folder. 

__All compile-firm and compile mode tests have to be semantically correct.__
Johannes Bechberger's avatar
Johannes Bechberger committed
156

Johannes Bechberger's avatar
Johannes Bechberger committed
157
__All bigger test cases (e.g. for benchmarking) should be placed into the `big` folder.__
Johannes Bechberger's avatar
Johannes Bechberger committed
158 159


Johannes Bechberger's avatar
Johannes Bechberger committed
160 161 162 163 164
Test runner
-----------

### Requirements
The following programs are required (and executable by simply calling their names).
165
- `python3` (at least python3.5, older python3 versions might work with the `typing` package installed)
166
- `javac` and `java` (for `.java` test cases)
Johannes Bechberger's avatar
Johannes Bechberger committed
167 168 169

### Installation

170
Just clone this repository and you're done:
Johannes Bechberger's avatar
Johannes Bechberger committed
171 172

```sh
Johannes Bechberger's avatar
Johannes Bechberger committed
173
git clone git@git.scc.kit.edu:IPDSnelting/mjtest.git
Johannes Bechberger's avatar
Johannes Bechberger committed
174
cd mjtest
Johannes Bechberger's avatar
Johannes Bechberger committed
175
git submodule update --init
Johannes Bechberger's avatar
Johannes Bechberger committed
176 177
```

178 179 180
To get colored output install the python3 module `termcolor`.


Johannes Bechberger's avatar
Johannes Bechberger committed
181
### Usage
182
Output of the `./mjt.py --help`
Johannes Bechberger's avatar
Johannes Bechberger committed
183

Johannes Bechberger's avatar
Johannes Bechberger committed
184
```sh
Johannes Bechberger's avatar
Johannes Bechberger committed
185 186
usage: mjt.py [-h] [--only_incorrect_tests] [--all_exec_tests]
              [--produce_no_reports] [--produce_all_reports] [--parallel]
187
              [--output_no_incorrect_reports] [--color] [--ci_testing]
188 189
              [--log_level LOG_LEVEL] [--bench_compiler_flag_1]
              [--bench_compiler_flag_2]
190
              {all,lexer,syntax,ast,semantic,compile-firm-only,compile-only,
191
               compile-firm,compile,exec,exec-firm,bench} MJ_RUN
Johannes Bechberger's avatar
Johannes Bechberger committed
192 193 194 195

MiniJava test runner

positional arguments:
196
  {all,lexer,syntax,ast,semantic,compile-firm-only,compile-only,compile-firm,exec,exec-firm}
Johannes Bechberger's avatar
Johannes Bechberger committed
197
                        What do you want to test?
198 199 200
  MJ_RUN                Command to run your MiniJava implementation, e.g.
                        `mj/run`, can be omitted by assigning the environment
                        variable MJ_RUN
Johannes Bechberger's avatar
Johannes Bechberger committed
201 202 203 204 205

optional arguments:
  -h, --help            show this help message and exit
  --only_incorrect_tests
                        Only run the tests that were incorrect the last run
Johannes Bechberger's avatar
Johannes Bechberger committed
206 207
  --all_exec_tests      Run all exec (compile-firm...) tests, not only the
                        small ones
208 209
  --produce_no_reports  Produce no long reports besides the command line
                        output
210 211
  --produce_all_reports
                        Produce reports also for correct test cases
Johannes Bechberger's avatar
Johannes Bechberger committed
212
  --parallel            Run the tests in parallel
213 214
  --output_no_incorrect_reports
                        Output the long report for every incorrect test case
215 216
  --color               Output color output even if the output goes to into a
                        file
217 218 219 220
  --ci_testing          In mode X the succeeding test cases of later
                        modes/phases should also succeed in this mode, and
                        failing test cases of prior modes/phases should also
                        fail in this phase.
221 222 223 224 225 226 227 228
  --bench_compiler_flag_1 BENCH_COMPILER_FLAG_1
                        Set the first compiler flag/mode that is used for
                        comparison, 'javac' for the java compiler
  --bench_compiler_flag_2 BENCH_COMPILER_FLAG_2
                        Set the first compiler flag/mode that is used for
                        comparison, 'javac' for the java compiler
  --bench_runs BENCH_RUNS
                        Number of times to run a benchmarked code
229 230
  --log_level LOG_LEVEL
                        Logging level (error, warn, info or debug)
Johannes Bechberger's avatar
Johannes Bechberger committed
231 232
```

233 234
For Windows users: Using environment variables doesn't work and you have to use prefix `./mjt.py` with `python`.

Johannes Bechberger's avatar
Johannes Bechberger committed
235 236
The timeout for the execution of normal programs in seconds can be set via the environment variable `MJ_TIMEOUT` (default is `10`) and for `big` programs via `MJ_BIG_TIMEOUT` (default is `60`).

Johannes Bechberger's avatar
Johannes Bechberger committed
237 238 239
### Example usage
Assuming you want to run the syntax tests and your MiniJava base folder is `~/code/mj` then run
```
240
MJ_RUN="~/code/mj/run" ./mjt.py syntax
Johannes Bechberger's avatar
Johannes Bechberger committed
241 242 243 244 245
```
This will…
- … create reports in a folder named after the current date and time inside the `reports` folder
- … output something like
```
Johannes Bechberger's avatar
Johannes Bechberger committed
246 247 248
[FAIL   ] syntax:test.invalid                     the error output doesn't contain the word "error"
----------------------------------------
Ran 1 tests, of which 1 failed.
249
A full report for each test can be found at mjtest/reports/30-10-16_08-05-10.failed
Johannes Bechberger's avatar
Johannes Bechberger committed
250 251 252 253 254 255 256 257
```
- … log that some test cases were executed correctly
- … return with an error code of `0` if all tests executed correct

Contributions
-------------
__Please contribute to this test runner and the accompanied test cases.__

Johannes Bechberger's avatar
Johannes Bechberger committed
258 259 260 261 262
To add test cases just open a pull request. 
The test cases must have unique names (in each mode folder) and can also be placed into sub folders.
All contributions are reviewed but should be accepted in general (even if this means that there are duplicate test cases).

Issues regarding tests, the test runner or the preprocessor are also appreciated.
Johannes Bechberger's avatar
Johannes Bechberger committed
263

264 265 266 267 268
MiniJava style recommendations
------------------------------

### General
- All code should be valid Java code
Johannes Bechberger's avatar
Johannes Bechberger committed
269 270
    - If code isn't valid Java code it should be located in a `.mj` file
    - All other code files should end with `.java`
Johannes Bechberger's avatar
Johannes Bechberger committed
271 272 273 274
- All functionality that could be useful for other exec test cases should be extracted into a importable files
- Such importable files should be placed in a folder (or sub folder) of `exec/lib`
    - the class `BooleanUtils` is an example for such an importable class
    - it can be used in test cases via `import lib.BooleanUtils.java`
275 276 277 278 279 280 281 282 283 284 285
- Classes and methods should be documented with Javadocs if there name isn't self explanatory

### Class member
- The name of private class members should start with and underscore
- Instance variables should be initialized (if needed) in the `init` method

### Class
- The constructor functionality should be implemented in an `init` method that returns `this` at the end
- If you need multiple constructors, implement a basic constructor in the `init` method and implement the other constructors via `initX` methods


Johannes Bechberger's avatar
Johannes Bechberger committed
286 287 288
Licence
-------
MIT, see LICENCE file for more information.