README.mdwn 9.22 KB
Newer Older
Johannes Bechberger's avatar
Johannes Bechberger committed
1
2
3
4
5
MJTest
======

A test runner (and suite) for the MiniJava compiler (and its parts) written in the compiler lab of the KIT.

Johannes Bechberger's avatar
Johannes Bechberger committed
6
It's heavily inspired by Sisyphus (and uses some of its code). See the `preproc` folder more information on the included preprocessor.
Johannes Bechberger's avatar
Johannes Bechberger committed
7

Johannes Bechberger's avatar
Johannes Bechberger committed
8
__Please contribute test cases and improve the existing ones.__
Johannes Bechberger's avatar
Johannes Bechberger committed
9

10
__Semantic test cases that are executable (and terminate) should be placed in the `exec` folder or sub folder and should follow the style recommendations from end of this document.__
Johannes Bechberger's avatar
Johannes Bechberger committed
11
12

__Exec test cases can now use the import statement to import other classes. See the exec test case folder and the `preproc` folder for more information__
Johannes Bechberger's avatar
Johannes Bechberger committed
13

Johannes Bechberger's avatar
Johannes Bechberger committed
14
15
16
*There's also some other additional code in this repo, that might provide fuzzing capabilities. Please consider contributing
such code as it probably helps the other teams (and could later be integrated into a bigger test suite).*

Johannes Bechberger's avatar
Johannes Bechberger committed
17
18
Test modes
----------
19
20
The test cases are divided in 5 'modes':
- __lexer__: Test cases that check the lexed token (and their correct output)
Johannes Bechberger's avatar
Johannes Bechberger committed
21
22
- __syntax__: Test cases that just check whether `./run --parsecheck` accepts as correct or rejects
 them.
Johannes Bechberger's avatar
Johannes Bechberger committed
23
- __ast__: Test cases that check the generated ast by using the pretty printing functionality.
Johannes Bechberger's avatar
Johannes Bechberger committed
24
- __semantic__: Test cases that check semantic checking of MiniJava programs
Johannes Bechberger's avatar
Johannes Bechberger committed
25
- __compile-firm__: Test cases that check the correct compilation and execution of MiniJava programs with the libfirm backend.
Johannes Bechberger's avatar
Johannes Bechberger committed
26

Johannes Bechberger's avatar
Johannes Bechberger committed
27
28
The test different test cases for each mode are located in a folder with the same name. 
Except the compile-firm test cases which are located in the `exec` folder.
Johannes Bechberger's avatar
Johannes Bechberger committed
29
30
31

The default directory that contains all test folders is `tests`.

32
33
Sub folders in side the test case folders are allowed.

Johannes Bechberger's avatar
Johannes Bechberger committed
34
35
The different types a test cases are differentiated by their file endings.

36
Side note: An error code greater than 0 should result in an error message on error output containing the word `error`.
Johannes Bechberger's avatar
Johannes Bechberger committed
37

Johannes Bechberger's avatar
Johannes Bechberger committed
38

39
40
41
42
43
44
Test types for the lexer mode
------------------------------

<table>
<tr><th>File ending(s) of test cases</th><th>Expected behaviour to complete a test of this type</th></tr>
<tr>
45
    <td><code>.valid.mj</code> <code>.mj</code></td>
Johannes Bechberger's avatar
Johannes Bechberger committed
46
    <td>Return code is <code>0</code> and the output matches the expected output (located in the file `[test file].out`)</td>
47
48
49
50
51
52
53
54
</tr>
<tr>
    <td><code>.invalid.mj</code>
    <td>Return code is <code>&gt; 0</code> and the error output contains the word <code>error</code></td>
</tr>
</table>


Johannes Bechberger's avatar
Johannes Bechberger committed
55
56
57
58
59
60
Test types for the syntax mode
------------------------------

<table>
<tr><th>File ending(s) of test cases</th><th>Expected behaviour to complete a test of this type</th></tr>
<tr>
61
    <td><code>.valid.mj</code> <code>.mj</code> <code>.valid.java</code> <code>.java</code>
Johannes Bechberger's avatar
Johannes Bechberger committed
62
63
64
    <td>Return code is <code>0</code>, i.e. the MiniJava is accepted as syntactically correct</td>
</tr>
<tr>
65
    <td><code>.invalid.mj</code> <code>.invalid.java</code></td>
Johannes Bechberger's avatar
Johannes Bechberger committed
66
67
68
69
    <td>Return code is <code>&gt; 0</code> and the error output contains the word <code>error</code></td>
</tr>
</table>

70
It uses all semantic mode tests implicitly if `--ci_testing` is given.
Johannes Bechberger's avatar
Johannes Bechberger committed
71
72
73
74
75
76
77

Test types for the ast mode
------------------------------

<table>
<tr><th>File ending(s) of test cases</th><th>Expected behaviour to complete a test of this type</th></tr>
<tr>
78
    <td><code>.valid.mj</code> <code>.mj</code> <code>.valid.java</code> <code>.java</code></td>
Johannes Bechberger's avatar
Johannes Bechberger committed
79
    <td>Pretty printing the source file should result in the same output as pretty printing the already pretty printed file.</td>
Johannes Bechberger's avatar
Johannes Bechberger committed
80
81
82
</tr>
</table>

Johannes Bechberger's avatar
Johannes Bechberger committed
83
84
It uses all syntax mode tests implicitly.

Johannes Bechberger's avatar
Johannes Bechberger committed
85

86
87
88
89
90
91
Test types for the semantic mode
------------------------------

<table>
<tr><th>File ending(s) of test cases</th><th>Expected behaviour to complete a test of this type</th></tr>
<tr>
Johannes Bechberger's avatar
Johannes Bechberger committed
92
    <td><code>.valid.mj</code> <code>.mj</code> <code>.valid.java</code> <code>.java</code></td>
93
94
95
96
97
98
99
100
    <td>Return code is <code>0</code>, i.e. the MiniJava is accepted as semantically correct</td>
</tr>
<tr>
    <td><code>.invalid.mj</code> <code>.invalid.java</code></td>
    <td>Return code is <code>&gt; 0</code> and the error output contains the word <code>error</code></td>
</tr>
</table>

Johannes Bechberger's avatar
Johannes Bechberger committed
101
__All semantic mode tests have to be syntactically correct___
102
103


Johannes Bechberger's avatar
Johannes Bechberger committed
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
Test types for the compile-firm mode
------------------------------------

<table>
<tr><th>File ending(s) of test cases</th><th>Expected behaviour to complete a test of this type</th></tr>
<tr>
    <td><code>.java</code> <code>.mj</code></td>
    <td>The test file should compile. The resulting binary should be runnable and its output should be:<br/>
    if a <code>[test file].out</code> exists: the content of this file<br/>
    if no such file exists: the same as the execution of the file with <code>java</code></td>
</tr>
</table>

__All compile-firm mode tests have to be semantically correct___

__All bigger test cases (e.g. for benchmarking) should be placed into the `big` folder__


Johannes Bechberger's avatar
Johannes Bechberger committed
122
123
124
125
126
Test runner
-----------

### Requirements
The following programs are required (and executable by simply calling their names).
127
- `python3` (at least python3.5, older python3 versions might work with the `typing` package installed)
128
- `javac` and `java` (for `.java` test cases)
Johannes Bechberger's avatar
Johannes Bechberger committed
129
130
131

### Installation

132
Just clone this repository and you're done:
Johannes Bechberger's avatar
Johannes Bechberger committed
133
134
135
136
137
138

```sh
git clone https://github.com/mj3-16/mjtest
cd mjtest
```

139
140
141
To get colored output install the python3 module `termcolor`.


Johannes Bechberger's avatar
Johannes Bechberger committed
142
### Usage
143
Output of the `./mjt.py --help`
Johannes Bechberger's avatar
Johannes Bechberger committed
144
145

```
Johannes Bechberger's avatar
Johannes Bechberger committed
146
147
usage: mjt.py [-h] [--only_incorrect_tests] [--all_exec_tests]
              [--produce_no_reports] [--produce_all_reports] [--parallel]
148
              [--output_no_incorrect_reports] [--color] [--ci_testing]
149
              [--log_level LOG_LEVEL]
Johannes Bechberger's avatar
Johannes Bechberger committed
150
              {all,lexer,syntax,ast,semantic,compile-firm,exec} MJ_RUN
Johannes Bechberger's avatar
Johannes Bechberger committed
151
152
153
154

MiniJava test runner

positional arguments:
155
  {all,lexer,syntax,ast,semantic,compile-firm,exec}
Johannes Bechberger's avatar
Johannes Bechberger committed
156
                        What do you want to test?
157
158
159
  MJ_RUN                Command to run your MiniJava implementation, e.g.
                        `mj/run`, can be omitted by assigning the environment
                        variable MJ_RUN
Johannes Bechberger's avatar
Johannes Bechberger committed
160
161
162
163
164

optional arguments:
  -h, --help            show this help message and exit
  --only_incorrect_tests
                        Only run the tests that were incorrect the last run
Johannes Bechberger's avatar
Johannes Bechberger committed
165
166
  --all_exec_tests      Run all exec (compile-firm...) tests, not only the
                        small ones
167
168
  --produce_no_reports  Produce no long reports besides the command line
                        output
169
170
  --produce_all_reports
                        Produce reports also for correct test cases
Johannes Bechberger's avatar
Johannes Bechberger committed
171
  --parallel            Run the tests in parallel
172
173
  --output_no_incorrect_reports
                        Output the long report for every incorrect test case
174
175
  --color               Output color output even if the output goes to into a
                        file
176
177
178
179
  --ci_testing          In mode X the succeeding test cases of later
                        modes/phases should also succeed in this mode, and
                        failing test cases of prior modes/phases should also
                        fail in this phase.
180
181
  --log_level LOG_LEVEL
                        Logging level (error, warn, info or debug)
Johannes Bechberger's avatar
Johannes Bechberger committed
182
183
```

184
185
For Windows users: Using environment variables doesn't work and you have to use prefix `./mjt.py` with `python`.

Johannes Bechberger's avatar
Johannes Bechberger committed
186
187
188
### Example usage
Assuming you want to run the syntax tests and your MiniJava base folder is `~/code/mj` then run
```
189
MJ_RUN="~/code/mj/run" ./mjt.py syntax
Johannes Bechberger's avatar
Johannes Bechberger committed
190
191
192
193
194
```
This will…
- … create reports in a folder named after the current date and time inside the `reports` folder
- … output something like
```
Johannes Bechberger's avatar
Johannes Bechberger committed
195
196
197
[FAIL   ] syntax:test.invalid                     the error output doesn't contain the word "error"
----------------------------------------
Ran 1 tests, of which 1 failed.
198
A full report for each test can be found at mjtest/reports/30-10-16_08-05-10.failed
Johannes Bechberger's avatar
Johannes Bechberger committed
199
200
201
202
203
204
205
206
```
- … log that some test cases were executed correctly
- … return with an error code of `0` if all tests executed correct

Contributions
-------------
__Please contribute to this test runner and the accompanied test cases.__

Johannes Bechberger's avatar
Johannes Bechberger committed
207
208
209
210
211
To add test cases just open a pull request. 
The test cases must have unique names (in each mode folder) and can also be placed into sub folders.
All contributions are reviewed but should be accepted in general (even if this means that there are duplicate test cases).

Issues regarding tests, the test runner or the preprocessor are also appreciated.
Johannes Bechberger's avatar
Johannes Bechberger committed
212

213
214
215
216
217
MiniJava style recommendations
------------------------------

### General
- All code should be valid Java code
Johannes Bechberger's avatar
Johannes Bechberger committed
218
219
    - If code isn't valid Java code it should be located in a `.mj` file
    - All other code files should end with `.java`
220
221
222
223
224
225
226
227
228
229
230
231
232
- All functionality that could be useful for other test cases should be extracted into an importable files
- Such importable files should be placed in a folder (or sub folder) of `lib`
- Classes and methods should be documented with Javadocs if there name isn't self explanatory

### Class member
- The name of private class members should start with and underscore
- Instance variables should be initialized (if needed) in the `init` method

### Class
- The constructor functionality should be implemented in an `init` method that returns `this` at the end
- If you need multiple constructors, implement a basic constructor in the `init` method and implement the other constructors via `initX` methods


Johannes Bechberger's avatar
Johannes Bechberger committed
233
234
235
Licence
-------
MIT, see LICENCE file for more information.