README.mdwn 5.02 KB
Newer Older
Johannes Bechberger's avatar
Johannes Bechberger committed
1 2 3 4 5 6 7
MJTest
======

A test runner (and suite) for the MiniJava compiler (and its parts) written in the compiler lab of the KIT.

It's heavily inspired by Sisyphus (and uses some of its code)

8
__Please contribute test cases.__
Johannes Bechberger's avatar
Johannes Bechberger committed
9

Johannes Bechberger's avatar
Johannes Bechberger committed
10 11 12
*There's also some other additional code in this repo, that might provide fuzzing capabilities. Please consider contributing
such code as it probably helps the other teams (and could later be integrated into a bigger test suite).*

Johannes Bechberger's avatar
Johannes Bechberger committed
13 14 15 16 17
Test modes
----------
The test cases are divided in 3 'modes':
- __syntax__: Test cases that just check whether `./run --parsecheck` accepts as correct or rejects
 them.
Johannes Bechberger's avatar
Johannes Bechberger committed
18
- __ast__: Test cases that check the generated ast.
Johannes Bechberger's avatar
Johannes Bechberger committed
19 20 21 22 23 24 25 26 27 28
- __semantic__: Test cases that check semantic checking of MiniJava programs
- __exec__: Test cases that check the correct compilation of MiniJava programs.

_Only the syntax mode is currently usable, but the other three will follow._

The test different test cases for each mode are located in a folder with the same name.
The default directory that contains all test folders is `tests`.

The different types a test cases are differentiated by their file endings.

29
Side note: An error code greater than 0 should result in an errror message on error output containing the word `error`.
Johannes Bechberger's avatar
Johannes Bechberger committed
30

Johannes Bechberger's avatar
Johannes Bechberger committed
31

Johannes Bechberger's avatar
Johannes Bechberger committed
32 33 34 35 36 37
Test types for the syntax mode
------------------------------

<table>
<tr><th>File ending(s) of test cases</th><th>Expected behaviour to complete a test of this type</th></tr>
<tr>
Johannes Bechberger's avatar
Johannes Bechberger committed
38
    <td><code>.valid.mj</code> <code>.mj</code>
Johannes Bechberger's avatar
Johannes Bechberger committed
39 40 41 42 43 44
    <td>Return code is <code>0</code>, i.e. the MiniJava is accepted as syntactically correct</td>
</tr>
<tr>
    <td><code>.invalid.mj</code>
    <td>Return code is <code>&gt; 0</code> and the error output contains the word <code>error</code></td>
</tr>
45 46
<tr>
    <td><code>.java</code>
Johannes Bechberger's avatar
Johannes Bechberger committed
47
    <td>If <code>javac</code> accepts the syntax of the file then the MiniJava compiler
48 49
    should accept its syntax too.</td>
</tr>
Johannes Bechberger's avatar
Johannes Bechberger committed
50 51
</table>

Johannes Bechberger's avatar
Johannes Bechberger committed
52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68

Test types for the ast mode
------------------------------

<table>
<tr><th>File ending(s) of test cases</th><th>Expected behaviour to complete a test of this type</th></tr>
<tr>
    <td><code>.valid.mj</code> <code>.mj</code>
    <td>Return code is <code>0</code> and the output matches the expected output (located in the file `[test file].out`</td>
</tr>
<tr>
    <td><code>.invalid.mj</code>
    <td>Return code is <code>&gt; 0</code> and the error output contains the word <code>error</code></td>
</tr>
</table>


Johannes Bechberger's avatar
Johannes Bechberger committed
69 70 71 72 73 74
Test runner
-----------

### Requirements
The following programs are required (and executable by simply calling their names).
- `python3` (at least Python3.3)
75
- `javac` and `java` (for `.java` test cases)
Johannes Bechberger's avatar
Johannes Bechberger committed
76 77 78

### Installation

79
Just clone this repository and you're done:
Johannes Bechberger's avatar
Johannes Bechberger committed
80 81 82 83 84 85

```sh
git clone https://github.com/mj3-16/mjtest
cd mjtest
```

86 87 88
To get colored output install the python3 module `termcolor`.


Johannes Bechberger's avatar
Johannes Bechberger committed
89
### Usage
90
Output of the `./mjt.py --help`
Johannes Bechberger's avatar
Johannes Bechberger committed
91 92

```
93
usage: mjt.py [-h] [--only_incorrect_tests] [--produce_no_reports]
94 95
              [--produce_all_reports] [--parallel]
              [--output_no_incorrect_reports] [--log_level LOG_LEVEL]
Johannes Bechberger's avatar
Johannes Bechberger committed
96
              {syntax,ast,semantic,exec} MJ_RUN
Johannes Bechberger's avatar
Johannes Bechberger committed
97 98 99 100

MiniJava test runner

positional arguments:
Johannes Bechberger's avatar
Johannes Bechberger committed
101
  {syntax,ast,semantic,exec}
Johannes Bechberger's avatar
Johannes Bechberger committed
102
                        What do you want to test?
103 104 105
  MJ_RUN                Command to run your MiniJava implementation, e.g.
                        `mj/run`, can be omitted by assigning the environment
                        variable MJ_RUN
Johannes Bechberger's avatar
Johannes Bechberger committed
106 107 108 109 110

optional arguments:
  -h, --help            show this help message and exit
  --only_incorrect_tests
                        Only run the tests that were incorrect the last run
111 112
  --produce_no_reports  Produce no long reports besides the command line
                        output
113 114
  --produce_all_reports
                        Produce reports also for correct test cases
Johannes Bechberger's avatar
Johannes Bechberger committed
115
  --parallel            Run the tests in parallel
116 117 118 119
  --output_no_incorrect_reports
                        Output the long report for every incorrect test case
  --log_level LOG_LEVEL
                        Logging level (error, warn, info or debug)
Johannes Bechberger's avatar
Johannes Bechberger committed
120 121
```

122 123
For Windows users: Using environment variables doesn't work and you have to use prefix `./mjt.py` with `python`.

Johannes Bechberger's avatar
Johannes Bechberger committed
124 125 126
### Example usage
Assuming you want to run the syntax tests and your MiniJava base folder is `~/code/mj` then run
```
127
MJ_RUN="~/code/mj/run" ./mjt.py syntax
Johannes Bechberger's avatar
Johannes Bechberger committed
128 129 130 131 132
```
This will…
- … create reports in a folder named after the current date and time inside the `reports` folder
- … output something like
```
Johannes Bechberger's avatar
Johannes Bechberger committed
133 134 135
[FAIL   ] syntax:test.invalid                     the error output doesn't contain the word "error"
----------------------------------------
Ran 1 tests, of which 1 failed.
136
A full report for each test can be found at mjtest/reports/30-10-16_08-05-10.failed
Johannes Bechberger's avatar
Johannes Bechberger committed
137 138 139 140 141 142 143 144 145 146 147 148 149
```
- … log that some test cases were executed correctly
- … return with an error code of `0` if all tests executed correct

Contributions
-------------
__Please contribute to this test runner and the accompanied test cases.__

To add test cases just open a pull request. The test cases must have unique names (in each mode folder).

Licence
-------
MIT, see LICENCE file for more information.