README.mdwn 3.54 KB
Newer Older
Johannes Bechberger's avatar
Johannes Bechberger committed
1 2 3 4 5 6 7
MJTest
======

A test runner (and suite) for the MiniJava compiler (and its parts) written in the compiler lab of the KIT.

It's heavily inspired by Sisyphus (and uses some of its code)

8
__Please contribute test cases.__
Johannes Bechberger's avatar
Johannes Bechberger committed
9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47

Test modes
----------
The test cases are divided in 3 'modes':
- __syntax__: Test cases that just check whether `./run --parsecheck` accepts as correct or rejects
 them.
- __semantic__: Test cases that check semantic checking of MiniJava programs
- __exec__: Test cases that check the correct compilation of MiniJava programs.

_Only the syntax mode is currently usable, but the other three will follow._

The test different test cases for each mode are located in a folder with the same name.
The default directory that contains all test folders is `tests`.

The different types a test cases are differentiated by their file endings.


Test types for the syntax mode
------------------------------

<table>
<tr><th>File ending(s) of test cases</th><th>Expected behaviour to complete a test of this type</th></tr>
<tr>
    <td><code>.valid.mj</code><code>.mj</code>
    <td>Return code is <code>0</code>, i.e. the MiniJava is accepted as syntactically correct</td>
</tr>
<tr>
    <td><code>.invalid.mj</code>
    <td>Return code is <code>&gt; 0</code> and the error output contains the word <code>error</code></td>
</tr>
</table>


Test runner
-----------

### Requirements
The following programs are required (and executable by simply calling their names).
- `python3` (at least Python3.3)
48
- `javac` and `java` (for `.java` test cases)
Johannes Bechberger's avatar
Johannes Bechberger committed
49 50 51

### Installation

52
Just clone this repository and you're done:
Johannes Bechberger's avatar
Johannes Bechberger committed
53 54 55 56 57 58

```sh
git clone https://github.com/mj3-16/mjtest
cd mjtest
```

59 60 61
To get colored output install the python3 module `termcolor`.


Johannes Bechberger's avatar
Johannes Bechberger committed
62
### Usage
63
Output of the `./mjt.py --help`
Johannes Bechberger's avatar
Johannes Bechberger committed
64 65

```
66 67
usage: mjt.py [-h] [--only_incorrect_tests] [--parallel]
              [--log_level LOG_LEVEL]
68
              {syntax,semantic,exec} MJ_RUN
Johannes Bechberger's avatar
Johannes Bechberger committed
69 70 71 72 73 74

MiniJava test runner

positional arguments:
  {syntax,semantic,exec}
                        What do you want to test?
75 76 77
  MJ_RUN                Command to run your MiniJava implementation, e.g.
                        `mj/run`, can be omitted by assigning the environment
                        variable MJ_RUN
Johannes Bechberger's avatar
Johannes Bechberger committed
78 79 80 81 82 83 84 85 86

optional arguments:
  -h, --help            show this help message and exit
  --only_incorrect_tests
                        Only run the tests that were incorrect the last run
  --parallel            Run the tests in parallel
  --log LOG             Logging level (error, warn, info or debug)
```

87 88
For Windows users: Using environment variables doesn't work and you have to use prefix `./mjt.py` with `python`.

Johannes Bechberger's avatar
Johannes Bechberger committed
89 90 91
### Example usage
Assuming you want to run the syntax tests and your MiniJava base folder is `~/code/mj` then run
```
92
MJ_RUN="~/code/mj/run" ./mjt.py syntax
Johannes Bechberger's avatar
Johannes Bechberger committed
93 94 95 96 97
```
This will…
- … create reports in a folder named after the current date and time inside the `reports` folder
- … output something like
```
Johannes Bechberger's avatar
Johannes Bechberger committed
98 99 100 101
[FAIL   ] syntax:test.invalid                     the error output doesn't contain the word "error"
----------------------------------------
Ran 1 tests, of which 1 failed.
A full report for each test can be found at mjtest/reports/30.10.16:08:05:10.failed
Johannes Bechberger's avatar
Johannes Bechberger committed
102 103 104 105 106 107 108 109 110 111 112 113 114
```
- … log that some test cases were executed correctly
- … return with an error code of `0` if all tests executed correct

Contributions
-------------
__Please contribute to this test runner and the accompanied test cases.__

To add test cases just open a pull request. The test cases must have unique names (in each mode folder).

Licence
-------
MIT, see LICENCE file for more information.