README.mdwn 3.94 KB
Newer Older
Johannes Bechberger's avatar
Johannes Bechberger committed
1 2 3 4 5 6 7
MJTest
======

A test runner (and suite) for the MiniJava compiler (and its parts) written in the compiler lab of the KIT.

It's heavily inspired by Sisyphus (and uses some of its code)

8
__Please contribute test cases.__
Johannes Bechberger's avatar
Johannes Bechberger committed
9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24

Test modes
----------
The test cases are divided in 3 'modes':
- __syntax__: Test cases that just check whether `./run --parsecheck` accepts as correct or rejects
 them.
- __semantic__: Test cases that check semantic checking of MiniJava programs
- __exec__: Test cases that check the correct compilation of MiniJava programs.

_Only the syntax mode is currently usable, but the other three will follow._

The test different test cases for each mode are located in a folder with the same name.
The default directory that contains all test folders is `tests`.

The different types a test cases are differentiated by their file endings.

25
Side note: An error code greater than 0 should result in an errror message on error output containing the word `error`.
Johannes Bechberger's avatar
Johannes Bechberger committed
26 27 28 29 30 31 32 33 34 35 36 37 38 39

Test types for the syntax mode
------------------------------

<table>
<tr><th>File ending(s) of test cases</th><th>Expected behaviour to complete a test of this type</th></tr>
<tr>
    <td><code>.valid.mj</code><code>.mj</code>
    <td>Return code is <code>0</code>, i.e. the MiniJava is accepted as syntactically correct</td>
</tr>
<tr>
    <td><code>.invalid.mj</code>
    <td>Return code is <code>&gt; 0</code> and the error output contains the word <code>error</code></td>
</tr>
40 41
<tr>
    <td><code>.java</code>
42
    <td>If <code>javac</code> accepts the syntax of the file then the MiniJava compiler
43 44
    should accept its syntax too.</td>
</tr>
Johannes Bechberger's avatar
Johannes Bechberger committed
45 46 47 48 49 50 51 52
</table>

Test runner
-----------

### Requirements
The following programs are required (and executable by simply calling their names).
- `python3` (at least Python3.3)
53
- `javac` and `java` (for `.java` test cases)
Johannes Bechberger's avatar
Johannes Bechberger committed
54 55 56

### Installation

57
Just clone this repository and you're done:
Johannes Bechberger's avatar
Johannes Bechberger committed
58 59 60 61 62 63

```sh
git clone https://github.com/mj3-16/mjtest
cd mjtest
```

64 65 66
To get colored output install the python3 module `termcolor`.


Johannes Bechberger's avatar
Johannes Bechberger committed
67
### Usage
68
Output of the `./mjt.py --help`
Johannes Bechberger's avatar
Johannes Bechberger committed
69 70

```
71 72
usage: mjt.py [-h] [--only_incorrect_tests] [--produce_no_reports]
              [--parallel] [--log_level LOG_LEVEL]
73
              {syntax,semantic,exec} MJ_RUN
Johannes Bechberger's avatar
Johannes Bechberger committed
74 75 76 77 78 79

MiniJava test runner

positional arguments:
  {syntax,semantic,exec}
                        What do you want to test?
80 81 82
  MJ_RUN                Command to run your MiniJava implementation, e.g.
                        `mj/run`, can be omitted by assigning the environment
                        variable MJ_RUN
Johannes Bechberger's avatar
Johannes Bechberger committed
83 84 85 86 87

optional arguments:
  -h, --help            show this help message and exit
  --only_incorrect_tests
                        Only run the tests that were incorrect the last run
88 89
  --produce_no_reports  Produce no long reports besides the command line
                        output
Johannes Bechberger's avatar
Johannes Bechberger committed
90 91 92 93
  --parallel            Run the tests in parallel
  --log LOG             Logging level (error, warn, info or debug)
```

94 95
For Windows users: Using environment variables doesn't work and you have to use prefix `./mjt.py` with `python`.

Johannes Bechberger's avatar
Johannes Bechberger committed
96 97 98
### Example usage
Assuming you want to run the syntax tests and your MiniJava base folder is `~/code/mj` then run
```
99
MJ_RUN="~/code/mj/run" ./mjt.py syntax
Johannes Bechberger's avatar
Johannes Bechberger committed
100 101 102 103 104
```
This will…
- … create reports in a folder named after the current date and time inside the `reports` folder
- … output something like
```
Johannes Bechberger's avatar
Johannes Bechberger committed
105 106 107 108
[FAIL   ] syntax:test.invalid                     the error output doesn't contain the word "error"
----------------------------------------
Ran 1 tests, of which 1 failed.
A full report for each test can be found at mjtest/reports/30.10.16:08:05:10.failed
Johannes Bechberger's avatar
Johannes Bechberger committed
109 110 111 112 113 114 115 116 117 118 119 120 121
```
- … log that some test cases were executed correctly
- … return with an error code of `0` if all tests executed correct

Contributions
-------------
__Please contribute to this test runner and the accompanied test cases.__

To add test cases just open a pull request. The test cases must have unique names (in each mode folder).

Licence
-------
MIT, see LICENCE file for more information.