README.mdwn 4.17 KB
Newer Older
Johannes Bechberger's avatar
Johannes Bechberger committed
1 2 3 4 5 6 7
MJTest
======

A test runner (and suite) for the MiniJava compiler (and its parts) written in the compiler lab of the KIT.

It's heavily inspired by Sisyphus (and uses some of its code)

8
__Please contribute test cases.__
Johannes Bechberger's avatar
Johannes Bechberger committed
9

Johannes Bechberger's avatar
Johannes Bechberger committed
10 11 12
*There's also some other additional code in this repo, that might provide fuzzing capabilities. Please consider contributing
such code as it probably helps the other teams (and could later be integrated into a bigger test suite).*

Johannes Bechberger's avatar
Johannes Bechberger committed
13 14 15 16 17 18 19 20 21 22 23 24 25 26 27
Test modes
----------
The test cases are divided in 3 'modes':
- __syntax__: Test cases that just check whether `./run --parsecheck` accepts as correct or rejects
 them.
- __semantic__: Test cases that check semantic checking of MiniJava programs
- __exec__: Test cases that check the correct compilation of MiniJava programs.

_Only the syntax mode is currently usable, but the other three will follow._

The test different test cases for each mode are located in a folder with the same name.
The default directory that contains all test folders is `tests`.

The different types a test cases are differentiated by their file endings.

28
Side note: An error code greater than 0 should result in an errror message on error output containing the word `error`.
Johannes Bechberger's avatar
Johannes Bechberger committed
29 30 31 32 33 34 35 36 37 38 39 40 41 42

Test types for the syntax mode
------------------------------

<table>
<tr><th>File ending(s) of test cases</th><th>Expected behaviour to complete a test of this type</th></tr>
<tr>
    <td><code>.valid.mj</code><code>.mj</code>
    <td>Return code is <code>0</code>, i.e. the MiniJava is accepted as syntactically correct</td>
</tr>
<tr>
    <td><code>.invalid.mj</code>
    <td>Return code is <code>&gt; 0</code> and the error output contains the word <code>error</code></td>
</tr>
43 44
<tr>
    <td><code>.java</code>
Johannes Bechberger's avatar
Johannes Bechberger committed
45
    <td>If <code>javac</code> accepts the syntax of the file then the MiniJava compiler
46 47
    should accept its syntax too.</td>
</tr>
Johannes Bechberger's avatar
Johannes Bechberger committed
48 49 50 51 52 53 54 55
</table>

Test runner
-----------

### Requirements
The following programs are required (and executable by simply calling their names).
- `python3` (at least Python3.3)
56
- `javac` and `java` (for `.java` test cases)
Johannes Bechberger's avatar
Johannes Bechberger committed
57 58 59

### Installation

60
Just clone this repository and you're done:
Johannes Bechberger's avatar
Johannes Bechberger committed
61 62 63 64 65 66

```sh
git clone https://github.com/mj3-16/mjtest
cd mjtest
```

67 68 69
To get colored output install the python3 module `termcolor`.


Johannes Bechberger's avatar
Johannes Bechberger committed
70
### Usage
71
Output of the `./mjt.py --help`
Johannes Bechberger's avatar
Johannes Bechberger committed
72 73

```
74 75
usage: mjt.py [-h] [--only_incorrect_tests] [--produce_no_reports]
              [--parallel] [--log_level LOG_LEVEL]
76
              {syntax,semantic,exec} MJ_RUN
Johannes Bechberger's avatar
Johannes Bechberger committed
77 78 79 80 81 82

MiniJava test runner

positional arguments:
  {syntax,semantic,exec}
                        What do you want to test?
83 84 85
  MJ_RUN                Command to run your MiniJava implementation, e.g.
                        `mj/run`, can be omitted by assigning the environment
                        variable MJ_RUN
Johannes Bechberger's avatar
Johannes Bechberger committed
86 87 88 89 90

optional arguments:
  -h, --help            show this help message and exit
  --only_incorrect_tests
                        Only run the tests that were incorrect the last run
91 92
  --produce_no_reports  Produce no long reports besides the command line
                        output
Johannes Bechberger's avatar
Johannes Bechberger committed
93 94 95 96
  --parallel            Run the tests in parallel
  --log LOG             Logging level (error, warn, info or debug)
```

97 98
For Windows users: Using environment variables doesn't work and you have to use prefix `./mjt.py` with `python`.

Johannes Bechberger's avatar
Johannes Bechberger committed
99 100 101
### Example usage
Assuming you want to run the syntax tests and your MiniJava base folder is `~/code/mj` then run
```
102
MJ_RUN="~/code/mj/run" ./mjt.py syntax
Johannes Bechberger's avatar
Johannes Bechberger committed
103 104 105 106 107
```
This will…
- … create reports in a folder named after the current date and time inside the `reports` folder
- … output something like
```
Johannes Bechberger's avatar
Johannes Bechberger committed
108 109 110 111
[FAIL   ] syntax:test.invalid                     the error output doesn't contain the word "error"
----------------------------------------
Ran 1 tests, of which 1 failed.
A full report for each test can be found at mjtest/reports/30.10.16:08:05:10.failed
Johannes Bechberger's avatar
Johannes Bechberger committed
112 113 114 115 116 117 118 119 120 121 122 123 124
```
- … log that some test cases were executed correctly
- … return with an error code of `0` if all tests executed correct

Contributions
-------------
__Please contribute to this test runner and the accompanied test cases.__

To add test cases just open a pull request. The test cases must have unique names (in each mode folder).

Licence
-------
MIT, see LICENCE file for more information.