MJTest ====== A test runner (and suite) for the MiniJava compiler (and its parts) written in the compiler lab of the KIT. It's heavily inspired by Sisyphus (and uses some of its code). See the `preproc` folder more information on the included preprocessor. __Please contribute test cases and improve the existing ones.__ __Semantic test cases that are executable (and terminate) should be placed in the `exec` folder or sub folder and should follow the style recommendations from end of this document.__ __Exec test cases can now use the import statement to import other classes. See the exec test case folder and the `preproc` folder for more information__ *There's also some other additional code in this repo, that might provide fuzzing capabilities. Please consider contributing such code as it probably helps the other teams (and could later be integrated into a bigger test suite).* **The test cases should be committed in the [mjtest-tests](https://git.scc.kit.edu/IPDSnelting/mjtest-tests) repository** Test modes ---------- The test cases are divided in 6 'modes': - __lexer__: Test cases that check the lexed token (and their correct output) - __syntax__: Test cases that just check whether `./run --parsecheck` accepts as correct or rejects them. - __ast__: Test cases that check the generated ast by using the pretty printing functionality. - __semantic__: Test cases that check semantic checking of MiniJava programs - __compile-firm-only__: Test cases that check the compilation of MiniJava programs with the libfirm backend. - __compile-firm__: Test cases that check the correct compilation and execution of MiniJava programs with the libfirm backend. - __compile-only__: Test cases that check the compilation MiniJava programs with the self implemented backend. - __compile__: Test cases that check the correct compilation and execution of MiniJava programs with the self implemented backend. The test different test cases for each mode are located in a folder with the same name. Except the compile-firm test cases which are located in the `exec` folder. The default directory that contains all test folders is `tests`. Sub folders in side the test case folders are allowed. The different types a test cases are differentiated by their file endings. Side note: An error code greater than 0 should result in an error message on error output containing the word `error`. Test types for the lexer mode ------------------------------
File ending(s) of test casesExpected behaviour to complete a test of this type
.valid.mj .mj Return code is 0 and the output matches the expected output (located in the file `[test file].out`)
.invalid.mj Return code is > 0 and the error output contains the word error
Test types for the syntax mode ------------------------------
File ending(s) of test casesExpected behaviour to complete a test of this type
.valid.mj .mj .valid.java .java Return code is 0, i.e. the MiniJava is accepted as syntactically correct
.invalid.mj .invalid.java Return code is > 0 and the error output contains the word error
It uses all semantic mode tests implicitly if `--ci_testing` is given. Test types for the ast mode ------------------------------
File ending(s) of test casesExpected behaviour to complete a test of this type
.valid.mj .mj .valid.java .java Pretty printing the source file should result in the same output as pretty printing the already pretty printed file.
_It uses all syntax mode tests implicitly._ Place the test files in the folder `ast`. Might not be all that useful, wasn't used at all in the two years ago. Test types for the semantic mode ------------------------------
File ending(s) of test casesExpected behaviour to complete a test of this type
.valid.mj .mj .valid.java .java Return code is 0, i.e. the MiniJava is accepted as semantically correct
.invalid.mj .invalid.java Return code is > 0 and the error output contains the word error
__All semantic mode tests have to be syntactically correct__ Test types for the compile-firm-only and compile-only mode ----------------------------------------------------------
File ending(s) of test casesExpected behaviour to complete a test of this type
.valid.mj .mj .valid.java .java Return code is 0 and the compiler produces an `a.out` file, i.e. the MiniJava code can be compiled
__All valid semantic test-cases are also used, the non-semantic compile-(firm)-only should be placed into the `compile-only` folder__ Test types for the compile-firm and compile mode ------------------------------------------------
File ending(s) of test casesExpected behaviour to complete a test of this type
.java .mj The test file should compile. The resulting binary should be runnable and its output should be:
if a [input file].out (for test cases with input) or [test file].out (else) exists: the content of this file
if no such file exists: the same as the execution of the file with java
.[number].inputc This a test case with an input. The input is interpreted as ASCII chars. It requires a corresponding .input.java or .input.mj file that is executed with this input. The rest is equivalent to normal test cases from the row above
.[number].input This is an input test case whichs input file contains space seperated decimal numbers that are used as input data.
.invalid.mj .invalid.java The test file should compile. The execution of the resulting binary should abort width a return code > 0
.inf.mj .inf.java The test file should run longer than the specified timeout. This causes long test run-times.
The test cases should be placed into the `exec` folder. __All compile-firm and compile mode tests have to be semantically correct.__ __All bigger test cases (e.g. for benchmarking) should be placed into the `big` folder.__ Test runner ----------- ### Requirements The following programs are required (and executable by simply calling their names). - `python3` (at least python3.5, older python3 versions might work with the `typing` package installed) - `javac` and `java` (for `.java` test cases) ### Installation Just clone this repository and you're done: ```sh git clone git@git.scc.kit.edu:IPDSnelting/mjtest.git cd mjtest git submodule update --init ``` To get colored output install the python3 module `termcolor`. ### Usage Output of the `./mjt.py --help` ```sh usage: mjt.py [-h] [--only_incorrect_tests] [--all_exec_tests] [--produce_no_reports] [--produce_all_reports] [--parallel] [--output_no_incorrect_reports] [--color] [--ci_testing] [--log_level LOG_LEVEL] {all,lexer,syntax,ast,semantic,compile-firm-only,compile-only, compile-firm,compile} MJ_RUN MiniJava test runner positional arguments: {all,lexer,syntax,ast,semantic,compile-firm-only,compile-only,compile-firm,exec} What do you want to test? MJ_RUN Command to run your MiniJava implementation, e.g. `mj/run`, can be omitted by assigning the environment variable MJ_RUN optional arguments: -h, --help show this help message and exit --only_incorrect_tests Only run the tests that were incorrect the last run --all_exec_tests Run all exec (compile-firm...) tests, not only the small ones --produce_no_reports Produce no long reports besides the command line output --produce_all_reports Produce reports also for correct test cases --parallel Run the tests in parallel --output_no_incorrect_reports Output the long report for every incorrect test case --color Output color output even if the output goes to into a file --ci_testing In mode X the succeeding test cases of later modes/phases should also succeed in this mode, and failing test cases of prior modes/phases should also fail in this phase. --log_level LOG_LEVEL Logging level (error, warn, info or debug) ``` For Windows users: Using environment variables doesn't work and you have to use prefix `./mjt.py` with `python`. The timeout for the execution of normal programs in seconds can be set via the environment variable `MJ_TIMEOUT` (default is `10`) and for `big` programs via `MJ_BIG_TIMEOUT` (default is `60`). ### Example usage Assuming you want to run the syntax tests and your MiniJava base folder is `~/code/mj` then run ``` MJ_RUN="~/code/mj/run" ./mjt.py syntax ``` This will… - … create reports in a folder named after the current date and time inside the `reports` folder - … output something like ``` [FAIL ] syntax:test.invalid the error output doesn't contain the word "error" ---------------------------------------- Ran 1 tests, of which 1 failed. A full report for each test can be found at mjtest/reports/30-10-16_08-05-10.failed ``` - … log that some test cases were executed correctly - … return with an error code of `0` if all tests executed correct Contributions ------------- __Please contribute to this test runner and the accompanied test cases.__ To add test cases just open a pull request. The test cases must have unique names (in each mode folder) and can also be placed into sub folders. All contributions are reviewed but should be accepted in general (even if this means that there are duplicate test cases). Issues regarding tests, the test runner or the preprocessor are also appreciated. MiniJava style recommendations ------------------------------ ### General - All code should be valid Java code - If code isn't valid Java code it should be located in a `.mj` file - All other code files should end with `.java` - All functionality that could be useful for other exec test cases should be extracted into a importable files - Such importable files should be placed in a folder (or sub folder) of `exec/lib` - the class `BooleanUtils` is an example for such an importable class - it can be used in test cases via `import lib.BooleanUtils.java` - Classes and methods should be documented with Javadocs if there name isn't self explanatory ### Class member - The name of private class members should start with and underscore - Instance variables should be initialized (if needed) in the `init` method ### Class - The constructor functionality should be implemented in an `init` method that returns `this` at the end - If you need multiple constructors, implement a basic constructor in the `init` method and implement the other constructors via `initX` methods Licence ------- MIT, see LICENCE file for more information.