tests.rst 3.62 KB
Newer Older
BorjaEst's avatar
BorjaEst committed
1
Test guidelines
BorjaEst's avatar
BorjaEst committed
2
3
==================================

BorjaEst's avatar
BorjaEst committed
4
5
Testing is based on sqa-baseline_ criteria, tox_ automation is used to 
simplify the testing process.
6

BorjaEst's avatar
BorjaEst committed
7
.. _sqa-baseline: https://indigo-dc.github.io/sqa-baseline/
BorjaEst's avatar
BorjaEst committed
8
9
10
11
12
13
14
15
.. _tox: https://tox.readthedocs.io/en/latest/

To install it with pip use:

.. code-block:: bash
    
    $ pip install tox

BorjaEst's avatar
BorjaEst committed
16
To run unit and functional tests together with reports use:
17
18
19

.. code-block:: bash

BorjaEst's avatar
BorjaEst committed
20
    $ tox
21
    ...
BorjaEst's avatar
BorjaEst committed
22
23
24
25
    clean: commands succeeded
    stylecheck: commands succeeded
    bandit: commands succeeded
    docs: commands succeeded
26
    py36: commands succeeded
BorjaEst's avatar
BorjaEst committed
27
    report: commands succeeded
28
29
    ...

BorjaEst's avatar
BorjaEst committed
30
31
The last coverage report output is produced at **htmlcov** which 
can be displayed in html format accessing to **index.html**.
32

BorjaEst's avatar
BorjaEst committed
33
34
The last Pep8 report produced by flake8 at the output file
**flake8.log**.
35

BorjaEst's avatar
BorjaEst committed
36
37
However, you can also run different test configurations using 
different tox environments:
38
39


BorjaEst's avatar
BorjaEst committed
40
41
42
43
Code Style [QC.Sty]
-----------------------
Test pep8 maintenance style conventions based on pylint format. 
To run stylecheck use:
44

BorjaEst's avatar
BorjaEst committed
45
.. code-block:: bash
46

BorjaEst's avatar
BorjaEst committed
47
48
49
50
51
52
53
54
55
56
57
58
    $ tox -e stylecheck
    ...
    stylecheck: commands succeeded
    ...


Unit Testing [QC.Uni]
-----------------------
All unit tests are placed inside the package (./o3skim/test). This 
helps to test easily functions at low level and ensure the functions
have the expected behavior. 
To run unit tests use:
BorjaEst's avatar
BorjaEst committed
59

60
61
.. code-block:: bash

BorjaEst's avatar
BorjaEst committed
62
    $ tox -e unittesting
63
    ...
BorjaEst's avatar
BorjaEst committed
64
    unittesting: commands succeeded
65
66
67
    ...


BorjaEst's avatar
BorjaEst committed
68
69
70
71
This environment also provide a coverage term report for the tests.
The design of Unit Tests is based on the python unittest_ framework, 
a simple and extended test framework which ships by default together 
with python.
72

BorjaEst's avatar
BorjaEst committed
73
.. _unittest: https://tox.readthedocs.io/en/latest/
74

BorjaEst's avatar
BorjaEst committed
75
The usage is very simple and straight forward for simple tests, but
76
77
78
79
the difficulty to parametrize and combine multiple test fixtures 
makes it not suitable for Black-Box testing without a very complex
customization.

BorjaEst's avatar
BorjaEst committed
80
81
82
83
84
85

Functional Testing [QC.Fun]
---------------------------
Located inside tests package folder (./tests). Functional testing is
used to test the system from a general overview of the application.
To run functional tests use:
BorjaEst's avatar
BorjaEst committed
86
87
88

.. code-block:: bash

BorjaEst's avatar
BorjaEst committed
89
    $ tox -e functional
BorjaEst's avatar
BorjaEst committed
90
    ...
BorjaEst's avatar
BorjaEst committed
91
    functional: commands succeeded
92
93
    ...

BorjaEst's avatar
BorjaEst committed
94
95
96
This environment also provide a coverage term report for the tests.
The framework used is pytest_ to provide a simple syntax to test all 
possible combinations from the user point of view.
97

BorjaEst's avatar
BorjaEst committed
98
99
100
101
Pytest detects directly all tests following the test_discovery_
naming conventions. Therefore all functional tests should be 
located on the **tests** folder at the package root and start 
with **test**. For example *test_sources.py*.
102

BorjaEst's avatar
BorjaEst committed
103
104
.. _pytest: https://docs.pytest.org/en/stable/
.. _test_discovery: https://docs.pytest.org/en/reorganize-docs/new-docs/user/naming_conventions.html
105

BorjaEst's avatar
BorjaEst committed
106
107
More than 500 test combinations are generated using which otherwise 
might not be feasible using other python test frameworks.
108
109


BorjaEst's avatar
BorjaEst committed
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
Security [QC.Sec]
-----------------------
Security checks are performed by bandit_, a tool designed to find 
common security issues in Python code.
To run security checks use:

.. code-block:: bash

    $ tox -e functional
    ...
    functional: commands succeeded
    ...

.. _bandit: https://pypi.org/project/bandit/


Documentation [QC.Doc]
-----------------------
Documentation is build using sphinx_, a tool designed to create 
documentation based on code.
To run documentation build checks use:

.. code-block:: bash

    $ tox -e docs
    ...
    docs: commands succeeded
    ...

.. _sphinx: https://www.sphinx-doc.org/en/master/

The HTML pages are build inside in docs/_build.
BorjaEst's avatar
BorjaEst committed
142