Commit bdf6642c authored by borja.sanchis's avatar borja.sanchis
Browse files

Merge branch 'dev' into 'master'

Dev

See merge request !28
parents a959cc73 d068463d
......@@ -71,3 +71,4 @@ coverage.xml
.pytest_cache/
cover/
flake8.log
tests/data
\ No newline at end of file
......@@ -26,9 +26,9 @@
- [About](#about)
- [Build using docker](#build)
- [Run using udocker](#deployment)
- [Documentation](#doc)
- [Testing](#testing)
- [Documentation](https://o3as.readthedocs.io/en/latest)
- [Authors](#authors)
- [Acknowledgments](#acknowledgement)
- [TODO](https://git.scc.kit.edu/synergy.o3as/o3skim/-/issues)
# About <a name = "about"></a>
......@@ -109,16 +109,37 @@ $ udocker run --user=application o3skim --help
```
# Documentation <a name = "doc"></a>
- [TODO]()
# Testing <a name = "testing"></a>
There are 2 types of tests in this package.
On top, [tox](https://tox.readthedocs.io/en/latest/) automation is used to simplify the testing process.
To run white and black box tests use:
```sh
$ tox
```
# Authors <a name = "authors"></a>
- [@V.Kozlov](https://git.scc.kit.edu/eo9869) - TBD
- [@T.Kerzenmacher](https://git.scc.kit.edu/px5501) - TBD
- [@B.Esteban](https://git.scc.kit.edu/zr5094) - TBD
## BlackBox tests - [Unittest framework](https://docs.python.org/3/library/unittest.html)
Located inside package modules (./o3skim). This helps to test easily functions at low level. It helps the developer to ensure the function is creating has the expected behavior.
To run only white tests use:
```sh
$ tox o3skim
```
## BlackBox tests - [Pytest framework](https://docs.pytest.org/en/stable/)
Located inside tests package folder (./tests). Black box testing is used to test the system from a general overview of the application. Pytest framework is selected in order to provide a simple syntax to test all possible combinations from the user point of view.
# Acknowledgements <a name = "acknowledgement"></a>
-
To run only black box tests use:
```sh
$ tox tests
```
# Documentation <a name = "doc"></a>
All o3as project module documents can be found at [o3as.readthedocs.io](https://o3as.readthedocs.io/en/latest/).
# Authors <a name = "authors"></a>
- [@V.Kozlov](https://git.scc.kit.edu/eo9869)
- [@T.Kerzenmacher](https://git.scc.kit.edu/px5501)
- [@B.Esteban](https://git.scc.kit.edu/zr5094)
Tests
==================================
Tests should run using tox_
Tests should run using tox_, an automation is used to simplify the
testing process.
.. _tox: https://tox.readthedocs.io/en/latest/
......@@ -12,12 +14,89 @@ To install it with pip use:
$ pip install tox
Tests are divided into two types:
- Black-Box tests: Based on pytest_ framework, test the functionality
of the application without peering into its internal structures or
workings.
- White tests: Based on unittest_ framework, test the internal
structures of the application.
.. _pytest: https://docs.pytest.org/en/stable/
.. _unittest: https://tox.readthedocs.io/en/latest/
To start testing simply run:
To run White and Black-Box tests use:
.. code-block:: bash
$ tox
...
py36: commands succeeded
...
This command generates a complete test report together with a
coverage and pep8 report.
Black-Box tests:
----------------
The framework used is pytest_ to provide a simple syntax to
test all possible combinations from the user point of view.
Pytest detects directly all tests following the test_discovery_
naming conventions. Therefore all Black-Box tests should be
located on the **tests** folder at the package root and start
with **test**. For example *test_sources.py*.
.. _test_discovery: https://docs.pytest.org/en/reorganize-docs/new-docs/user/naming_conventions.html
More than 500 test combinations are generated using which otherwise
might not be feasible using other python test frameworks. To run
only Black-Box tests simply call tox followed by the folder with the
test location:
.. code-block:: bash
$ tox tests
...
py36: commands succeeded
...
White tests:
------------
The framework used is unittest_, a simple and extended test framework
which ships by default together with python.
The usage is very simple and straight forward for simple test, but
the difficulty to parametrize and combine multiple test fixtures
makes it not suitable for Black-Box testing without a very complex
customization.
To simplify code usage and testing, the white tests should be located
inside the package folder. To run only White tests simply call tox
followed by the package name:
.. code-block:: bash
$ tox o3skim
...
py36: commands succeeded
...
Coverage and Pep8 reports:
--------------------------
One of the benefits of tox test automation is the capability to
generate code reports during testing.
The last coverage report output is produced at **htmlcov** which
can be displayed in html format accessing to **index.html**.
The last Pep8 report produced by flake8 at the output file
**flake8.log**.
......@@ -3,25 +3,7 @@ o3skim package
.. automodule:: o3skim
:members:
:undoc-members:
:show-inheritance:
o3skim.standardization module
-----------------------------
.. automodule:: o3skim.standardization
:members:
:undoc-members:
:show-inheritance:
o3skim.utils module
------------------------------
.. automodule:: o3skim.utils
:members:
:undoc-members:
:show-inheritance:
Source,
load,
cd
......@@ -49,14 +49,46 @@ argument is configured at the :doc:`../getting_started/cli` call.
# [CUSTOMIZABLE_KEY -- MANDATORY]
CCMI-1:
# Source metadata; common to all models in this source
# [FIXED_KEY -- OPTIONAL]
metadata:
# A metadata information example related to the source
# [CUSTOMIZABLE_KEY -- OPTIONAL]
meta_0: Source metadata string example
# A metadata information example related to the source
# [CUSTOMIZABLE_KEY -- OPTIONAL]
meta_1: Source metadata example replaced later by model
# This is the preceded -y1- string at the output folder: '[x1]_[y1]'
# [CUSTOMIZABLE_KEY -- MANDATORY]
IPSL:
# Model metadata; Unique key values for this model
# [FIXED_KEY -- OPTIONAL]
metadata:
# A metadata information example related to the source
# [CUSTOMIZABLE_KEY -- OPTIONAL]
meta_1: Replaces the metadata from the source
# A metadata information example related to the source
# [CUSTOMIZABLE_KEY -- OPTIONAL]
meta_2: Model metadata string example
# Represents the information related to tco3 data
# [FIXED_KEY -- OPTIONAL]
tco3_zm:
# TCO3 metadata; metadata for variable tco3
# [FIXED_KEY -- OPTIONAL]
metadata:
# A tco3 metadata attribute example
# [CUSTOMIZABLE_KEY -- OPTIONAL]
meta_0: Structured as tco3_zm: -> meta_0:
# Variable name for tco3 array inside the dataset
# [FIXED_KEY -- MANDATORY]: [CORRECT_VALUE -- MANDATORY]
name: toz
......@@ -82,6 +114,14 @@ argument is configured at the :doc:`../getting_started/cli` call.
# [FIXED_KEY -- OPTIONAL]
vmro3_zm:
# VMRO3 metadata; metadata for variable vmro3
# [FIXED_KEY -- OPTIONAL]
metadata:
# A vmro3 metadata attribute example
# [CUSTOMIZABLE_KEY -- OPTIONAL]
meta_0: Structured as vmro3_zm: -> meta_0:
# Variable name for vmro3 array inside the dataset
# [FIXED_KEY -- MANDATORY]: [CORRECT_VALUE -- MANDATORY]
name: vmro3
......@@ -92,8 +132,8 @@ argument is configured at the :doc:`../getting_started/cli` call.
# Coordinates description for vmro3 data.
# [FIXED_KEY -- MANDATORY]:
coordinates:
# [FIXED_KEY -- MANDATORY]: [CORRECT_VALUE -- MANDATORY]
time: time
......@@ -129,95 +169,155 @@ key *path* should be the same for both variables. The output expected at
.. code-block:: yaml
# This is the preceded -x2- string at the output folder: '[x2]_[y-]'
# [CUSTOMIZABLE_KEY -- MANDATORY]
ECMWF:
# This is the preceded -x2- string at the output folder: '[x2]_[y-]'
# [CUSTOMIZABLE_KEY -- MANDATORY]
ECMWF:
# This is the preceded -y1- string at the output folder: '[x2]_[y1]'
# [CUSTOMIZABLE_KEY -- MANDATORY]
ERA-5:
# Source metadata; common to all models in this source
# [FIXED_KEY -- OPTIONAL]
metadata:
# Represents the information related to tco3 data
# [FIXED_KEY -- OPTIONAL]
tco3_zm:
# A metadata information example related to the source
# [CUSTOMIZABLE_KEY -- OPTIONAL]
meta_0: Source metadata string example
# Variable name for tco3 array inside the dataset
# [FIXED_KEY -- MANDATORY]: [CORRECT_VALUE -- MANDATORY]
name: tco3
# A metadata information example related to the source
# [CUSTOMIZABLE_KEY -- OPTIONAL]
meta_1: Source metadata example replaced later by model
# Reg expression, how to load the netCDF files
# [FIXED_KEY -- MANDATORY]: [CORRECT_VALUE -- MANDATORY]
paths: Ecmwf/Era5
# This is the preceded -y1- string at the output folder: '[x2]_[y1]'
# [CUSTOMIZABLE_KEY -- MANDATORY]
ERA-5:
# Coordinates description for tco3 data.
# [FIXED_KEY -- MANDATORY]:
coordinates:
# Model metadata; Unique key values for this model
# [FIXED_KEY -- OPTIONAL]
metadata:
# [FIXED_KEY -- MANDATORY]: [CORRECT_VALUE -- MANDATORY]
lat: latitude
# A metadata information example related to the source
# [CUSTOMIZABLE_KEY -- OPTIONAL]
meta_1: Replaces the metadata from the source
# [FIXED_KEY -- MANDATORY]: [CORRECT_VALUE -- MANDATORY]
lon: longitude
# A metadata information example related to the source
# [CUSTOMIZABLE_KEY -- OPTIONAL]
meta_2: Model metadata string example
# [FIXED_KEY -- MANDATORY]: [CORRECT_VALUE -- MANDATORY]
time: time
# Represents the information related to tco3 data
# [FIXED_KEY -- OPTIONAL]
tco3_zm:
# This is the preceded -y2- string at the output folder: '[x2]_[y2]'
# [CUSTOMIZABLE_KEY -- MANDATORY]
ERA-i:
# TCO3 metadata; metadata for variable tco3
# [FIXED_KEY -- OPTIONAL]
metadata:
# Represents the information related to tco3 data
# [FIXED_KEY -- OPTIONAL]
tco3_zm:
# A tco3 metadata attribute example
# [CUSTOMIZABLE_KEY -- OPTIONAL]
meta_0: Structured as tco3_zm: -> meta_0:
# Variable name for tco3 array inside the dataset
# [FIXED_KEY -- MANDATORY]: [CORRECT_VALUE -- MANDATORY]
name: toz
# Variable name for tco3 array inside the dataset
# [FIXED_KEY -- MANDATORY]: [CORRECT_VALUE -- MANDATORY]
name: tco3
# Reg expression, how to load the netCDF files
# [FIXED_KEY -- MANDATORY]: [CORRECT_VALUE -- MANDATORY]
paths: Ecmwf/Erai
# Reg expression, how to load the netCDF files
# [FIXED_KEY -- MANDATORY]: [CORRECT_VALUE -- MANDATORY]
paths: Ecmwf/Era5
# Coordinates description for tco3 data.
# [FIXED_KEY -- MANDATORY]:
coordinates:
# Coordinates description for tco3 data.
# [FIXED_KEY -- MANDATORY]:
coordinates:
# [FIXED_KEY -- MANDATORY]: [CORRECT_VALUE -- MANDATORY]
time: time
# [FIXED_KEY -- MANDATORY]: [CORRECT_VALUE -- MANDATORY]
lat: latitude
# [FIXED_KEY -- MANDATORY]: [CORRECT_VALUE -- MANDATORY]
lat: latitude
# [FIXED_KEY -- MANDATORY]: [CORRECT_VALUE -- MANDATORY]
lon: longitude
# [FIXED_KEY -- MANDATORY]: [CORRECT_VALUE -- MANDATORY]
lon: longitude
# [FIXED_KEY -- MANDATORY]: [CORRECT_VALUE -- MANDATORY]
time: time
# Represents the information related to vmro3 data
# [FIXED_KEY -- OPTIONAL]
vmro3_zm:
# This is the preceded -y2- string at the output folder: '[x2]_[y2]'
# [CUSTOMIZABLE_KEY -- MANDATORY]
ERA-i:
# Variable name for vmro3 array inside the dataset
# [FIXED_KEY -- MANDATORY]: [CORRECT_VALUE -- MANDATORY]
name: vmro3
# Model metadata; Unique key values for this model
# [FIXED_KEY -- OPTIONAL]
metadata:
# Reg expression, how to load the netCDF files
# [FIXED_KEY -- MANDATORY]: [CORRECT_VALUE -- MANDATORY]
paths: Ecmwf/Erai
# A metadata information example related to the source
# [CUSTOMIZABLE_KEY -- OPTIONAL]
meta_1: Replaces the metadata from the source
# Coordinates description for vmro3 data.
# [FIXED_KEY -- MANDATORY]:
# A metadata information example related to the source
# [CUSTOMIZABLE_KEY -- OPTIONAL]
meta_2: Model metadata string example
coordinates:
# [FIXED_KEY -- MANDATORY]: [CORRECT_VALUE -- MANDATORY]
time: time
# Represents the information related to tco3 data
# [FIXED_KEY -- OPTIONAL]
tco3_zm:
# [FIXED_KEY -- MANDATORY]: [CORRECT_VALUE -- MANDATORY]
plev: level
# TCO3 metadata; metadata for variable tco3
# [FIXED_KEY -- OPTIONAL]
metadata:
# [FIXED_KEY -- MANDATORY]: [CORRECT_VALUE -- MANDATORY]
lat: latitude
# A tco3 metadata attribute example
# [CUSTOMIZABLE_KEY -- OPTIONAL]
meta_0: Structured as tco3_zm: -> meta_0:
# Variable name for tco3 array inside the dataset
# [FIXED_KEY -- MANDATORY]: [CORRECT_VALUE -- MANDATORY]
name: toz
# [FIXED_KEY -- MANDATORY]: [CORRECT_VALUE -- MANDATORY]
lon: longitude
# Reg expression, how to load the netCDF files
# [FIXED_KEY -- MANDATORY]: [CORRECT_VALUE -- MANDATORY]
paths: Ecmwf/Erai
# Coordinates description for tco3 data.
# [FIXED_KEY -- MANDATORY]:
coordinates:
# [FIXED_KEY -- MANDATORY]: [CORRECT_VALUE -- MANDATORY]
time: time
# [FIXED_KEY -- MANDATORY]: [CORRECT_VALUE -- MANDATORY]
lat: latitude
# [FIXED_KEY -- MANDATORY]: [CORRECT_VALUE -- MANDATORY]
lon: longitude
# Represents the information related to vmro3 data
# [FIXED_KEY -- OPTIONAL]
vmro3_zm:
# VMRO3 metadata; metadata for variable vmro3
# [FIXED_KEY -- OPTIONAL]
metadata:
# A vmro3 metadata attribute example
# [CUSTOMIZABLE_KEY -- OPTIONAL]
meta_0: Structured as vmro3_zm: -> meta_0:
# Variable name for vmro3 array inside the dataset
# [FIXED_KEY -- MANDATORY]: [CORRECT_VALUE -- MANDATORY]
name: vmro3
# Reg expression, how to load the netCDF files
# [FIXED_KEY -- MANDATORY]: [CORRECT_VALUE -- MANDATORY]
paths: Ecmwf/Erai
# Coordinates description for vmro3 data.
# [FIXED_KEY -- MANDATORY]:
coordinates:
# [FIXED_KEY -- MANDATORY]: [CORRECT_VALUE -- MANDATORY]
time: time
# [FIXED_KEY -- MANDATORY]: [CORRECT_VALUE -- MANDATORY]
plev: level
# [FIXED_KEY -- MANDATORY]: [CORRECT_VALUE -- MANDATORY]
lat: latitude
# [FIXED_KEY -- MANDATORY]: [CORRECT_VALUE -- MANDATORY]
lon: longitude
One or two files?
-----------------
......
# For more information see the documentation at
# ./docs/user_guide/source-file.rst
CCMI-1:
IPSL:
SourceSplit:
metadata:
meta_0: Source metadata string example
ModelTCO3:
metadata:
meta_1: Model metadata string example
meta_2: 0
tco3_zm:
name: toz
paths: Ccmi/mon/toz/*.nc
name: tco3
paths: SourceSplit/tco3_????.nc
coordinates:
time: time
lat: lat
lon: lon
lat: latitude
lon: longitude
metadata:
meta_tco3_1: TCO3 metadata string example
meta_tco3_2: 0
ModelVMRO3:
metadata:
meta_1: Model metadata string example
meta_2: 0
vmro3_zm:
name: vmro3
paths: Ccmi/mon/toz/*.nc
paths: SourceSplit/vmro3_????.nc
coordinates:
time: time
plev: plev
lat: lat
lon: lon
plev: pressure_level
lat: latitude
lon: longitude
metadata:
meta_vmro3_1: VMRO3 metadata string example
meta_vmro3_2: 0
ModelALL:
metadata:
meta_1: Model metadata string example
meta_2: 0
tco3_zm:
name: tco3
paths: SourceSplit/tco3_????.nc
coordinates:
time: time
lat: latitude
lon: longitude
metadata:
meta_tco3_1: TCO3 metadata string example
meta_tco3_2: 0
vmro3_zm:
name: vmro3
paths: SourceSplit/vmro3_????.nc
coordinates:
time: time
plev: pressure_level
lat: latitude
lon: longitude
metadata:
meta_vmro3_1: VMRO3 metadata string example
meta_vmro3_2: 0
SourceMerged:
metadata:
meta_0: Source metadata string example
ModelTCO3:
metadata:
meta_1: Model metadata string example
meta_2: 0
tco3_zm:
name: tco3
paths: SourceMerged/merged_????.nc
coordinates:
time: time
lat: latitude
lon: longitude
metadata:
meta_tco3_1: TCO3 metadata string example
meta_tco3_2: 0
ModelVMRO3:
metadata:
meta_1: Model metadata string example
meta_2: 0
vmro3_zm:
name: vmro3
paths: SourceMerged/merged_????.nc
coordinates:
time: time
plev: pressure_level
lat: latitude
lon: longitude
metadata:
meta_vmro3_1: VMRO3 metadata string example
meta_vmro3_2: 0
ModelALL:
metadata:
meta_1: Model metadata string example
meta_2: 0
tco3_zm:
name: tco3
paths: SourceMerged/merged_????.nc
coordinates:
time: time
lat: latitude
lon: longitude
metadata:
meta_tco3_1: TCO3 metadata string example
meta_tco3_2: 0
vmro3_zm:
name: vmro3
paths: SourceMerged/merged_????.nc
coordinates:
time: time
plev: pressure_level
lat: latitude
lon: longitude
metadata:
meta_vmro3_1: VMRO3 metadata string example
meta_vmro3_2: 0
......@@ -8,7 +8,6 @@ import logging
import warnings
import o3skim
from o3skim import utils
warnings.simplefilter(action='ignore', category=FutureWarning)
......@@ -37,15 +36,15 @@ if __name__ == '__main__':
# Configuration load
logging.info("Lookinf for config at: '%s'", args.sources_file)
config = utils.load(args.sources_file)
config = o3skim.load(args.sources_file)
# Create sources
logging.info("Loading data from './data' ")
with utils.cd("data"):
ds = {name: o3skim.Source(name, collection) for
with o3skim.cd("data"):
ds = {name: o3skim.Source(name, **collection) for
name, collection in config.items()}
# Skim output
logging.info("Skimming data to './output' ")
with utils.cd("output"):
with o3skim.cd("output"):
[source.skim(groupby=args.split_by) for source in ds.