Commit 68e430ed authored by BorjaEst's avatar BorjaEst
Browse files

Docs fixing to new package structure

parent 5892d5e7
Tests
==================================
Tests should run using tox_
Tests should run using tox_, an automation is used to simplify the
testing process.
.. _tox: https://tox.readthedocs.io/en/latest/
......@@ -12,12 +14,94 @@ To install it with pip use:
$ pip install tox
Tests are divided into two types:
- Black-Box tests: Based on pytest_ framework, test the functionality
of the application without peering into its internal structures or
workings.
- White tests: Based on unittest_ framework, test the internal
structures of the application.
.. _pytest: https://docs.pytest.org/en/stable/
.. _unittest: https://tox.readthedocs.io/en/latest/
To run White and Black-Box tests use:
.. code-block:: bash
$ tox tests o3skim/*.py
...
py36: commands succeeded
...
This command generates a complete test report together with a
coverage and pep8 report.
Black-Box tests:
----------------
The framework used is pytest_ to provide a simple syntax to
test all possible combinations from the user point of view.
Pytest detects directly all tests following the test_discovery_
naming conventions. Therefore all Black-Box tests should be
located on the **tests** folder at the package root and start
with **test**. For example *test_sources.py*.
More than 500 test combinations are generated using which otherwise
might not be feasible using other python test frameworks.
.. _pytest: https://docs.pytest.org/en/stable/
.. _test_discovery: https://docs.pytest.org/en/reorganize-docs/new-docs/user/naming_conventions.html
To run only Black-Box tests simply call tox followed by the
folder with the test location:
To start testing simply run:
.. code-block:: bash
$ tox tests
...
py36: commands succeeded
...
White tests:
------------
The framework used is unittest_, a simple and extended test framework
which ships by default together with python.
The usage is very simple and straight forward for simple test, but
the difficulty to parametrize and combine multiple test fixtures
makes it not suitable for Black-Box testing without a very complex
customization.
To simplify code usage and testing, the white tests should be located
on the same file than the function / class are supposed to test.
To run only White tests simply call tox followed by the module files
you would like to test. You can also use the wildcard '*' to selected
and test all python modules:
.. code-block:: bash
$ tox
$ tox o3skim/*.py
...
py36: commands succeeded
...
Coverage and Pep8 reports:
--------------------------
One of the benefits of tox test automation is the capability to
generate code reports during testing.
The last coverage report output is produced at **htmlcov** which
can be displayed in html format accessing to **index.html**.
The last Pep8 report produced by flake8 at the output file
**flake8.log**.
......@@ -3,25 +3,7 @@ o3skim package
.. automodule:: o3skim
:members:
:undoc-members:
:show-inheritance:
o3skim.standardization module
-----------------------------
.. automodule:: o3skim.standardization
:members:
:undoc-members:
:show-inheritance:
o3skim.utils module
------------------------------
.. automodule:: o3skim.utils
:members:
:undoc-members:
:show-inheritance:
Source,
load,
cd
......@@ -8,7 +8,6 @@ import logging
import warnings
import o3skim
from o3skim import utils
warnings.simplefilter(action='ignore', category=FutureWarning)
......@@ -37,15 +36,15 @@ if __name__ == '__main__':
# Configuration load
logging.info("Lookinf for config at: '%s'", args.sources_file)
config = utils.load(args.sources_file)
config = o3skim.load(args.sources_file)
# Create sources
logging.info("Loading data from './data' ")
with utils.cd("data"):
with o3skim.cd("data"):
ds = {name: o3skim.Source(name, collection) for
name, collection in config.items()}
# Skim output
logging.info("Skimming data to './output' ")
with utils.cd("output"):
with o3skim.cd("output"):
[source.skim(groupby=args.split_by) for source in ds.values()]
"""
Main package with classes and utilities to handle ozone data skimming.
O3as package with classes and utilities to handle ozone data skimming.
After the data are loaded and the instances created,
it is possible to use the internal methods to produce
the desired output.
The main usage of the package resides on the generation of "Source"
class instances from the loading of netCDF files though collection
descriptions generated by source files.
However, as the generation of collection descriptions in a form of dict
type might be difficult, a yaml file can be converted into the required
variable by using the util "load" with the path to the 'sources.yaml'
file as input.
Once the desired collection is formated in the shape of dict variable,
a source object can be created from using the class "Source". During
this step the model data is loaded into the source instance involving
on the background process the data sorting and standardization. Note
that those models with errors on the collection specification are not
loaded neither interrupt the loading process. However, notice of those
event is logged on strerr together with the stack execution info.
After the data has been loaded the object intstance correctly created,
it is possible to use the internal methods to generate a reduced output
on the current folder.
To simplify the management of the data and output directories, the
package offest a "cd" utility which can be used together with a "with"
statement to change temporary the location of the execution folder.
"""
from o3skim import extended_xarray
......@@ -12,3 +33,5 @@ from o3skim import standardization
from o3skim import utils
Source = source.Source
load = utils.load
cd = utils.cd
"""
Module in charge of source implementation.
Module in charge of Source class implementation.
Sources are responsible of loading the netCDF files from data and do
the standardization during the process.
Sources are responsible of loading netCDF collections from data and
do the standardization during the process. Each source is compose
therefore from 0 to N models which can be accessed as subscriptable
object by it's model name.
It also implement internal methods which can be used to operate the
model data. For example using the method "skim" generates a reduced
version of the models data on the current folder.
"""
import logging
......@@ -21,15 +27,19 @@ logger = logging.getLogger('source')
class Source:
"""Conceptual class for a data source. It is produced by the loading
and generation of internal instances from each data source model.
and standardization of multiple data models.
The current supported model variables are "tco3_zm" and "vmro3_zm",
which should contain the information on how to retrieve the data
from the netCDF collection.
:param name: Name to provide to the source. The folder name with the
skimmed output data is preceded with this name before '_'.
:param name: Name to provide to the source.
:type name: str
:param collections: Dictionary where each 'key' is a name and its
value another dictionary with the variables contained at this
model. See :class:`o3skim.Model` for further details.
:param collections: Dictionary where each 'key' is a model name
and its value another dictionary with the variable loading
statements for that model.
{name:str, paths: str, coordinates: dict}
:type collections: dict
"""
......@@ -51,9 +61,14 @@ class Source:
return list(self._models.keys())
def skim(self, groupby=None):
"""Request to skim all source data into the current folder
"""Request to skim all source data into the current folder.
The output is generated into multiple folder where
each model output is generated in a forder with the source
name defined at the source initialization followed by
'_' and the model name: "<source_name>_<model_name>"
:param groupby: How to group output (None, year, decade).
:param groupby: How to group output (None, 'year', 'decade').
:type groupby: str, optional
"""
for model in self._models:
......
......@@ -14,7 +14,20 @@ logger = logging.getLogger('o3skim.standardization')
@utils.return_on_failure("Error when loading '{0}'".format('tco3_zm'),
default=xr.Dataset())
def standardize_tco3(dataset, variable, coordinates):
"""Standardizes a tco3 dataset"""
"""Standardizes a tco3 dataset.
:param dataset: Dataset to standardize.
:type dataset: xarray.Dataset
:param variable: Variable name for the tco3 on the original dataset.
:type variable: str
:param coordinates: Coordinates map for tco3 variable.
:type coordinates: {'lon':str, 'lat':str, 'time':str}
:return: Standardized dataset.
:rtype: xarray.Dataset
"""
array = dataset[variable]
array.name = 'tco3_zm'
array = squeeze(array)
......@@ -32,7 +45,20 @@ def rename_coords_tco3(array, time, lat, lon):
@utils.return_on_failure("Error when loading '{0}'".format('vmro3_zm'),
default=xr.Dataset())
def standardize_vmro3(dataset, variable, coordinates):
"""Standardizes a vmro3 dataset"""
"""Standardizes a vmro3 dataset.
:param dataset: Dataset to standardize.
:type dataset: xarray.Dataset
:param variable: Variable name for the vmro3 on the original dataset.
:type variable: str
:param coordinates: Coordinates map for vmro3 variable.
:type coordinates: {'lon':str, 'lat':str, 'plev':str, 'time':str}
:return: Standardized dataset.
:rtype: xarray.Dataset
"""
array = dataset[variable]
array.name = 'vmro3_zm'
array = squeeze(array)
......
......@@ -53,6 +53,7 @@ def return_on_failure(message, default=None):
# Log error with stack using root (not utils)
logging.error(message, exc_info=True)
return default
applicator.__doc__ = function.__doc__
return applicator
return decorate
......
......@@ -26,7 +26,7 @@ def data_dir(tmpdir_factory):
for source in sources:
source_dir = data_dir.join(source)
os.mkdir(source_dir)
with o3skim.utils.cd(source_dir):
with o3skim.cd(source_dir):
if source == 'SourceMerged':
mockup_data.combined(year_line)
mockup_data.noise(name='merged_noise.nc')
......@@ -45,7 +45,7 @@ def config_file(request):
@pytest.fixture(scope='session')
def config_dict(config_file):
return o3skim.utils.load(config_file)
return o3skim.load(config_file)
# package fixtures --------------------------------------------------
......@@ -69,14 +69,14 @@ def source_name(request):
@pytest.fixture(scope='module')
def source(config_dict, source_name, data_dir):
with o3skim.utils.cd(data_dir):
with o3skim.cd(data_dir):
source = o3skim.Source(source_name, config_dict[source_name])
return source
@pytest.fixture(scope='module')
def skimmed(groupby, source, output_dir):
with o3skim.utils.cd(output_dir):
with o3skim.cd(output_dir):
source.skim(groupby=groupby)
yield groupby, source.name
......@@ -108,7 +108,7 @@ def variable(request):
@pytest.fixture()
def skimed_file(skimmed, model_name, variable, year):
groupby, source_name = skimmed
with o3skim.utils.cd("{}_{}".format(source_name, model_name)):
with o3skim.cd("{}_{}".format(source_name, model_name)):
if not groupby:
yield "{}.nc".format(variable)
if groupby == 'year':
......
......@@ -13,14 +13,14 @@ import xarray
class TestSource:
def test_constructor(self, config_dict, source_name, data_dir):
with o3skim.utils.cd(data_dir):
with o3skim.cd(data_dir):
source = o3skim.Source(source_name, config_dict[source_name])
assert type(source.name) is str
assert type(source.models) is list
assert source.models != []
def test_skimming(self, source, output_dir):
with o3skim.utils.cd(output_dir):
with o3skim.cd(output_dir):
assert None == source.skim(groupby=None)
assert None == source.skim(groupby='year')
assert None == source.skim(groupby='decade')
......@@ -36,7 +36,7 @@ class TestSource:
class TestExceptions:
def test_constructor(self, config_dict, source_name, data_dir):
with o3skim.utils.cd(data_dir):
with o3skim.cd(data_dir):
source = o3skim.Source(source_name, config_dict[source_name])
assert source.name == source_name
assert source.models == []
......
Supports Markdown
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment