Commit 42301eac authored by BorjaEst's avatar BorjaEst
Browse files

Merge branch '24-generate-documentation'

parents 619b4b8d 4ec24945
// For format details, see https://aka.ms/vscode-remote/devcontainer.json or this file's README at:
// https://github.com/microsoft/vscode-dev-containers/tree/v0.134.1/containers/docker-existing-dockerfile
{
"name": "Existing Dockerfile",
// Sets the run context to one level up instead of the .devcontainer folder.
"context": "..",
// Update the 'dockerFile' property if you aren't using the standard 'Dockerfile' filename.
"dockerFile": "../Dockerfile",
// Set *default* container specific settings.json values on container create.
"settings": {
"terminal.integrated.shell.linux": "/bin/bash"
},
// Add the IDs of extensions you want installed when the container is created.
"extensions": [
"ms-python.python",
"lextudio.restructuredtext"
]
// Use 'forwardPorts' to make a list of ports inside the container available locally.
// "forwardPorts": [],
// Uncomment the next line to run commands after the container is created - for example installing curl.
// "postCreateCommand": "apt-get update && apt-get install -y curl",
// Uncomment when using a ptrace-based debugger like C++, Go, and Rust
// "runArgs": [ "--cap-add=SYS_PTRACE", "--security-opt", "seccomp=unconfined" ],
// Uncomment to use the Docker CLI from inside the container. See https://aka.ms/vscode-remote/samples/docker-from-docker.
// "mounts": [ "source=/var/run/docker.sock,target=/var/run/docker.sock,type=bind" ],
// Uncomment to connect as a non-root user if you've added one. See https://aka.ms/vscode-remote/containers/non-root.
// "remoteUser": "vscode"
}
......@@ -61,5 +61,5 @@ USER ${user}
# Start default script
ENTRYPOINT [ "main" ]
CMD [ "-v 1" ]
CMD [ "--verbosity ERROR" ]
......@@ -22,20 +22,15 @@
# 📝 Table of Contents
- [About](#about)
- [Getting Started](#getting_started)
- [Deployment](#deployment)
- [Built Using](#built_using)
- [Installing](#Installing)
- [Build using docker](#build)
- [Run using udocker](#deployment)
- [Documentation](#doc)
- [Authors](#authors)
- [Acknowledgments](#acknowledgement)
- [TODO](https://git.scc.kit.edu/synergy.o3as/o3skim/-/issues)
# About <a name = "about"></a>
This project provides the tools to preprocess, standarise and reduce ozone data for later transfer and plot.
# Getting Started <a name = "getting_started"></a>
See [deployment](#deployment) for notes on how to deploy the project on a live system.
This project provides the tools to preprocess, standardize and reduce ozone data for later transfer and plot.
## Prerequisites
To run the project as container, you need the following systems and container technologies:
......@@ -45,7 +40,7 @@ To run the project as container, you need the following systems and container te
> Note udocker cannot be used to build containers, only to run them.
## Built using docker <a name = "built_using"></a>
# Built using docker <a name = "build"></a>
Download the repository at the __Build machine__ using git.
```sh
$ git clone git@git.scc.kit.edu:synergy.o3as/o3skim.git
......@@ -59,7 +54,7 @@ $ docker build --tag o3skim .
Successfully built 69587025a70a
Successfully tagged o3skim:latest
```
If the build process succeded, you can list the image on the docker image list:
If the build process succeeded, you can list the image on the docker image list:
```sh
$ docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
......@@ -67,42 +62,14 @@ o3skim latest 69587025a70a xx se
...
```
## Running the tests <a name = "tests"></a>
To run tests, you need to install the tool in your system without docker.
As first step ensure you have the following dependencies:
- [python 3.8](https://www.python.org/downloads/release/python-385/)
- [pip 20.0.2](https://pypi.org/)
- [gcc](https://gcc.gnu.org/)
- [g++]()
After download and dependencies check, install with pip:
```sh
$ pip install -e .
```
Tests should run using
[tox](https://tox.readthedocs.io/en/latest/).
To install it with pip use:
```sh
$ pip install tox
```
To start testing simply run:
```sh
$ tox
...
py37: commands succeeded
py38: commands succeeded
```
# Deployment <a name = "deployment"></a>
# Run using udocker <a name = "deployment"></a>
To deploy the the application using __udocker__ at the __Runtime machine__ you need:
- Input path with data to skim, to be mounter on `/app/data` inside the container.
- Output path for skimmed results, to be mounted on `/app/output` inside the container.
- Configuration file with a data structure desctiption at the input path in [YAML](https://yaml.org/) format. This configuration file has to be mounted on `/app/sources.yaml` inside the container. See [sources_example.yaml](/sources_example.yaml) for a configuration example.
- Configuration file with a data structure description at the input path in [YAML](https://yaml.org/) format.
This configuration file has to be mounted on `/app/sources.yaml` inside the container.
Once the requierement are needed, pull the image from the image registry.
Once the requirement are completed, pull the image from the image registry.
For example, to pull it from the synergy-imk official registry use:
```sh
$ udocker pull synergyimk/o3skim
......@@ -140,10 +107,15 @@ $ udocker run --user=application o3skim --help
```
# Documentation <a name = "doc"></a>
- [TODO]()
# Authors <a name = "authors"></a>
- [@V.Kozlov](https://git.scc.kit.edu/eo9869) - TBD
- [@T.Kerzenmacher](https://git.scc.kit.edu/px5501) - TBD
- [@B.steban](https://git.scc.kit.edu/zr5094) - TBD
- [@B.Esteban](https://git.scc.kit.edu/zr5094) - TBD
# Acknowledgements <a name = "acknowledgement"></a>
-
......
### Documentation builds
_build
# Minimal makefile for Sphinx documentation
#
# You can set these variables from the command line, and also
# from the environment for the first two.
SPHINXOPTS ?=
SPHINXBUILD ?= sphinx-build
SOURCEDIR = .
BUILDDIR = _build
# Put it first so that "make" without argument is like "make help".
help:
@$(SPHINXBUILD) -M help "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O)
.PHONY: help Makefile
# Catch-all target: route all unknown targets to Sphinx using the new
# "make mode" option. $(O) is meant as a shortcut for $(SPHINXOPTS).
%: Makefile
@$(SPHINXBUILD) -M $@ "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O)
# Configuration file for the Sphinx documentation builder.
#
# This file only contains a selection of the most common options. For a full
# list see the documentation:
# https://www.sphinx-doc.org/en/master/usage/configuration.html
# -- Path setup --------------------------------------------------------------
# If extensions (or modules to document with autodoc) are in another directory,
# add these directories to sys.path here. If the directory is relative to the
# documentation root, use os.path.abspath to make it absolute, like shown here.
import os
import sys
sys.path.insert(0, os.path.abspath('..'))
# -- Project information -----------------------------------------------------
project = 'o3skim'
copyright = '2020, T.Kerzenmacher, V.Kozlov, B.Esteban'
author = 'T.Kerzenmacher, V.Kozlov, B.Esteban'
# The full version, including alpha/beta/rc tags
release = '0.1.0'
# -- General configuration ---------------------------------------------------
# Add any Sphinx extension module names here, as strings. They can be
# extensions coming with Sphinx (named 'sphinx.ext.*') or your custom
# ones.
extensions = [
'sphinx.ext.autodoc'
]
# Add any paths that contain templates here, relative to this directory.
templates_path = ['_templates']
# List of patterns, relative to source directory, that match files and
# directories to ignore when looking for source files.
# This pattern also affects html_static_path and html_extra_path.
exclude_patterns = ['_build', 'Thumbs.db', '.DS_Store']
# -- Options for HTML output -------------------------------------------------
# The theme to use for HTML and HTML Help pages. See the documentation for
# a list of builtin themes.
#
html_theme = 'alabaster'
# Add any paths that contain custom static files (such as style sheets) here,
# relative to this directory. They are copied after the builtin static files,
# so a file named "default.css" will overwrite the builtin "default.css".
html_static_path = ['_static']
\ No newline at end of file
Local installation
==================================
To run tests, you need to install the tool in your system without docker.
As first step ensure you have the following dependencies:
- python_ > 3.8
- pip_ > 20.0.2
- gcc_
- g++_
After downloading the repository and installing dependencies check, install with pip:
.. code-block:: bash
$ pip install -e .
.. _python: https://www.python.org
.. _pip: https://pypi.org
.. _gcc: https://gcc.gnu.org
.. _g++:
Tests
==================================
Tests should run using tox_
.. _tox: https://tox.readthedocs.io/en/latest/
To install it with pip use:
.. code-block:: bash
$ pip install tox
To start testing simply run:
.. code-block:: bash
$ tox
...
py37: commands succeeded
py38: commands succeeded
Build
===================
Download the code from the o3skim_ repository at the **Build machine**.
For example, using git_:
.. code-block:: bash
$ git clone git@git.scc.kit.edu:synergy.o3as/o3skim.git
Cloning into 'o3skim'...
...
.. _o3skim: https://git.scc.kit.edu/synergy.o3as/o3skim
.. _git: https://git-scm.com/
Build the container image at the **Build machine**.
For example, using docker_:
.. code-block:: bash
$ docker build --tag o3skim .
...
Successfully built 69587025a70a
Successfully tagged o3skim:latest
.. _docker: https://docs.docker.com/engine/reference/commandline/build
If the build process succeeded, then you should see the image name on the container images list:
.. code-block:: bash
$ docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
o3skim latest 69587025a70a xx seconds ago 557MB
...
To use your new generated image on the **Runtime machine**, the easiest way is to
push to a dockerhub repository. For example, with docker_:
.. code-block:: bash
$ docker push <repository>/o3skim:<tag>
The push refers to repository [docker.io/........./o3skim]
...
7e84795fccac: Preparing
7e84795fccac: Layer already exists
ffaeb20d9e23: Layer already exists
4cdd6a90e552: Layer already exists
3e0762bebc71: Layer already exists
1e441fe06d90: Layer already exists
98ff2784e9f5: Layer already exists
2b99e2403063: Layer already exists
d0f104dc0a1f: Layer already exists
...: digest: sha256:...................... size: 2004
If you do not have internet access from the **Build machine** or **Runtime machine**
it is also possible to use `docker save`_ to export your images.
.. _`docker save`: https://docs.docker.com/engine/reference/commandline/save/
Command Line Interface
=======================
Usage:
.. code-block:: bash
usage: main [-h] [-f SOURCES_FILE] [-s {year,decade}]
[-v {DEBUG,INFO,WARNING,ERROR,CRITICAL}]
To run the the application using **udocker** at the **Runtime machine**
you need to provide the following volumes to the container:
- --volume, mount `/app/data`: Input path with data to skim.
- --volume, mount `/app/output`: Output path for skimmed results.
- --volume, mount `/app/sources.yaml`: Configuration file with a data structure
description at the input path in YAML_ format.
See :doc:`/user_guide/source-file` for a configuration example.
.. _YAML: https://yaml.org/
Also, in the specific case of udocker_, it is needed to specify that the
user `application` should run inside the container:
.. _udocker: https://indigo-dc.gitbook.io/udocker
For example,to run the container using udocker_ use the following:
.. code-block:: bash
$ udocker run \
--user=application \
--volume=${PWD}/sources.yaml:/app/sources.yaml \
--volume=${PWD}/data:/app/data \
--volume=${PWD}/output:/app/output \
o3skim --verbosity INFO
...
executing: main
...
2020-08-25 12:42:34,151 - INFO - Configuration found at: './sources.yaml'
2020-08-25 12:42:34,152 - INFO - Loading data from './data'
2020-08-25 12:42:34,261 - INFO - Skimming data to './output'
For the main function description and commands help you can call:
.. code-block:: bash
$ udocker run --user=application o3skim --help
As optional arguments, it is possible to indicate:
- -h, --help: show this help message and exit
- -f, --sources_file SOURCES_FILE: Custom sources YAML configuration. (default: ./sources.yaml)
- -s, --split_by {year,decade}: Period time to split output (default: None)
- -v, --verbosity {DEBUG,INFO,WARNING,ERROR,CRITICAL}: Sets the logging level (default: ERROR)
Note that SOURCES_FILE is only modified for development purposes as usually any
file from host can be mounted using the container directive '--volume'.
Deployment
==================================
To deploy the the application using **udocker** at the **Runtime machine**
you need the o3skim container image.
The easiest way to deploy in your **Runtime machine** is by pulling the image
from a remote registry. You can use the official registry at synergyimk_ or use
the instructions at :doc:`build` to create your image and uploaded at your own registry.
.. _synergyimk: https://hub.docker.com/r/synergyim
Once you decide from which registry download, pull the image that image registry.
For example, to pull it from the synergy-imk official registry use:
.. code-block:: bash
$ udocker pull synergyimk/o3skim
...
Downloading layer: sha256:......
...
Note it is also possible to use `udocker load`_ to import images generated by
`docker save`_.
.. _`udocker load`: https://indigo-dc.gitbook.io/udocker/user_manual#1-4-basic-flow
.. _`docker save`: https://docs.docker.com/engine/reference/commandline/save
Once the image is downloaded or imported, create the local container.
For example, if it was downloaded from synergyimk registry you can use:
.. code-block:: bash
$ udocker create --name=o3skim synergyimk/o3skim
fa42a912-b0d4-3bfb-987f-1c243863802d
Check the containers
available at the **Runtime machine**:
.. code-block:: bash
$ udocker ps
CONTAINER ID P M NAMES IMAGE
...
fa42a912-b0d4-3bfb-987f-1c243863802d . W ['o3skim'] synergyimk/o3skim:latest
Now you are ready to start using the container as `o3skim`. Read how to use the
:doc:`cli` as first steps to skim your data.
Prerequisites
==================================
To run the project as container, you need the following systems and container technologies:
- **Build machine** with docker_ in case you want/need to build your own image.
- **Runtime machine** with udocker_ and access to the data to skim.
In case you do not want to create your image, last images are uploaded in dockerhub
at synergyimk_.
.. rubric:: Note udocker_ cannot build containers but run them.
.. _docker: https://docs.docker.com/engine/install/
.. _udocker: https://indigo-dc.gitbook.io/udocker/installation_manual
.. _synergyimk: https://hub.docker.com/r/synergyimk
=========================================
o3skim
=========================================
.. rubric:: Data skimming for ozone assessment.
**o3skim** is an open source project and Python package that provides the
tools to pre-process, standardize and reduce ozone data from netCDF_ models to
simplify and speed up ozone data transfer and plot.
.. _netCDF: http://www.unidata.ucar.edu/software/netcdf
Documentation
-------------
**Getting started**
* Which :doc:`getting_started/prerequisites` you need to start.
* How to :doc:`getting_started/build` your o3skim container.
* How to :doc:`getting_started/deployment` your o3skim container.
* How to use the o3skim :doc:`getting_started/cli`
.. toctree::
:maxdepth: 2
:hidden:
:caption: Getting started
getting_started/prerequisites
getting_started/build
getting_started/deployment
getting_started/cli
**User guide**
* Create your :doc:`user_guide/source-file` to point what to skim.
* Learn how to skim by reading the :doc:`user_guide/o3skim`
.. toctree::
:maxdepth: 2
:hidden:
:caption: User guide:
user_guide/source-file
user_guide/o3skim
**Developer guide**
* How to do a :doc:`dev_guide/local-install`
* How to run and create new :doc:`dev_guide/tests`
.. toctree::
:maxdepth: 2
:hidden:
:caption: Developer guide:
dev_guide/local-install
dev_guide/tests
See also
--------
* :ref:`genindex`
* :ref:`modindex`
* :ref:`search`
Authors
-------
- `@V.Kozlov <https://git.scc.kit.edu/eo9869>`_ - TBD
- `@T.Kerzenmacher <https://git.scc.kit.edu/px5501>`_ - TBD
- `@B.Esteban <https://git.scc.kit.edu/zr5094>`_ - TBD
Acknowledgements
----------------
@ECHO OFF
pushd %~dp0
REM Command file for Sphinx documentation
if "%SPHINXBUILD%" == "" (
set SPHINXBUILD=sphinx-build
)
set SOURCEDIR=.
set BUILDDIR=_build
if "%1" == "" goto help
%SPHINXBUILD% >NUL 2>NUL
if errorlevel 9009 (
echo.
echo.The 'sphinx-build' command was not found. Make sure you have Sphinx
echo.installed, then set the SPHINXBUILD environment variable to point
echo.to the full path of the 'sphinx-build' executable. Alternatively you
echo.may add the Sphinx directory to PATH.
echo.
echo.If you don't have Sphinx installed, grab it from
echo.http://sphinx-doc.org/
exit /b 1
)
%SPHINXBUILD% -M %1 %SOURCEDIR% %BUILDDIR% %SPHINXOPTS% %O%
goto end
:help
%SPHINXBUILD% -M help %SOURCEDIR% %BUILDDIR% %SPHINXOPTS% %O%
:end
popd
o3skim package
==============
.. automodule:: o3skim
:members:
:undoc-members:
:show-inheritance:
o3skim.sources module
---------------------
.. automodule:: o3skim.sources
:members:
:undoc-members:
:show-inheritance:
o3skim.utils module
-------------------
.. automodule:: o3skim.utils
:members:
:undoc-members:
:show-inheritance: