README.md 5.56 KB
Newer Older
BorjaEst's avatar
BorjaEst committed
1
2
3
4
5
<p align="center">
  <a href="" rel="noopener">
 <img width=200px height=200px src="https://i.imgur.com/6wj0hh6.jpg" alt="Project logo"></a>
</p>

BorjaEst's avatar
BorjaEst committed
6
<h3 align="center">o3skim</h3>
BorjaEst's avatar
BorjaEst committed
7
8
9

<div align="center">

10
11
  [![Build Status](https://jenkins.eosc-synergy.eu/buildStatus/icon?job=eosc-synergy-org%2Fo3skim%2Ftest)](https://jenkins.eosc-synergy.eu/job/eosc-synergy-org/job/o3skim/job/test/)
  [![Documentation Status](https://readthedocs.org/projects/o3as/badge/?version=latest)](https://o3as.readthedocs.io/en/latest/?badge=latest)
12
13
14
15
  [![pipeline status](https://git.scc.kit.edu/synergy.o3as/o3skim/badges/master/pipeline.svg)](https://git.scc.kit.edu/synergy.o3as/o3skim/-/commits/master)
  [![coverage status](https://git.scc.kit.edu/synergy.o3as/o3skim/badges/master/coverage.svg)](https://git.scc.kit.edu/synergy.o3as/o3skim/-/commits/master)
  [![License](https://img.shields.io/badge/license-GPL-blue.svg)](https://git.scc.kit.edu/synergy.o3as/o3skim/-/commits/master)
  [![Status](https://img.shields.io/badge/status-building-blue.svg)](https://git.scc.kit.edu/synergy.o3as/o3skim/-/commits/master) 
BorjaEst's avatar
BorjaEst committed
16
17
18
19
20

</div>

---

BorjaEst's avatar
BorjaEst committed
21
<p align="center"> Data pre-processing for ozone models 
BorjaEst's avatar
BorjaEst committed
22
23
24
    <br> 
</p>

BorjaEst's avatar
BorjaEst committed
25
# 📝 Table of Contents
BorjaEst's avatar
BorjaEst committed
26
- [About](#about)
27
28
- [Build using docker](#build)
- [Run using udocker](#deployment)
BorjaEst's avatar
BorjaEst committed
29
- [Testing](#testing)
30
- [Documentation](#doc)
BorjaEst's avatar
BorjaEst committed
31
32
- [Authors](#authors)
- [Acknowledgments](#acknowledgement)
BorjaEst's avatar
BorjaEst committed
33
- [TODO](https://git.scc.kit.edu/synergy.o3as/o3skim/-/issues)
BorjaEst's avatar
BorjaEst committed
34

35
# About <a name = "about"></a>
36
This project provides the tools to preprocess, standardize and reduce ozone data for later transfer and plot. 
BorjaEst's avatar
BorjaEst committed
37

BorjaEst's avatar
BorjaEst committed
38
## Prerequisites
39
40
41
To run the project as container, you need the following systems and container technologies:
- __Build machine__ with [docker](https://docs.docker.com/engine/install/) 
- __Runtime machine__ with [udocker](https://indigo-dc.gitbook.io/udocker/installation_manual)
BorjaEst's avatar
BorjaEst committed
42

BorjaEst's avatar
BorjaEst committed
43
> Note udocker cannot be used to build containers, only to run them. 
BorjaEst's avatar
BorjaEst committed
44
45


46
# Built using docker <a name = "build"></a>
47
Download the repository at the __Build machine__ using git.
BorjaEst's avatar
BorjaEst committed
48
49
```sh
$ git clone git@git.scc.kit.edu:synergy.o3as/o3skim.git
50
51
Cloning into 'o3skim'...
...
BorjaEst's avatar
BorjaEst committed
52
```
53
Build the docker image at the __Build machine__ using docker.
BorjaEst's avatar
BorjaEst committed
54
55
56
57
58
```sh
$ docker build --tag o3skim .
...
Successfully built 69587025a70a
Successfully tagged o3skim:latest
BorjaEst's avatar
BorjaEst committed
59
```
60
If the build process succeeded, you can list the image on the docker image list:
BorjaEst's avatar
BorjaEst committed
61
62
63
64
65
```sh
$ docker images
REPOSITORY                         TAG                 IMAGE ID            CREATED              SIZE
o3skim                             latest              69587025a70a        xx seconds ago      557MB
...
BorjaEst's avatar
BorjaEst committed
66
67
```

68
# Run using udocker <a name = "deployment"></a>
69
70
71
To deploy the the application using __udocker__ at the __Runtime machine__ you need:
 - Input path with data to skim, to be mounter on `/app/data` inside the container.
 - Output path for skimmed results, to be mounted on `/app/output` inside the container.
72
73
 - Configuration file with a data structure description at the input path in [YAML](https://yaml.org/) format.
   This configuration file has to be mounted on `/app/sources.yaml` inside the container.
74

75
Once the requirement are completed, pull the image from the image registry.
76
77
78
79
80
81
82
83
84
For example, to pull it from the synergy-imk official registry use:
```sh
$ udocker pull synergyimk/o3skim
...
Downloading layer: sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4
...
```

Once the repository is added and the image downloaded, create the local container: 
BorjaEst's avatar
BorjaEst committed
85
```sh
86
87
88
89
90
91
92
93
94
95
96
$ udocker create --name=o3skim synergyimk/o3skim
fa42a912-b0d4-3bfb-987f-1c243863802d
```

Finally, run the container. Note the described _data_, _output_ and _sources.yaml_ have to be provided. Also it is needed to specify the user _application_ should run inside the container:
```sh
$ udocker run \
  --user=application \
  --volume=${PWD}/sources.yaml:/app/sources.yaml \
  --volume=${PWD}/data:/app/data \
  --volume=${PWD}/output:/app/output \
BorjaEst's avatar
BorjaEst committed
97
98
  o3skim --verbosity INFO
...
99
100
101
102
103
104
105
106
107
108
109
executing: main
...
2020-08-25 12:42:34,151 - INFO     - Configuration found at: './sources.yaml'
2020-08-25 12:42:34,152 - INFO     - Loading data from './data' 
2020-08-25 12:42:34,261 - INFO     - Skimming data to './output' 
```

For the main function description and commands help you can call:
```sh  
$ udocker run --user=application o3skim --help
...
BorjaEst's avatar
BorjaEst committed
110
111
112
```


BorjaEst's avatar
BorjaEst committed
113
114
115
# Testing <a name = "testing"></a>
There are 2 types of tests in this package.
On top, [tox](https://tox.readthedocs.io/en/latest/) automation is used to simplify the testing process.
116

BorjaEst's avatar
BorjaEst committed
117
To run white and black box tests use:
BorjaEst's avatar
BorjaEst committed
118
119
120
```sh
$ tox tests o3skim/*.py
```
121

BorjaEst's avatar
BorjaEst committed
122
123
124
125
126
127
128
## BlackBox tests - [Unittest framework](https://docs.python.org/3/library/unittest.html)
Located inside package modules (./o3skim). This helps to test easily functions at low level. It helps the developer to ensure the function is creating has the expected behavior.

To run only white tests use:
```sh
$ tox o3skim/*.py
```
BorjaEst's avatar
BorjaEst committed
129

BorjaEst's avatar
BorjaEst committed
130
131
## BlackBox tests - [Pytest framework](https://docs.pytest.org/en/stable/)
Located inside tests package folder (./tests). Black box testing is used to test the system from a general overview of the application. Pytest framework is selected in order to provide a simple syntax to test all possible combinations form the user point of view.
132

BorjaEst's avatar
BorjaEst committed
133
134
135
136
137
138
139
140
141
142
143
144
145
To run only black box tests use:
```sh
$ tox tests
```

# Documentation <a name = "doc"></a>
All o3as project module documents can be found at [o3as.readthedocs.io](https://o3as.readthedocs.io/en/latest/). 


# Authors <a name = "authors"></a>
- [@V.Kozlov](https://git.scc.kit.edu/eo9869)
- [@T.Kerzenmacher](https://git.scc.kit.edu/px5501)
- [@B.Esteban](https://git.scc.kit.edu/zr5094)
BorjaEst's avatar
BorjaEst committed
146