README.md 4.91 KB
Newer Older
BorjaEst's avatar
BorjaEst committed
1
2
3
4
5
<p align="center">
  <a href="" rel="noopener">
 <img width=200px height=200px src="https://i.imgur.com/6wj0hh6.jpg" alt="Project logo"></a>
</p>

BorjaEst's avatar
BorjaEst committed
6
<h3 align="center">o3skim</h3>
BorjaEst's avatar
BorjaEst committed
7
8
9

<div align="center">

10
  [![Build Status](https://jenkins.eosc-synergy.eu/buildStatus/icon?job=eosc-synergy-org%2Fo3skim%2Ftest)](https://jenkins.eosc-synergy.eu/job/eosc-synergy-org/job/o3skim/job/master)
11
  [![Documentation Status](https://readthedocs.org/projects/o3as/badge/?version=latest)](https://o3as.readthedocs.io/en/latest/?badge=latest)
12
13
14
15
  [![pipeline status](https://git.scc.kit.edu/synergy.o3as/o3skim/badges/master/pipeline.svg)](https://git.scc.kit.edu/synergy.o3as/o3skim/-/commits/master)
  [![coverage status](https://git.scc.kit.edu/synergy.o3as/o3skim/badges/master/coverage.svg)](https://git.scc.kit.edu/synergy.o3as/o3skim/-/commits/master)
  [![License](https://img.shields.io/badge/license-GPL-blue.svg)](https://git.scc.kit.edu/synergy.o3as/o3skim/-/commits/master)
  [![Status](https://img.shields.io/badge/status-building-blue.svg)](https://git.scc.kit.edu/synergy.o3as/o3skim/-/commits/master) 
BorjaEst's avatar
BorjaEst committed
16
17
18
19
20

</div>

---

BorjaEst's avatar
BorjaEst committed
21
<p align="center"> Data pre-processing for ozone models 
BorjaEst's avatar
BorjaEst committed
22
23
24
    <br> 
</p>

BorjaEst's avatar
BorjaEst committed
25
# 📝 Table of Contents
BorjaEst's avatar
BorjaEst committed
26
- [About](#about)
27
28
- [Build using docker](#build)
- [Run using udocker](#deployment)
BorjaEst's avatar
BorjaEst committed
29
- [Testing](#testing)
BorjaEst's avatar
BorjaEst committed
30
- [Documentation](https://o3as.readthedocs.io/en/latest)
BorjaEst's avatar
BorjaEst committed
31
- [Authors](#authors)
BorjaEst's avatar
BorjaEst committed
32
- [TODO](https://git.scc.kit.edu/synergy.o3as/o3skim/-/issues)
BorjaEst's avatar
BorjaEst committed
33

34
# About <a name = "about"></a>
35
This project provides the tools to preprocess, standardize and reduce ozone data for later transfer and plot. 
BorjaEst's avatar
BorjaEst committed
36

BorjaEst's avatar
BorjaEst committed
37
## Prerequisites
38
39
40
To run the project as container, you need the following systems and container technologies:
- __Build machine__ with [docker](https://docs.docker.com/engine/install/) 
- __Runtime machine__ with [udocker](https://indigo-dc.gitbook.io/udocker/installation_manual)
BorjaEst's avatar
BorjaEst committed
41

BorjaEst's avatar
BorjaEst committed
42
> Note udocker cannot be used to build containers, only to run them. 
BorjaEst's avatar
BorjaEst committed
43
44


45
# Built using docker <a name = "build"></a>
46
Download the repository at the __Build machine__ using git.
BorjaEst's avatar
BorjaEst committed
47
48
```sh
$ git clone git@git.scc.kit.edu:synergy.o3as/o3skim.git
49
50
Cloning into 'o3skim'...
...
BorjaEst's avatar
BorjaEst committed
51
```
52
Build the docker image at the __Build machine__ using docker.
BorjaEst's avatar
BorjaEst committed
53
54
55
56
57
```sh
$ docker build --tag o3skim .
...
Successfully built 69587025a70a
Successfully tagged o3skim:latest
BorjaEst's avatar
BorjaEst committed
58
```
59
If the build process succeeded, you can list the image on the docker image list:
BorjaEst's avatar
BorjaEst committed
60
61
62
63
64
```sh
$ docker images
REPOSITORY                         TAG                 IMAGE ID            CREATED              SIZE
o3skim                             latest              69587025a70a        xx seconds ago      557MB
...
BorjaEst's avatar
BorjaEst committed
65
66
```

67
# Run using udocker <a name = "deployment"></a>
68
69
70
To deploy the the application using __udocker__ at the __Runtime machine__ you need:
 - Input path with data to skim, to be mounter on `/app/data` inside the container.
 - Output path for skimmed results, to be mounted on `/app/output` inside the container.
71
72
 - Configuration file with a data structure description at the input path in [YAML](https://yaml.org/) format.
   This configuration file has to be mounted on `/app/sources.yaml` inside the container.
73

74
Once the requirement are completed, pull the image from the image registry.
75
76
77
78
79
80
81
82
83
For example, to pull it from the synergy-imk official registry use:
```sh
$ udocker pull synergyimk/o3skim
...
Downloading layer: sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4
...
```

Once the repository is added and the image downloaded, create the local container: 
BorjaEst's avatar
BorjaEst committed
84
```sh
85
86
87
88
89
90
91
92
93
94
95
$ udocker create --name=o3skim synergyimk/o3skim
fa42a912-b0d4-3bfb-987f-1c243863802d
```

Finally, run the container. Note the described _data_, _output_ and _sources.yaml_ have to be provided. Also it is needed to specify the user _application_ should run inside the container:
```sh
$ udocker run \
  --user=application \
  --volume=${PWD}/sources.yaml:/app/sources.yaml \
  --volume=${PWD}/data:/app/data \
  --volume=${PWD}/output:/app/output \
BorjaEst's avatar
BorjaEst committed
96
  o3skim --verbosity INFO ${action1} ${action2}
BorjaEst's avatar
BorjaEst committed
97
...
98
99
100
101
102
103
104
105
106
107
108
executing: main
...
2020-08-25 12:42:34,151 - INFO     - Configuration found at: './sources.yaml'
2020-08-25 12:42:34,152 - INFO     - Loading data from './data' 
2020-08-25 12:42:34,261 - INFO     - Skimming data to './output' 
```

For the main function description and commands help you can call:
```sh  
$ udocker run --user=application o3skim --help
...
BorjaEst's avatar
BorjaEst committed
109
110
```

BorjaEst's avatar
BorjaEst committed
111
# Testing <a name = "testing"></a>
BorjaEst's avatar
BorjaEst committed
112
Testing is based on [sqa-baseline](https://indigo-dc.github.io/sqa-baseline/) criteria. On top, [tox](https://tox.readthedocs.io/en/latest/) automation is used to simplify the testing process.
113

BorjaEst's avatar
BorjaEst committed
114
To run unit and functional tests with coverage use:
BorjaEst's avatar
BorjaEst committed
115
```sh
BorjaEst's avatar
BorjaEst committed
116
$ tox
BorjaEst's avatar
BorjaEst committed
117
```
118

BorjaEst's avatar
BorjaEst committed
119
120
121
122
123
124
125
126
127

# Documentation <a name = "doc"></a>
All o3as project module documents can be found at [o3as.readthedocs.io](https://o3as.readthedocs.io/en/latest/). 


# Authors <a name = "authors"></a>
- [@V.Kozlov](https://git.scc.kit.edu/eo9869)
- [@T.Kerzenmacher](https://git.scc.kit.edu/px5501)
- [@B.Esteban](https://git.scc.kit.edu/zr5094)
BorjaEst's avatar
BorjaEst committed
128