README.md 5.04 KB
Newer Older
BorjaEst's avatar
BorjaEst committed
1
2
3
4
5
<p align="center">
  <a href="" rel="noopener">
 <img width=200px height=200px src="https://i.imgur.com/6wj0hh6.jpg" alt="Project logo"></a>
</p>

BorjaEst's avatar
BorjaEst committed
6
<h3 align="center">o3skim</h3>
BorjaEst's avatar
BorjaEst committed
7
8
9

<div align="center">

10
11
12
13
  [![pipeline status](https://git.scc.kit.edu/synergy.o3as/o3skim/badges/master/pipeline.svg)](https://git.scc.kit.edu/synergy.o3as/o3skim/-/commits/master)
  [![coverage status](https://git.scc.kit.edu/synergy.o3as/o3skim/badges/master/coverage.svg)](https://git.scc.kit.edu/synergy.o3as/o3skim/-/commits/master)
  [![License](https://img.shields.io/badge/license-GPL-blue.svg)](https://git.scc.kit.edu/synergy.o3as/o3skim/-/commits/master)
  [![Status](https://img.shields.io/badge/status-building-blue.svg)](https://git.scc.kit.edu/synergy.o3as/o3skim/-/commits/master) 
BorjaEst's avatar
BorjaEst committed
14
15
16
17
18

</div>

---

BorjaEst's avatar
BorjaEst committed
19
<p align="center"> Data pre-processing for ozone models 
BorjaEst's avatar
BorjaEst committed
20
21
22
    <br> 
</p>

BorjaEst's avatar
BorjaEst committed
23
# 📝 Table of Contents
BorjaEst's avatar
BorjaEst committed
24
25
26
27
- [About](#about)
- [Getting Started](#getting_started)
- [Deployment](#deployment)
- [Built Using](#built_using)
BorjaEst's avatar
BorjaEst committed
28
- [Installing](#Installing)
BorjaEst's avatar
BorjaEst committed
29
30
- [Authors](#authors)
- [Acknowledgments](#acknowledgement)
BorjaEst's avatar
BorjaEst committed
31
- [TODO](https://git.scc.kit.edu/synergy.o3as/o3skim/-/issues)
BorjaEst's avatar
BorjaEst committed
32

33
# About <a name = "about"></a>
BorjaEst's avatar
BorjaEst committed
34
This project provides the tools to preprocess, standarise and reduce ozone data for later transfer and plot. 
BorjaEst's avatar
BorjaEst committed
35
36


37
# Getting Started <a name = "getting_started"></a>
BorjaEst's avatar
BorjaEst committed
38
See [deployment](#deployment) for notes on how to deploy the project on a live system.
BorjaEst's avatar
BorjaEst committed
39

BorjaEst's avatar
BorjaEst committed
40
## Prerequisites
41
42
43
To run the project as container, you need the following systems and container technologies:
- __Build machine__ with [docker](https://docs.docker.com/engine/install/) 
- __Runtime machine__ with [udocker](https://indigo-dc.gitbook.io/udocker/installation_manual)
BorjaEst's avatar
BorjaEst committed
44

BorjaEst's avatar
BorjaEst committed
45
> Note udocker cannot be used to build containers, only to run them. 
BorjaEst's avatar
BorjaEst committed
46
47


48
## Built using docker <a name = "built_using"></a>
49
Download the repository at the __Build machine__ using git.
BorjaEst's avatar
BorjaEst committed
50
51
```sh
$ git clone git@git.scc.kit.edu:synergy.o3as/o3skim.git
52
53
Cloning into 'o3skim'...
...
BorjaEst's avatar
BorjaEst committed
54
```
55
Build the docker image at the __Build machine__ using docker.
BorjaEst's avatar
BorjaEst committed
56
57
58
59
60
```sh
$ docker build --tag o3skim .
...
Successfully built 69587025a70a
Successfully tagged o3skim:latest
BorjaEst's avatar
BorjaEst committed
61
```
BorjaEst's avatar
BorjaEst committed
62
63
64
65
66
67
If the build process succeded, you can list the image on the docker image list:
```sh
$ docker images
REPOSITORY                         TAG                 IMAGE ID            CREATED              SIZE
o3skim                             latest              69587025a70a        xx seconds ago      557MB
...
BorjaEst's avatar
BorjaEst committed
68
69
```

70
71
72
73
74
75
76
77
78
79
## Running the tests <a name = "tests"></a>
To run tests, you need to install the tool in your system without docker.

As first step ensure you have the following dependencies:
- [python 3.8](https://www.python.org/downloads/release/python-385/)
- [pip 20.0.2](https://pypi.org/)
- [gcc](https://gcc.gnu.org/)
- [g++]()

After download and dependencies check, install with pip:
BorjaEst's avatar
BorjaEst committed
80
81
82
```sh
$ pip install -e .
```
BorjaEst's avatar
BorjaEst committed
83

84
85
86
Tests should run using 
[tox](https://tox.readthedocs.io/en/latest/).
To install it with pip use:
BorjaEst's avatar
BorjaEst committed
87
88
89
```sh
$ pip install tox
```
90

BorjaEst's avatar
BorjaEst committed
91
To start testing simply run:
92
93
94
95
96
97
98
```sh
$ tox
...
py37: commands succeeded
py38: commands succeeded
```

99
# Deployment <a name = "deployment"></a>
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
To deploy the the application using __udocker__ at the __Runtime machine__ you need:
 - Input path with data to skim, to be mounter on `/app/data` inside the container.
 - Output path for skimmed results, to be mounted on `/app/output` inside the container.
 - Configuration file with a data structure desctiption at the input path in [YAML](https://yaml.org/) format. This configuration file has to be mounted on `/app/sources.yaml` inside the container. See [sources_example.yaml](/sources_example.yaml) for a configuration example.

Once the requierement are needed, pull the image from the image registry.
For example, to pull it from the synergy-imk official registry use:
```sh
$ udocker pull synergyimk/o3skim
...
Downloading layer: sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4
...
```

Once the repository is added and the image downloaded, create the local container: 
BorjaEst's avatar
BorjaEst committed
115
```sh
116
117
118
119
120
121
122
123
124
125
126
$ udocker create --name=o3skim synergyimk/o3skim
fa42a912-b0d4-3bfb-987f-1c243863802d
```

Finally, run the container. Note the described _data_, _output_ and _sources.yaml_ have to be provided. Also it is needed to specify the user _application_ should run inside the container:
```sh
$ udocker run \
  --user=application \
  --volume=${PWD}/sources.yaml:/app/sources.yaml \
  --volume=${PWD}/data:/app/data \
  --volume=${PWD}/output:/app/output \
BorjaEst's avatar
BorjaEst committed
127
128
  o3skim --verbosity INFO
...
129
130
131
132
133
134
135
136
137
138
139
executing: main
...
2020-08-25 12:42:34,151 - INFO     - Configuration found at: './sources.yaml'
2020-08-25 12:42:34,152 - INFO     - Loading data from './data' 
2020-08-25 12:42:34,261 - INFO     - Skimming data to './output' 
```

For the main function description and commands help you can call:
```sh  
$ udocker run --user=application o3skim --help
...
BorjaEst's avatar
BorjaEst committed
140
141
142
```


143
# Authors <a name = "authors"></a>
144
145
- [@V.Kozlov](https://git.scc.kit.edu/eo9869) - TBD
- [@T.Kerzenmacher](https://git.scc.kit.edu/px5501) - TBD
BorjaEst's avatar
BorjaEst committed
146
- [@B.Esteban](https://git.scc.kit.edu/zr5094) - TBD
BorjaEst's avatar
BorjaEst committed
147

148
# Acknowledgements <a name = "acknowledgement"></a>
BorjaEst's avatar
BorjaEst committed
149
- 
BorjaEst's avatar
BorjaEst committed
150