Unverified Commit 266d079b authored by greole's avatar greole Committed by GitHub
Browse files

v0.3 (#22)

* Ir (#1)

* update script, needs refactoring

* update

* fix benchmark

* run at least for 10 ts

* rename GKOBase to GKOlduBase

* create ldu and Ldu matrices

* add ldu GKOCG and GKOBiCGStab

* add Ldu matrix

* add asymetric solver

* clean up

* start implementing of stopping criteria

* add common file

* update common

* inject c++14

* update

* use modified boxTurb16 case

* update benchmark

* keep CF constant

* start reuse sorting idx

* add IR solver, make sorting optional

* add line buffering, further cli args

* Add basic implementation of storing sorting idxs

* add IR implementation

* make sorting default

* store gkomatrix

* add common.C

* make executor an IOPtr

* prepare for forHLR test

* fix block clean for different block start syntax

* transfer to desktop

* hardcoded stored omp executor

* load different executor

* fix min/maxIter bug

* debug reference executor issue

* add clang format to keep formating constant

* fix cuda executor

* clean up

* clean up

* remove unneeded bash benchmark script

* format + prepare preconditioner

* run omp benchmarks for different number of threads

* make pressure solver settings compatible with OF8

* changed size of cases

* fix benchmarking folders

* set dns case boundaries to walls

* Fix several compiler warnings, reordering issues,

* create additional executor to force eager copy

* log succes state of run, keep retrying on failure at least for now

* move OpenFoam stopping criterion to separate file

* Use Foam switches to read dictionary entries

* Make export system a free function, clean logger and imports

* Move IterationLogger to common.H

* Fix execution if sysMatrix is not stored

if sysMatrix is not stored and thus the gkomatrix_ptr==NULL the system
matrix is recreated

* fix loop ctr

* Fix IR solver

- initialize IR with CG
- remove BJ for now

* Add Solver classes to distinguish between OF and GKO solvers, set time out for runs, fix log files

* fix formating

* Fix typos in smoothSolver class

* Fix int to unsingned long conversion

* Add initial CMakeLists file

* Add basic cmake files

* Fix return type warning

* move setting device_exec to separate function

* add third_party folder

* Update README.md

* Update README.md

* Use FOAM_ enviromental variables

* install libginkgo.so files along with libOGL.so if no external ginkgo is used

* Create a initial cmake.yml file

* clone and source OpenFOAM

* Fix non ascii symbol

* display OpenFOAM folder content

* show content of OpenFOAM-8 folder

* Update cmake.yml

* set OpenFOAM env variables manually

* add build badge

* Update Readme file

* remove Test folder since it is no longer needed

* re-enable exporting the system matrix, rhs and initial guess

* Read maxBlockSize from dictionary

* Rename controlDict_ member variable to dictionary_

* Make BiCGStabFactory consistent with CGFactory

* Clean unneeded comment

* Fix sorting bounds of sorting_idxs

* Update cmake.yml

* add basic travis file

* remove github actions file

* update .travis.yml

* fix OpenFOAM path

* update travis file

* compile full OpenFOAM for now

* install openmpi-dev

* install flexc++

* add OpenFOAM-dev to build matrix

* add fl dev

* install openfoam from the repository

* for now just test if OGL builds

* add build badge

* fix sourcing depending on of version

* switch to ubuntu 20.04

* make OGL_USE_EXTERNAL_GINKGO an option

* use --no-check-certificate since importing the key fails regularily

* pass --no-check-certificate to wget

* fix type of tmp_col_idxs and tmp_row_idxs

* sort the system matrix to avoid  cuda convergence issue

* add preconditioner to path

* remove benchmark script to separate repository

* fix find_package

* Fix find_package execution after external ginkgo was included #5

Co-authored-by: Terry Cojean's avatartcojean <terry.cojean@kit.edu>

* Add OpenFOAM version shields

* Travis external ginkgo (#9)

* add external ginkgo flag to ci/cd matrix

* fix ci/cd matrix + test conditional job

* add a test script for external ginkgo build

* fix external ginkgo build flags to speedup build

* Add missing sudo to install ginkgo to system.

* fix cmake build arguments

* Clean ldu ldu base (#13)

* create separate IOExecutorHandler class

* add IOExecutorHandler files

* fix compilation issue

* create separate IOSortingIdxHandler class

* Implement separate IOGKOMatrixHandler class

* move host matrix handling to separate class

* fix matrix sorting/updating host matrix

* Fix #10

 remove deprecated wmake folder

* This PR cleans the common lduLduBase class (#16)

* create separate IOExecutorHandler class

* add IOExecutorHandler files

* fix compilation issue

* create separate IOSortingIdxHandler class

* Implement separate IOGKOMatrixHandler class

* move host matrix handling to separate class

* fix matrix sorting/updating host matrix

* Reuse init guess (#19)

* create separate IOExecutorHandler class

* add IOExecutorHandler files

* fix compilation issue

* create separate IOSortingIdxHandler class

* Implement separate IOGKOMatrixHandler class

* move host matrix handling to separate class

* fix matrix sorting/updating host matrix

* implement reusable initial guess

* fix host executor call

* fix device_x size, IR improvements

* Split init and update of host matrix from sparsity pattern (#21)
Co-authored-by: Terry Cojean's avatartcojean <terry.cojean@kit.edu>
parent 781cfd14
......@@ -8,8 +8,10 @@ os:
- linux
env:
- VERSION=8
- VERSION=7
- VERSION=8 EXTERNAL_GINKGO=OFF
- VERSION=8 EXTERNAL_GINKGO=ON
- VERSION=7 EXTERNAL_GINKGO=OFF
- VERSION=7 EXTERNAL_GINKGO=ON
cache:
directories:
......@@ -33,12 +35,13 @@ before_install:
install:
- sudo apt-get -y install openfoam$VERSION
- source /opt/openfoam$VERSION/etc/bashrc
- ./scripts/travis_install_external_ginkgo.sh
before_script:
- cmake --version
- cmake --version
- mkdir build
- cd build
- cmake ..
- cmake -DOGL_USE_EXTERNAL_GINKGO=$EXTERNAL_GINKGO ..
script:
- make -j4
......@@ -92,6 +92,11 @@ add_compile_definitions(
target_sources(OGL
PRIVATE
common/common.C
IOHandler/IOPtr/IOPtr.C
IOHandler/IOSortingIdxHandler/IOSortingIdxHandler.C
IOHandler/IOExecutorHandler/IOExecutorHandler.C
IOHandler/IOGKOMatrixHandler/IOGKOMatrixHandler.C
lduMatrix/GKOlduBase/GKOlduBase.C
lduMatrix/GKOCG/GKOCG.C
lduMatrix/GKOIR/GKOIR.C
......@@ -99,6 +104,11 @@ lduMatrix/GKOBiCGStab/GKOBiCGStab.C
LduMatrix/GKOLduBase/GKOLduBase.C
LduMatrix/GKOACG/GKOACG.C
PUBLIC
common/common.H
IOHandler/IOPtr/IOPtr.H
IOHandler/IOSortingIdxHandler/IOSortingIdxHandler.H
IOHandler/IOExecutorHandler/IOExecutorHandler.H
IOHandler/IOGKOMatrixHandler/IOGKOMatrixHandler.H
lduMatrix/GKOlduBase/GKOlduBase.H
lduMatrix/GKOCG/GKOCG.H
lduMatrix/GKOIR/GKOIR.H
......@@ -110,13 +120,17 @@ LduMatrix/GKOACG/GKOACG.H
target_include_directories(OGL
PUBLIC
$ENV{FOAM_SRC}/OpenFOAM/lnInclude
$ENV{FOAM_SRC}/meshTools/lnInclude
$ENV{FOAM_SRC}/finiteVolume/lnInclude
$ENV{FOAM_SRC}/meshTools/lnInclude
$ENV{FOAM_SRC}/OpenFOAM/lnInclude
$ENV{FOAM_SRC}/OSspecific/POSIX/lnInclude
common/
lduMatrix/GKOlduBase
LduMatrix/GKOLduBase
common/
IOHandler/IOPtr
IOHandler/IOExecutorHandler/
IOHandler/IOSortingIdxHandler/
IOHandler/IOGKOMatrixHandler/
lduMatrix/GKOlduBase
LduMatrix/GKOLduBase
)
......
#include <ginkgo/ginkgo.hpp>
#include <type_traits>
#include "IOExecutorHandler.H"
namespace Foam {
IOExecutorHandler::IOExecutorHandler(const objectRegistry &db,
const dictionary &controlDict)
: device_executor_name_(
controlDict.lookupOrDefault("executor", word("reference"))),
app_executor_name_(
controlDict.lookupOrDefault("app_executor", word("reference")))
{
// create executors
bool app_exec_stored = db.foundObject<regIOobject>(app_executor_name_);
if (app_exec_stored) {
ref_exec_ptr_ =
&db.lookupObjectRef<GKOReferenceExecPtr>(app_executor_name_);
if (device_executor_name_ == app_executor_name_) {
return;
}
}
bool device_exec_stored =
db.foundObject<regIOobject>(device_executor_name_);
if (device_exec_stored) {
if (device_executor_name_ == "omp") {
omp_exec_ptr_ =
&db.lookupObjectRef<GKOOmpExecPtr>(device_executor_name_);
return;
}
if (device_executor_name_ == "cuda") {
cuda_exec_ptr_ =
&db.lookupObjectRef<GKOCudaExecPtr>(device_executor_name_);
return;
}
if (device_executor_name_ == "hip") {
hip_exec_ptr_ =
&db.lookupObjectRef<GKOHipExecPtr>(device_executor_name_);
return;
}
}
const fileName app_exec_store = app_executor_name_;
ref_exec_ptr_ =
new GKOReferenceExecPtr(IOobject(app_exec_store, db),
gko::give(gko::ReferenceExecutor::create()));
const fileName device_exec_store = device_executor_name_;
const fileName omp_exec_store = "omp";
omp_exec_ptr_ = new GKOOmpExecPtr(IOobject(omp_exec_store, db),
gko::OmpExecutor::create());
if (device_executor_name_ == "cuda") {
cuda_exec_ptr_ =
new GKOCudaExecPtr(IOobject(device_exec_store, db),
gko::give(gko::CudaExecutor::create(
0, omp_exec_ptr_->get_ptr(), true)));
}
if (device_executor_name_ == "hip") {
hip_exec_ptr_ =
new GKOHipExecPtr(IOobject(device_exec_store, db),
gko::give(gko::HipExecutor::create(
0, omp_exec_ptr_->get_ptr(), true)));
}
if (device_executor_name_ == "omp") {
omp_exec_ptr_ = new GKOOmpExecPtr(IOobject(device_exec_store, db),
omp_exec_ptr_->get_ptr());
}
}
defineTemplateTypeNameWithName(GKOExecPtr, "ExecIOPtr");
defineTemplateTypeNameWithName(GKOCudaExecPtr, "CudaExecIOPtr");
defineTemplateTypeNameWithName(GKOOmpExecPtr, "OmpExecIOPtr");
defineTemplateTypeNameWithName(GKOHipExecPtr, "HipExecIOPtr");
defineTemplateTypeNameWithName(GKOReferenceExecPtr, "ReferenceExecIOPtr");
} // namespace Foam
/*---------------------------------------------------------------------------*\
License
This file is part of OGL.
OGL is free software: you can redistribute it and/or modify it
under the terms of the GNU General Public License as published by
the Free Software Foundation, either version 3 of the License, or
(at your option) any later version.
OpenFOAM is distributed in the hope that it will be useful, but WITHOUT
ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License
for more details.
You should have received a copy of the GNU General Public License
along with OGL. If not, see <http://www.gnu.org/licenses/>.
Class
Foam::IOExecutorHandler
Author: Gregor Olenik <go@hpsim.de>
SourceFiles
IOExecutorHandler.C
\*---------------------------------------------------------------------------*/
#ifndef OGL_IOExecutorHandler_INCLUDED_H
#define OGL_IOExecutorHandler_INCLUDED_H
#include "../IOPtr/IOPtr.H"
#include "fvCFD.H"
#include "regIOobject.H"
#include <ginkgo/ginkgo.hpp>
namespace Foam {
using vec = gko::matrix::Dense<scalar>;
using mtx = gko::matrix::Coo<scalar>;
class IOExecutorHandler {
private:
const word device_executor_name_;
const word app_executor_name_;
// executor where Ginkgo will perform the computation
GKOExecPtr *device_exec_ptr_;
// executor of the application
GKOExecPtr *app_exec_ptr_;
GKOReferenceExecPtr *ref_exec_ptr_;
GKOCudaExecPtr *cuda_exec_ptr_;
GKOOmpExecPtr *omp_exec_ptr_;
GKOHipExecPtr *hip_exec_ptr_;
public:
IOExecutorHandler(const objectRegistry &db, const dictionary &controlDict);
word get_device_executor_name() const { return device_executor_name_; }
std::shared_ptr<gko::Executor> get_device_executor() const
{
const word device_executor_name{get_device_executor_name()};
if (device_executor_name == "omp") {
return omp_exec();
}
if (device_executor_name == "cuda") {
return cuda_exec();
}
if (device_executor_name == "hip") {
return hip_exec();
}
return ref_exec();
};
std::shared_ptr<gko::Executor> app_exec() const
{
return app_exec_ptr_->get_ptr();
};
std::shared_ptr<gko::OmpExecutor> omp_exec() const
{
return omp_exec_ptr_->get_ptr();
};
std::shared_ptr<gko::CudaExecutor> cuda_exec() const
{
return cuda_exec_ptr_->get_ptr();
};
std::shared_ptr<gko::HipExecutor> hip_exec() const
{
return hip_exec_ptr_->get_ptr();
};
std::shared_ptr<gko::ReferenceExecutor> ref_exec() const
{
return ref_exec_ptr_->get_ptr();
};
};
} // namespace Foam
#endif
/*---------------------------------------------------------------------------*\
License
This file is part of OGL.
OGL is free software: you can redistribute it and/or modify it
under the terms of the GNU General Public License as published by
the Free Software Foundation, either version 3 of the License, or
(at your option) any later version.
OpenFOAM is distributed in the hope that it will be useful, but WITHOUT
ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License
for more details.
You should have received a copy of the GNU General Public License
along with OGL. If not, see <http://www.gnu.org/licenses/>.
Class
Foam::GKOCG
Author: Gregor Olenik <go@hpsim.de>
SourceFiles
GKOCG.C
\*---------------------------------------------------------------------------*/
#include <ginkgo/ginkgo.hpp>
#include "IOGKOMatrixHandler.H"
namespace Foam {
void IOGKOMatrixHandler::init_device_matrix(
const objectRegistry &db, std::vector<scalar> &values_host,
std::vector<label> &col_idxs_host, std::vector<label> &row_idxs_host,
const label nElems, const label nCells, const bool store) const
{
std::shared_ptr<gko::Executor> device_exec = get_device_executor();
if (sys_matrix_stored_) {
gkomatrix_ptr_ = &db.lookupObjectRef<GKOCOOIOPtr>(sys_matrix_name_);
return;
}
std::shared_ptr<idx_array> col_idx;
std::shared_ptr<idx_array> row_idx;
if (sparsity_pattern_stored_) {
io_col_idxs_ptr_ =
&db.lookupObjectRef<GKOIDXIOPtr>(sparsity_pattern_name_cols_);
io_row_idxs_ptr_ =
&db.lookupObjectRef<GKOIDXIOPtr>(sparsity_pattern_name_rows_);
col_idx = io_col_idxs_ptr_->get_ptr();
row_idx = io_row_idxs_ptr_->get_ptr();
} else {
// if not stored yet create sparsity pattern from correspondent
// views
auto col_idx_view = idx_array::view(gko::ReferenceExecutor::create(),
nElems, &col_idxs_host[0]);
auto row_idx_view = idx_array::view(gko::ReferenceExecutor::create(),
nElems, &row_idxs_host[0]);
// copy sparsity pattern to device and leave it there
col_idx = std::make_shared<idx_array>(col_idx_view);
col_idx->set_executor(device_exec);
row_idx = std::make_shared<idx_array>(row_idx_view);
row_idx->set_executor(device_exec);
}
// if system matrix is not stored create it and set shared pointer
auto gkomatrix =
gko::share(mtx::create(device_exec, gko::dim<2>(nCells, nCells),
val_array::view(gko::ReferenceExecutor::create(),
nElems, &values_host[0]),
*col_idx.get(), *row_idx.get()));
// if updating system matrix is not needed store ptr in obj registry
if (store) {
const fileName path = sys_matrix_name_;
gkomatrix_ptr_ = new GKOCOOIOPtr(IOobject(path, db), gkomatrix);
} else {
gkomatrix_ptr_ = NULL;
gkomatrix_ = gkomatrix;
}
// in any case store sparsity pattern
const fileName path_col = sparsity_pattern_name_cols_;
const fileName path_row = sparsity_pattern_name_rows_;
io_col_idxs_ptr_ = new GKOIDXIOPtr(IOobject(path_col, db), col_idx);
io_row_idxs_ptr_ = new GKOIDXIOPtr(IOobject(path_row, db), row_idx);
};
defineTemplateTypeNameWithName(GKOIDXIOPtr, "IDXIOPtr");
defineTemplateTypeNameWithName(GKOCOOIOPtr, "COOIOPtr");
defineTemplateTypeNameWithName(GKOVECIOPtr, "VECIOPtr");
} // namespace Foam
/*---------------------------------------------------------------------------*\
License
This file is part of OGL.
OGL is free software: you can redistribute it and/or modify it
under the terms of the GNU General Public License as published by
the Free Software Foundation, either version 3 of the License, or
(at your option) any later version.
OpenFOAM is distributed in the hope that it will be useful, but WITHOUT
ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License
for more details.
You should have received a copy of the GNU General Public License
along with OGL. If not, see <http://www.gnu.org/licenses/>.
Class
Foam::GKOCG
Author: Gregor Olenik <go@hpsim.de>
SourceFiles
GKOCG.C
\*---------------------------------------------------------------------------*/
#ifndef OGL_IOGKOMatrixHandler_INCLUDED_H
#define OGL_IOGKOMatrixHandler_INCLUDED_H
#include "fvCFD.H"
#include "regIOobject.H"
#include "../IOExecutorHandler/IOExecutorHandler.H"
#include "../IOPtr/IOPtr.H"
namespace Foam {
using mtx = gko::matrix::Coo<scalar>;
using vec = gko::matrix::Dense<scalar>;
using val_array = gko::Array<scalar>;
using idx_array = gko::Array<label>;
class IOGKOMatrixHandler : public IOExecutorHandler {
private:
const word sys_matrix_name_;
const bool sys_matrix_stored_;
const word update_sysMatrix_;
const word sparsity_pattern_name_cols_;
const word sparsity_pattern_name_rows_;
const bool sparsity_pattern_stored_;
const word init_guess_vector_name_;
const bool init_guess_vector_stored_;
const bool update_init_guess_vector_;
const bool export_;
mutable std::shared_ptr<mtx> gkomatrix_ = NULL;
mutable GKOCOOIOPtr *gkomatrix_ptr_ = NULL;
mutable GKOIDXIOPtr *io_col_idxs_ptr_ = NULL;
mutable GKOIDXIOPtr *io_row_idxs_ptr_ = NULL;
mutable std::shared_ptr<vec> init_guess_ = NULL;
mutable GKOVECIOPtr *io_init_guess_ptr_ = NULL;
public:
IOGKOMatrixHandler(const objectRegistry &db, const dictionary &controlDict,
const word sys_matrix_name)
: IOExecutorHandler(db, controlDict),
sys_matrix_name_(sys_matrix_name + "_gko"),
sys_matrix_stored_(db.foundObject<regIOobject>(sys_matrix_name_)),
update_sysMatrix_(
controlDict.lookupOrDefault("updateSysMatrix", word("yes"))),
sparsity_pattern_name_cols_("gko_sparsity_pattern_cols"),
sparsity_pattern_name_rows_("gko_sparsity_pattern_rows"),
sparsity_pattern_stored_(
db.foundObject<regIOobject>(sparsity_pattern_name_cols_)),
init_guess_vector_name_(sys_matrix_name_ + "_init_guess"),
init_guess_vector_stored_(
db.foundObject<regIOobject>(init_guess_vector_name_)),
update_init_guess_vector_(
controlDict.lookupOrDefault<Switch>("updateInitVector", false)),
export_(controlDict.lookupOrDefault<Switch>("export", false)){};
bool get_sys_matrix_stored() const { return sys_matrix_stored_; };
void init_device_matrix(const objectRegistry &db,
std::vector<scalar> &values_host,
std::vector<label> &col_idxs_host,
std::vector<label> &row_idxs_host,
const label nElems, const label nCells,
const bool store) const;
std::shared_ptr<mtx> get_gkomatrix() const
{
if (gkomatrix_ptr_ == NULL) {
return gkomatrix_;
}
return gkomatrix_ptr_->get_ptr();
};
void init_initial_guess(const scalarField &psi, const objectRegistry &db,
const label nCells) const
{
std::shared_ptr<gko::Executor> device_exec = get_device_executor();
if (init_guess_vector_stored_ && !update_init_guess_vector_) {
std::cout << " init_initial_guess do nothing" << std::endl;
io_init_guess_ptr_ =
&db.lookupObjectRef<GKOVECIOPtr>(init_guess_vector_name_);
return;
}
std::cout << " init_initial_guess create psi_view" << std::endl;
auto psi_view = val_array::view(gko::ReferenceExecutor::create(),
nCells, const_cast<scalar *>(&psi[0]));
std::cout << " init_initial_guess create x" << std::endl;
auto x = gko::share(
vec::create(device_exec, gko::dim<2>(nCells, 1), psi_view, 1));
const fileName path_init_guess = init_guess_vector_name_;
std::cout << " init_initial_guess create io_init_guess_ptr_"
<< std::endl;
io_init_guess_ptr_ = new GKOVECIOPtr(IOobject(path_init_guess, db), x);
}
std::shared_ptr<vec> get_initial_guess() const
{
std::cout << "get_initial_guess" << std::endl;
if (io_init_guess_ptr_ == NULL) {
std::cout << "is Null" << std::endl;
return init_guess_;
}
return io_init_guess_ptr_->get_ptr();
}
void copy_result_back(const scalarField &psi, const label nCells) const
{
std::cout << "copy back" << std::endl;
auto device_x = vec::create(ref_exec(), gko::dim<2>(nCells, 1));
std::cout << "copy back 2" << std::endl;
device_x->copy_from(gko::lend(get_initial_guess()));
std::cout << "create x_view back" << std::endl;
auto x_view =
val_array::view(ref_exec(), nCells, device_x->get_values());
// for (label i = 0; i < nCells; i++) {
// std::cout << x_view[i] << std::endl;
// }
// move frome device
std::cout << "create psi_view " << std::endl;
auto psi_view =
val_array::view(ref_exec(), nCells, const_cast<scalar *>(&psi[0]));
std::cout << "copy x_view to psi_view " << std::endl;
psi_view = x_view;
}
bool get_update_sys_matrix() const { return (update_sysMatrix_ == "yes"); }
bool get_export() const { return export_; }
};
} // namespace Foam
#endif
#include "IOPtr.H"
namespace Foam {
} // namespace Foam
/*---------------------------------------------------------------------------*\
License
This file is part of OGL.
OGL is free software: you can redistribute it and/or modify it
under the terms of the GNU General Public License as published by
the Free Software Foundation, either version 3 of the License, or
(at your option) any later version.
OpenFOAM is distributed in the hope that it will be useful, but WITHOUT
ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License
for more details.
You should have received a copy of the GNU General Public License
along with OGL. If not, see <http://www.gnu.org/licenses/>.
Class
Foam::IOExecutorHandler
Author: Gregor Olenik <go@hpsim.de>
SourceFiles
IOExecutorHandler.C
\*---------------------------------------------------------------------------*/
#ifndef OGL_IOPtr_INCLUDED_H
#define OGL_IOPtr_INCLUDED_H
#include <ginkgo/ginkgo.hpp>
#include "fvCFD.H"
namespace Foam {
using mtx = gko::matrix::Coo<scalar>;
using val_array = gko::Array<scalar>;
using idx_array = gko::Array<label>;
template <class T>
class IOPtr : public regIOobject {
private:
std::shared_ptr<T> ptr_;
public: