deploy STA-Ingest API with backround jobs
The FROST STA Ingest API was recently changed to perform imports in the background using pythons celery (see https://git.geomar.de/dm/services/frost-sta/sta-ingest-rest-api/-/issues/4).
For this, the docker compose stack needs to be changed:
- add celery services
- celery worker to do the actual work
- celery flower WebApp to check on tasks
- celery beat for scheduled tasks (not used yet but no extra effort needed to deploy now)
- add redis service to be used by celery as broker and result backend
I will add the docker-compose.yml in a branch for this issue.
Testing the deployment locally on my dev laptop works. Testing it locally on geom-sctr-test fails with a 503 error for the /soop/API endpoint.
The current work-in-progress stack is located in this issues branch #1-deploy-sta-ingest-api-with-backround-jobs as well as locally on geom-sctr-test:/docker/compose/soop-sta/docker-compose.yml_wip
Rolling back to the original docker stack (slightly adjusted for the changes in tne STA-ingest-API) does not resolve the 503.
So far, all changes were don locally on the server an no changes were made in ansible yet
TODOs:
-
fix 503 issue (simple restart of ha-proxy did not work) -
test stack locally on geom-sctr-test (docker-compose.yml in issue branch; stack for your leptop here ) -
if needed, adapt frost-ingest-api (current WIP in own branch ) -
if needed, add env vars to ansible
once this works, think about adding persistent storage for redis and creating an enpoint for the celery flower webapp.