O2A Documentation

data web service (DWS) retirement -- update ​

shutdown -> migration -> upgrade ​

Some weeks ago we informed about the upcoming revision and upgrate of the data web service (https://o2a-data.de/blog/2025-06-25_dws_revision.md). We are about to finalize the preparatory works and drafted a migration schedule. The last and ulitmate step will be the migration of the old database to the new one. Therefore it will be necessary to interrupt some services (as you can imagine it will take a while for > 5000 tables, > 7 TB of size, and > 21 * 10^9 [<- that's billion, thus 1 point is approx. 1 ppb] data points to be transferred). This will include

  • the centralized and managed ingest of data, also known as squid, and
  • the near real-time database, also known as the database or dws or data-xxl or ...

The shutdown will take place

2025-08-26 08:00:00 UTC

2025-09-03 08:00:00 UTC

By 2025-08-29 the services are planned to be available again. No data will be lost. Data that goes through several ingest pipelines (squid), remain in the staging area and will be processed when a valid database connection is available again.

revised API ​

Along with the database and the UI the API gets a facelift too. API requests against the old API will then be redirected. Nonetheless the API is a new one and some things might change during the upgrade. Most and rather trivial requests will still be compatible, but we strongly suggest to revise your API calls and if necessary adapt to the modernized syntax. The two most important links are:

The query syntax does not differ from the old one, but some parameters were renamed, e.g. to go along with standards. Fortunately the deprecated parameters are indicated in the swagger.

Swagger docu indicates deprecated endpoints and suggests their substitution

So, a query for some parameters of a so-called ferrybox (actually a continuous water sampler) installed to RV Polarstern would look like that:

bash
curl -X GET \ 'https://ingest.o2a-data.de/rest/data?codes=vessel:polarstern:fb-732403:sbe45_45-0699:sbe45_temp&codes=vessel:polarstern:fb-732403:sbe45_45-0699:sbe45_sal&codes=vessel:polarstern:fb-732403:meinsberg_ph_2418083:ph_temp&datetimeMin=2025-02-26T00:00:00&datetimeMax=2025-02-27T00:00:00&aggregate=HOUR&aggregateFunctions=MEAN' \
-H 'accept: application/json'
  • API baselink -> https://ingest.o2a-data.de/rest/
  • endpoint -> data
  • query for -> ?
  • 1st parameter code -> codes=vessel:polarstern:fb-732403:sbe45_45-0699:sbe45_temp
  • 2nd parameter code -> &codes=vessel:polarstern:fb-732403:sbe45_45-0699:sbe45_sal
  • 3rd parameter code & codes=vessel:polarstern:fb-732403:meinsberg_ph_2418083:ph_temp
  • temporal query 1 -> &datetimeMin=
  • temporal value 1 -> 2025-02-26T00:00:00
  • temporal query 2 -> &datetimeMax=
  • temporal query 2 -> 2025-02-27T00:00:00
  • temporal aggregation -> &aggregate=
  • temporal resolution -> HOUR
  • temporal aggregation selector -> &aggregateFunctions=
  • aggregation function -> MEAN

As the concept of creating IDs per dataset (as in a submitted/posted file) proved valuable with good performance, dataset IDs are also available via the new API along with rsql statements. These already have been introduced together with the new registry API, see this post. And because it creates a lot of load to retrieve all info for certain data streams we apply rsql statements to filter for specific content, e.g. find all datasets since a certain date:

bash
curl -X GET "https://ingest.o2a-data.de/rest/datasets?where=datetimeMin%3E='2025-04-01';datetimeMax%3C='2025-07-01';streams.code=IN=('station:mythaw_app:meta:dom')"

or all datasets that cover a period of time:

bash
curl -X GET
"https://ingest.o2a-data.de/rest/datasets?where=datetimeMin%3E='2025-04-01';datetimeMax%3C='2025-04-10';streams.code=IN=('vessel:maria_s_merian:rsws_msm:rsws_b2516_mc2:salinity')"

It depends a bit of the weapon of choice, but some characters need to be escaped in order to post them. Here is a list of the most common unicode characters used in this context:

As another novel feature users can submit their NRT-conform data directly to the NRT database. But that is going to be introduced in a separate how-to article in this place.