Скачать или смотреть 2023 11 01 12 - Mathias Bavay - Data Access Made Easy: flexible data standardization and processing

Data Access Made Easy: flexible, on the fly data standardization and processing

Data Access Made Easy: flexible, on the fly data standardization and processing
Mathias Bavay¹, Patrick Leibersperger¹, Øystein Godøy², Charles Fierz¹, Rodica Nitu³
¹ WSL Institute for Snow and Avalanche Research SLF, Davos, Switzerland
² Norwegian Meteorological Institute, Oslo, Norway
³ World Meteorological Organization, Geneva, Switzerland
Automatic Weather Stations (AWS) deployed in the context of research projects provide very valuable data thanks to the flexibility they offer in term of measured meteorological parameters, choice of sensors and quick deployment and redeployment. However this flexibility is a challenge in terms of metadata and data management. Traditional approaches based on networks of standard stations can not accommodate these needs and often no tools are available to manage these research AWS, leading to wasted data periods because of difficult data reuse, low reactivity in identifying potential measurement problems, and lack of metadata to document what happened.
The Data Access Made Easy (DAME) effort is our answer to these challenges. At its core, it relies on the mature and flexible open source MeteoIO meteorological pre-processing library. It was originally developed as a flexible data processing engine for the needs of numerical models consuming meteorological data and further developed as a data standardization engine for the Global Cryosphere Watch (GCW) of the World Meteorological Organization (WMO). For each AWS, a single configuration file describes how to read and parse the data, defines a mapping between the available fields and a set of standardized names and provides relevant Attribute Conventions Dataset Discovery (ACDD) metadata fields, if necessary on a per input file basis. Low level data editing is also available, such as excluding a given sensor, swapping sensors or merging data from another AWS, for any given time period. Moreover an arbitrary number of filters can be applied on each meteorological parameter, restricted to specific time periods if required. This allows to describe the whole history of an AWS within a single configuration file and to deliver a single, consistent, standardized output file possibly spanning many years, many input data files and many changes both in format and available sensors. Finally, all configuration files are versioned in order to document their history.
A web interface has been developed that allows data owners to manage the configuration files for their stations, refresh their data at regular intervals, inspect the data QA log files and allow on-demand data generation. The same interface allows other users to request data on-demand for any time period.
This project has received funding from the World Meteorological Organization under grant agreement No. 29539/2022-1.9 as well as the European Union’s Horizon 2020 research and innovation programme under grant agreement No. 101003472.