C3S 311a Lot 2 Global Land and Marine Observations Database Service Land Data Policy
Version History
| Version | Author(s) | Date | CDS release version that pertains |
|---|---|---|---|
| V1.0 | Kate Willett, Simon Noone, Robert Dunn, Peter Thorne | 14/7/20 | V1.0 |
1. Introduction
The C3S 311a Lot 2 Global Land and Marine Observations Database service (hereafter, the Service) is undertaken in collaboration with the US National Oceanic and Atmospheric Administration's National Centre for Environmental Information (NCEI). The aim of the Service is to improve access to, and management of, historical land and marine surface meteorological data holdings. Long-term historical observations allow us to understand past changes and variability in climate, and hence to contextualise projected future changes and their impacts.
Sharing and onward usage policy relating to land surface meteorological data is complex and highly variable depending on the owning country or entity, and in most cases is not clearly associated with the data itself, requiring case by case detailed investigation. Historical changes to network designations, political boundaries, data archives and policy practices further complicate the issue. This means that in the majority of cases there is no immediately clear data policy associated with a station or collection of stations, and significant investigation is required.
Out of an abundance of caution, the default approach of the Service is to determine data to be restricted to non-commercial usage unless and until a more open policy can be ascertained.
Modern use of meteorological data is governed by the World Meteorological Organization (WMO) policy, practice and guidelines for the exchange of meteorological and related data and products. This is laid out in the following resolutions:
- WMO Resolution 40 of the Twelfth WMO Congress 1995 - http://datastore.copernicus-climate.eu/documents/in-situ/wmo_827_enCG-XII-Res40.pdf, https://www.wmo.int/pages/prog/www/ois/Operational_Information/Publications/Congress/Cg_XII/res40_en.html;
- WMO Resolution 60 of the Seventeenth WMO Congress 2015 - https://library.wmo.int/doc_num.php?explnum_id=4192.
Resolution 40 calls for free and unrestricted international exchange of meteorological data as well as related data and products. It mandates this for those data that are considered 'essential' to 'services in support of the protection of life and property and the well‐being of all nations'. All other meteorological data and products are considered to be 'additional' whereby owners are asked to share data freely but with the ability to place restrictions on their commercial use. Hence, the 'essential' and 'additional' data categories can be interpreted as relating to use cases of 'commercial' and 'non-commercial', respectively. The extent to which WMO Resolution 40 applies retrospectively to data that pre-dates the Resolution is not clear. Neither is it clear what happens in the case of stations changing their network designation and therefore whether they are classed as 'essential' or not through time.
Some countries have declared an open data policy relating to all their meteorological holdings. However, it is not clear how this relates to stations located within that country that may have previously been made by another country (e.g. changes in international borders, or data being held by a former colonial state), nor for stations located within other countries that were previously governed by the open data policy country. The hope is that over time more countries will declare their meteorological holdings open. There is currently no definitive single point of reference listing the data policy within each country.
Historical data that pre-dates the electronic exchange era was originally recorded on paper and in many cases remains solely in paper form. Efforts to digitise such data are often done as part of grassroots or international research projects and so the digital data have been made publicly available. It is rare that a clear data policy is attached to these data but the assumption is that they can be used at a minimum for non-commercial applications as they are already in the public domain. Where digitisation has taken place, the data are not necessarily available under an open data policy.
Daily and monthly meteorological data may be provided directly from the owning country. Therefore, unless it is a CLIMAT designated observation, whereby it comes under the WMO Resolution 40 'essential' data category, there may be restrictions on its onward use. In those cases where the C3S Service has derived daily and monthly values from hourly observations it is not possible to easily retrieve the original hourly observations. However, the hourly observations themselves can be identified and traced. Hence, where those hourly data are not 'essential' data and have restrictions placed on their onwards use, these restrictions also are determined to pertain to derived daily and monthly values.
The Service is a collation of meteorological observations collected electronically from previously archived international data exchanges (GTS, etc), individual agreements (with Institutions, NHMSs etc.) or ad hoc submissions of data collections and data rescue missions. Hence, data policy relating to these holdings includes all of the above scenarios.
2. Data Policy Categories Applied in the Present Release
As noted in the prior section, out of an abundance of caution, in early data releases of the Service, more data than may be necessary are being flagged as being restricted to non-commercial use only. This is to avoid raising issues with ultimate data rights holders and any resulting negative impacts on real-time data exchange which, for example, may imperil the provision of operational forecasts. Over successive releases an increasing volume of data will be made available for commercial applications once we have a sufficiently documented basis for doing so.
2.1 Nationally Open Data Policies
A number of countries have declared their meteorological collections as open datasets available to all for unrestricted onwards use cases, including commercial activities. Where we are satisfied of this policy at a national level, data will be flagged as open, and therefore available for commercial use applications without restriction. An updated list of open data policy countries can be found in Annex A. There will likely be more complicated cases. For example, the ownership of a particular time series or set of data may not be clear because the country instigating the original measurements was not the country where the measurements took place. This will be addressed on a case by case basis as queries arise. Some countries have partial open data policies. At present, it is not possible to decipher which stations with the Service the open parts cover and so data from these countries remains restricted to non-commercial use only. Over time we hope that it becomes easier to decipher which records within a country can be made open. The data policy for those records will then be amended accordingly. A Global data source contains stations located in many countries and the actual source data policy may not have been verified and will be assumed to be for non-commercial use only. However, in this case the stations located in a country that has been verified as having a fully open data policy will be listed as open access, this has led to some Global data sources containing both open access and non-commercial data policies. In addition, because of the station merge process some stations may contain segments of data which have derived from a source with open data policy and other segments which come from a source with a non-commercial data policy.
2.2 Pre-Resolution 40 Adoption Data Policy (pre 1995)
Data pre-dating electronic global exchange has mostly come from existing global and national public archives and various significant digitisation efforts (e.g., Atmospheric Circulation Reconstructions over the Earth (ACRE) – met-acre.net, oldweather.org). Such efforts are now supported by the C3S Data Rescue Service and periodically more newly digitised historical data should become available through the C3S Service. Generally, there is no link to source IPR policy documents for such data. Unless otherwise documented an assumption is made that such data can be used on a non-commercial basis only. Sources with a clear open data policy are listed in Annex A.
The data policy applying to data exchanged electronically before the agreement of WMO Resolution 40 is also unclear. Given that these are not explicitly listed in the 'essential' criteria, we have interpreted the guidance to mean that these are 'additional' and therefore made available on a non-commercial basis only. Sources with a clear open data policy are listed in Annex A.
2.3 WMO Resolution 40 Essential Data
WMO Resolution 40 states that all 'essential' data are to be provided on a free and unrestricted basis:
"Members shall provide on a free and unrestricted basis essential data and products which are necessary for the provision of services in support of the protection of life and property and the well‐being of all nations, particularly those basic data and products, as, at a minimum, described in Annex 1 to this resolution, required to describe and forecast accurately weather and climate, and support WMO Programmes."
Following the WMO Resolution 40 Annex 1 definition (http://datastore.copernicus-climate.eu/documents/in-situ/wmo_827_enCG-XII-Res40.pdf), any 00, 06, 12, 18Z hours observations from stations identified as part of the Regional Basic Surface Network (RBSN), or CLIMAT reports, are considered to be 'essential'. These data are therefore available on a free and unrestricted basis, which includes for commercial use and efforts are underway to identify all such reports as open within the Service.
2.4 WMO Resolution 40 Additional Data
WMO Resolution 40 states that all 'additional' data should also be provided on a free and unrestricted basis but that this is not mandatory:
"Members should also provide the additional data and products which are required to sustain WMO Programmes at the global, regional, and national levels and, further, as agreed, to assist other Members in the provision of meteorological services in their countries. While increasing the volume of data and products available to all Members by providing these additional data and products, it is understood that WMO Members may be justified in placing conditions on their re‐export for commercial purposes outside of the receiving country or group of countries forming a single economic group, for reasons such as national laws or costs of production."
"Members should provide to the research and education communities, for their non‐commercial activities, free and unrestricted access to all data and products exchanged under the auspices of WMO with the understanding that their commercial activities are subject to the same conditions identified in ADOPTS (2) above."
Hence, any land meteorological data not already identified as under the above categories are considered to be 'additional' data. These data are therefore available on a free but restricted basis, where only non-commercial activities are permitted. They will be identified as non-commercial within the Service.
2.5 Derived Data Values
In many cases, the Service's daily and monthly values are derived from hourly and daily data respectively. Although the original values are not retrievable from the averages, they are traceable through the station ID. Therefore, we follow the WMO Resolution 40 Annex 2 point 6 statement that the derived data 'should carry the same conditions on their re‐export for commercial purposes as those [source] additional data or products'. At present, all derived data are made available with restriction to non-commercial use only. In the future we will progress this effort such that those daily and monthly averages derived from open sources will be made available as open.
3. Policy Queries and Change Process
There will undoubtedly be cases that arise where data may be accidently listed under the incorrect data policy category. We have a process to follow in this event, whether it is a user-led change or a Service detected change.
In the event of a user discovered query, please contact Copernicus-support@ecmwf.int or log in to the C3S Enquiry Portal (https://jira.ecmwf.int/servicedesk/customer/portal/1/user/login?destination=portal%2F1%2Fgroup%2F6%3FgroupId%3D6). Please state 'C3S 311a Lot 2 Data Policy' and describe the issue found with as much detail and documentation as possible.
Where open data are queried, they will be withheld while investigation takes place. Any formal change of policy must be supported by sufficient evidence such as the data owner's written consent or a link to a formal policy document. A copy of these documents will be held by the Service along with the initial query and investigation trail. We aim to resolve all queries within 6 months.
4. Acknowledgement and Citation Requirements
All data accessed from the Service will have the relevant acknowledgements and citations attached. We ask that users reproduce these data provider statements whenever their use of the data results in a publication. These are often key for data providers to justify continued provision of observations to their funders. For example, Met Éireann (Irish Met Service) state that all users acknowledge them when using their data:
"Under the CC-BY-SA Licence, users must acknowledge Met Éireann as the source of the data by inserting this attribution statement in their product or application: Contains Met Éireann Data licensed"
Data acknowledgements and citations requirements pertaining to each download will be provided in a text file. Users agree to use acknowledgments and citations in any application that uses the data provided.
There is no additional formal requirement to cite or acknowledge the Service although relevant papers and documentation are available through the CDS documentation tab and are citeable should the user wish to do so.
Annex A – Countries and Sources with Verified Open Data Policies
Table A1. Countries with an open data policy that covers all of their meteorological data
Table A2. Sources with an open data policy that covers all of their meteorological data
| Source ID | Source and data set name | Policy Link | Notes / Licence type |
|---|---|---|---|
| 120 | U.S. Cooperative Summary of the Day Transmitted via WxCoder3 (NCDC DSI-3207) | https://www.ncdc.noaa.gov/wdcmet https://obamawhitehouse.archives.gov/sites/default/files/omb/memoranda/2013/m-13-13.pdf Verified via personal communication with M. Menne at NOAA/NCEI | Open access |
| 159 | U.S. ASOS data for October 2000 December 2005 (NCDC DSI-3211) | https://www.ncdc.noaa.gov/wdcmet https://obamawhitehouse.archives.gov/sites/default/files/omb/memoranda/2013/m-13-13.pdf Verified via personal communication with M. Menne at NOAA/NCEI | Open access |
| 160 | High Plains Regional Climate Center real time data | https://www.ncdc.noaa.gov/wdcmet https://obamawhitehouse.archives.gov/sites/default/files/omb/memoranda/2013/m-13-13.pdf Verified via personal communication with M. Menne at NOAA/NCEI | Open access |
| 161 | U.S. Cooperative Summary of the Day (NCDC DSI-3200) | https://www.ncdc.noaa.gov/wdcmet https://obamawhitehouse.archives.gov/sites/default/files/omb/memoranda/2013/m-13-13.pdf Verified via personal communication with M. Menne at NOAA/NCEI | Open access |
| 162 | CDMP Cooperative Summary of the Day (NCDC DSI-3206) | https://www.ncdc.noaa.gov/wdcmet https://obamawhitehouse.archives.gov/sites/default/files/omb/memoranda/2013/m-13-13.pdf Verified via personal communication with M. Menne at NOAA/NCEI | Open access |
| 164 | U.S. First-Order Summary of the Day (NCDC DSI-3210) | https://www.ncdc.noaa.gov/wdcmet https://obamawhitehouse.archives.gov/sites/default/files/omb/memoranda/2013/m-13-13.pdf Verified via personal communication with M. Menne at NOAA/NCEI | Open access |
| 224 | U.S. Automated Surface Observing System (ASOS)real-time data (since January 1 2006) | https://www.ncdc.noaa.gov/wdcmet https://obamawhitehouse.archives.gov/sites/default/files/omb/memoranda/2013/m-13-13.pdf Verified via personal communication with M. Menne at NOAA/NCEI | Open access |
| 228 | Short time delay US National Weather Service CF6 daily summaries provided by the High Plains Regional Climate Center | https://www.ncdc.noaa.gov/wdcmet https://obamawhitehouse.archives.gov/sites/default/files/omb/memoranda/2013/m-13-13.pdf Verified via personal communication with M. Menne at NOAA/NCEI | Open access |
| 230 | U.S. Fort data | https://www.ncdc.noaa.gov/wdcmet https://obamawhitehouse.archives.gov/sites/default/files/omb/memoranda/2013/m-13-13.pdf Verified via personal communication with M. Menne at NOAA/NCEI | Open access |
| 233 | U.S. Cooperative Summary of the Day data digitized from paper observer forms (from 2011 to present) | https://www.ncdc.noaa.gov/wdcmet https://obamawhitehouse.archives.gov/sites/default/files/omb/memoranda/2013/m-13-13.pdf Verified via personal communication with M. Menne at NOAA/NCEI | Open access |
| 235 | Community Collaborative Rain Hail and Snow (CoCoRaHS) | https://www.cocorahs.org/Content.aspx?page=datausagepolicy | Open access |
| 247 | ISPD_International Surface Pressure databank_CDMP SAO/1001 Forms USA_sub_daily | https://www.ucar.edu/terms-of-use/data | Open access |
| 250 | ISPD_International Surface Pressure databank_Hadley Center UK_sub_daily | personal communication via email with data source | Open access |
| 252 | ISPD_International Surface Pressure databank_READER Antarctic&Southern Hemisphere_sub_daily | https://data.bas.ac.uk/full-record.php?id=GB/NERC/BAS/PDC/00794 | Open access |
| 253 | ISPD_International Surface Pressure databank_KNMI Holland_sub_daily | https://www.knmi.nl/copyright, https://www.digitaleoverheid.nl/overzicht-van-alle-onderwerpen/nieuwe-technologieen-data-en-ethiek/open-data/, no additional data - https://community.wmo.int/notifications | https://creativecommons.org/publicdomain/zero/1.0/legalcode.nl |
| 254 | ISPD_International Surface Pressure databank_US Army Signal Service and other 19th Century Voluntary Obs_sub_daily | https://www.ncdc.noaa.gov/wdcmet https://obamawhitehouse.archives.gov/sites/default/files/omb/memoranda/2013/m-13-13.pdf Verified via personal communication with M. Menne at NOAA/NCEI | Open access |
| 255 | ISPD_International Surface Pressure databank_international stations recovered by Atmospheric Circulation Reconstructions over the Earth (ACRE) initiative_sub_daily | personal communication via email with data source | Open access |
| 256 | ISPD_International Surface Pressure databank_Early Arctic observations_sub_daily | personal communication via email with data source | Open access |
| 257 | ISPD_International Surface Pressure databank_EURO4M/MEDARE/C3 hourly SLP observations for North African stations_sub_daily | personal communication via email with data source | Open access |
| 258 | ISPD_International Surface Pressure databank_International stations, University of South Carolina Historical Climate Lab_sub_daily | personal communication via email with data source | Open access |
| 262 | ISPD_International Surface Pressure databank_Environment Canada Pressure Obs_sub_daily | personal communication via email with data source | Open access |
| 263 | ISPD_International Surface Pressure databank_West African Synoptic observations digitized by MeteoFrance_sub_daily | personal communication via email with data source | Open access |
| 267 | ISPD_International Surface Pressure databank_Spanish Hourly Pressure Observations _sub_daily | personal communication via email with data source | Open access |
| 268 | ISPD_International Surface Pressure databank_German climate observations_sub_daily | https://opendata.dwd.de/climate\_environment/CDC/Terms\_of\_use.pdf | https://opendata.dwd.de/climate\_environment/CDC/Terms\_of\_use.pdf |
| 269 | ISPD_International Surface Pressure databank_ZAMG Austrian station observations _sub_daily | personal communication via email with data source | Open access |
| 270 | ISPD_International Surface Pressure databank_Meteoswiss station collection_sub_daily | personal communication via email with data source | Open access |
| 271 | ISPD_International Surface Pressure databank_South African Weather Service Meteorological collection_sub_daily | personal communication via email with data source | Open access |
| 272 | ISPD_International Surface Pressure databank_National Norwegian meteorological database_sub_daily | https://www.met.no/en/free-meteorological-data/licensing-and-crediting, https://www.met.no/en/free-meteorological-data, no additional data - https://community.wmo.int/notifications | Creative Commons 4.0 BY http://creativecommons.org/licenses/by/4.0 |
| 274 | ISPD_International Surface Pressure databank_Signatures of environmental change in the observations of the Geophysical Institutes_sub_daily | personal communication via email with data source | Open access |
| 277 | ISPD_International Surface Pressure databank_ACRE-Pacific: NIWA and NZMet Service_sub_daily | personal communication via email with data source | Open access |
| 289 | ISPD_International Surface Pressure databank_Canadian Volunteer Climate Data Rescue project_sub_daily | personal communication via email with data source | Open access |
| 290 | ISPD_International Surface Pressure databank_University of Aberdeen historical pressure observations_sub_daily | http://www.nationalarchives.gov.uk/doc/open-government-licence/version/3/ | Open access |
| 292 | ISPD_International Surface Pressure databank_ERA-CLIM FFCUL_sub_daily_ | personal communication via email with data source | Open access |
| 294 | ISPD_International Surface Pressure databank_Project IMPROVE_sub_daily | personal communication via email with data source | Open access |
| 296 | ISPD_International Surface Pressure databank_University of Bern_sub_daily | personal communication via email with data source | Open access |
| 297 | ISPD_International Surface Pressure databank_Stockholm University_sub_daily | https://www.smhi.se/en/research/more-about/open-access-to-data-for-research-and-development-1.31497 | Creative Commons Recognition 4.0 SE https://www.smhi.se/data/oppna-data/villkor-for-anvandning-1.30622 |
| 298 | ISPD_International Surface Pressure databank_University of East Anglia _sub_daily | http://www.nationalarchives.gov.uk/doc/open-government-licence/version/3/ | Open access |
| 300 | ISPD_International Surface Pressure databank_ACRE-Pacific: Cook Island Met Services_sub_daily | personal communication via email with data source | Open access |
| 301 | ISPD_International Surface Pressure databank_ACRE-Pacific: Pacific Island Met Services_sub_daily | personal communication via email with data source | Open access |
| 304 | ISPD_International Surface Pressure databank_William Hutchinson pressure, Liverpool_sub_daily | http://www.nationalarchives.gov.uk/doc/open-government-licence/version/3/ | Open access |
| 305 | ISPD_International Surface Pressure databank_Jersey, Channel Island Pressure Obs_sub_daily | personal communication via email with data source | Open access |
| 306 | ISPD_International Surface Pressure databank_CMDP-USNO sub_daily | https://www.ncdc.noaa.gov/wdcmet https://obamawhitehouse.archives.gov/sites/default/files/omb/memoranda/2013/m-13-13.pdf Verified via personal communication with M. Menne at NOAA/NCEI | Open access |
| 307 | ISPD_International Surface Pressure databank_Russian Sitka Sea Level Pressure, University of South Carolina Climate Lab_sub_daily | http://www.meteorf.ru/opendata/ | Open access |
| 308 | ISPD_International Surface Pressure databank_University of Toronto British Everest Expedition meteorological observation collection_sub_daily | personal communication via email with data source | Open access |
| 310 | ISPD_International Surface Pressure databank_University of Helsinki_sub_daily | https://en.ilmatieteenlaitos.fi/open-data | Creative Commons Attribution 4.0 https://en.ilmatieteenlaitos.fi/open-data-licence, https://creativecommons.org/licenses/by/4.0/ |
| 311 | ISPD_International Surface Pressure databank_Antarctic Expeditions_sub_daily | personal communication via email with data source | Open access |
| 312 | ISPD_International Surface Pressure databank_Canadian Arctic Fort Rae SLP_sub_daily | personal communication via email with data source | Open access |
| 313 | The Climate Database Modernization Program (CDMP) | https://www.ucar.edu/terms-of-use/data | Open access |
| 315 | The Climate Database Modernization Program (CDMP) | https://www.ucar.edu/terms-of-use/data | Open access |
| 314 | The Climate Database Modernization Program (CDMP) | https://www.ucar.edu/terms-of-use/data | Open access |
| 316 | The UK Met Office Daily Weather Reports | personal communication via email with data source | Open access |
| 248 | ISPD_International Surface Pressure databank_Russian Empire Stations sub_daily | personal communication via email with data source | Open access |
| 285 | ISPD_International Surface Pressure databank_All-Russia Research Institute of Hydrometeorological Information - World Data Centre (RIHMI-WDC)_sub_daily | personal communication via email with data source | Open access |
| 286 | ISPD_International Surface Pressure databank_Data from Russian Hydrometcentre sub_daily | personal communication via email with data source | Open access |
| 287 | ISPD_International Surface Pressure databank_Early Russian Empire Stations, , digitized in LDEO from Kupffers Annuaires_sub_daily | personal communication via email with data source | Open access |
| 240 | SNOwpack TELemtry (SNOTEL) data obtained from the U.S. Department of Agriculture's Natural Resources Conservation Service | https://www.ncdc.noaa.gov/wdcmet https://obamawhitehouse.archives.gov/sites/default/files/omb/memoranda/2013/m-13-13.pdf | Open access |
| 281 | ISPD_International Surface Pressure databank_Japan Agency for Marine-earth Science and Technology (JAMSTEC)archive_sub_daily | personal communication via email with data source | Open access |
| 266 | ISPD_International Surface Pressure databank_Hourly Surface Observations for Brazile_sub_daily | personal communication via email with data source | Open access |