Dear WIGOS community,
as WDQMS has become more widely known, we have noticed increased scraping of the system, particularly the API, likely from search engines or AI bots. This usage, often running at several requests per second, makes the system slow for all users.
As a reaction to this we want to introduce API throttling to the next release. Under the proposed approach, each IP address can download a number of files per day without limitations. If usage goes over this threshold, a token, which can be obtained by registering, needs to be provided. In this way access to the system remains free, but usage beyond normal limits is traceable.
We propose that the threshold is set so that most community access is not affected by this change. Our proposal would be to set the threshold to somewhere between 100 and 1000 overall daily requests to the API (file download).
Does anyone download more files than 100 per day on a regular basis, and if yes, would you be able to adjust your (likely automated) process to provide a token in the HTTP header?
For example with curl, a token can be provided with the "-H command line switch" like this: curl -H "Authorization: Token YOUR_ACTUAL_TOKEN" https://wdqms.wmo.int/wdqmsapi/v1/download/nwp/synop/six_hour/availability/?date=2026-08-30&period=18&variable=pressure¢ers=COMBINED&baseline=OSCAR
The team would like to implement this function together with the Marine Availability release which will be the next release of WDQMS.
with kind regards,
Timo for the WDQMS team