WIGOS Learning Portal: planned release of WDQMS 1.5.2

Dear WIGOS community,

we would like to inform the community of the planned release of WDQMS webtool v1.5.2 on the 20th of November 2024.

This release, which can be pre-viewed at https://wdqms-test.wmo.int, will contain the following enhancements and bug-fixes requested and reported by the community.

Updated algorithm to calculate the number of expected observations from OSCAR/Surface. The new algorithm better covers a number of corner cases. We have informed affected countries and RWC of this change where needed.
Monthly download to contain RMSE.
Gross errors can be downloaded for surface by using the extra_columns parameter (e.g https://wdqms-test.wmo.int/wdqmsapi/v1/download/synop/monthly/quality/?date=2024-09&variable=pressure&centers=DWD,ECMWF,JMA,NCEP&extra_columns=gross_error_percent)
Baseline switcher in NWP module to switch between OSCAR/Surface (default) and hourly/2-daily reporting for 6-h, daily and monthly surface, and daily and monthly upper-air reports.
Bug-fix in upper-air availability map regarding completeness
Filtering of BUFR and TAC TEMPSHIP with status = 9
Other minor and internal improvements

please have a look, test and send feedback on the new version ahead of the release

with kind regards,

WDQMS development team

Re: planned release of WDQMS 1.5.2

от Karl Monnik - вторник, 12 ноября 2024, 06:26

Hello Timo

Thank you for the improvements. WDQMS is such a user-friendly tool and helpful.

Filtering of BUFR and TAC TEMPSHIP with status = 9
I'm not sure how this is done?

One issue Australia would like resolved related to our sub-national divisions for South Australia and Northern Territory - both state/territory have a time zone that is offset by 30 min from UTC time. Locally the stations deliver SYNOP reports to the GTS 3 hourly on the half-hour. Then our internal systems also generate a hourly SYNOP on the UTC hour to confirm with the GTS. This results in the following message configuration:
On the UTC hour - 24 hourly messages
On the UTC half-hour - 8 3-hourly messages (aligned with local reporting times of 09, 12, 15, 18 etc)
This station in OSCAR Surface has been set up with two pressure deployments to cover the 2 reporting programs.
However, it still appears as a pink dot indicating >100% of the data was delivered.
https://oscar.wmo.int/surface/index.html#/management/station/stationReportDetails/0-20000-0-94110/

See below
SYNOPS from 94110, Oenpelli Airport (Australia)
SM 10/11/2024 00:00-> AAXX 10004 94110 45/// /0605 10319 20223 30122 40132==
SN 10/11/2024 01:00-> AAXX 10014 94110 45/// /0806 10332 20207 30115 40125==
SN 10/11/2024 02:00-> AAXX 10024 94110 45/// /0507 10350 20190 30109 40119==
SN 10/11/2024 02:30-> AAXX 10024 94110 45/// /0905 10365 20194 30103 40113 58019 90230 333 20248==
SI 10/11/2024 03:00-> AAXX 10034 94110 45/// /0907 10364 20189 30098 40108==
etc

Re: planned release of WDQMS 1.5.2

от Timo Proescholdt - вторник, 12 ноября 2024, 14:06

Dear Karl,

thanks for the quick feedback.

We will improve the schedule library to take into account overlapping schedules with diurnal offset in the next release (after 1.5.2).

The filtering is to exclude TEMPSHIP observations from the upper-air land maps by using the CodeType. This happens internally when processing the files provided by NWP center.

with kind regards,
Timo

Re: planned release of WDQMS 1.5.2

от Martina Suaya - четверг, 14 ноября 2024, 19:19

Dear Timo,

I'm very happy to see all these changes implemented. Thank you and all the team for taking care of the issues raised.

A couple of commentes:

1) Select monthly, availability, baseline Hourly. The filterin option is not working. In the exable I want to remove the pink dots and I get green dots (and more colors) hidden

2) select daily (or monthly), availability, all centers, hourly and download the file. What you get is a GBON file instead of the NWP file, and you don't get what is in the map

For example check this station

Is not in the downloaded file.

Also this files only contain All Centers information instead each center

name

wigosid

country code

in OSCAR

longitude

latitude

#received

#expected (GBON)

color code

description

variable

date

center

AEROPARQUE AERO

0-20000-0-87582

ARG

VERDADERO

-5841982

-3455891

green

Normal (>= 80%)

pressure

12/11/2024 21:00

All Centers

3) the grey dots ar lost in the hourly. option I can understand why this is but I'm not sure is ok

4) monthly RMSE/GROSS error
It works very nice and we can get a lot of information when used together

Thanks a lot

Martina

Re: planned release of WDQMS 1.5.2

от Amro Abouelenein - вторник, 19 ноября 2024, 18:22

Dear Martina,

Thank you for reporting this.

Could you please check it again, we fixed the legend filter for the first point, and the download is now pointing to the right csv file also with the list of centres individually.

Please let me know if all works fine now.

Kindly check in the URL https://wdqms-dev.wmo.int/

Thanks

Regards

Amro

Re: planned release of WDQMS 1.5.2

от Martina Suaya - среда, 20 ноября 2024, 14:19

Dear Amro

In wdqms-dev I see

Issue #1 solved
issue #2 solved

But please check on this:

1) daily, availability OSCAR option the filtering is not working. Removing green color statons, remove yellows and greys

2) daily, availability hourly option removing REDS remove YELLOWS

3) daily, availability hourly option downloaded file doesn't match the map. This case is Yellow in the map but is downloaded as RED.
Unknown,83971,,False,-51.16999816894531,-29.989999771118164,6,24,False,red,Availability issues (< 30%),pressure,2024-11-19 00:00:00+00:00,ECMWF

I would solve this as in the GREYS. The map could show the color representing how far from 24 is the observation frequency, and not the correspondance to OSCAR. The same for all time aggregations

I hope this is clear enough

Best

Martina

PS: in the operational wdqms module for NWP there's a problem when filtering. Removing GREENS, remove YELLOWS and GREYS

Re: planned release of WDQMS 1.5.2

от Timo Proescholdt - четверг, 21 ноября 2024, 11:57

Dear Martina,

thanks for your extensive testing.

We have fixed #2 and #3, but could not reproduce #1. Could you please have a look if it works ok now?

with kind regards,
Timo

Re: planned release of WDQMS 1.5.2

от Martina Suaya - четверг, 21 ноября 2024, 13:12

Good morning!

# 1 is working in wdqms-dev daily OSCAR/hourly options. But on the operational wdmqs.int is not:

Oscar option (Daily and availability) in wdqms.int

And with GREEN filter you can see below that the greys and yellows have disapeared as well:

In the hourly option wdqms.int

All colors look like:

And when filterin green we lost the pinks:

# 3 solved

now the example shows yellow in the map and yellow in the csv

Unknown,83971,,False,-51.16999816894531,-29.989999771118164,6,24,False,yellow,No match in OSCAR/Surface,pressure,2024-11-20 00:00:00+00:00,ECMWF

Thanks

Martina

Re: planned release of WDQMS 1.5.2

от Timo Proescholdt - пятница, 22 ноября 2024, 14:38

Dear Martina, all,

we have fixed the remaining issues in https://wdqms-test.wmo.int and are ready to deploy it.

Based on your feedback, we decided to remove the yellow category from the legend for the hourly/2-daily baseline. Yellow stations are colored according to number of reports received.

with kind regards,

Timo

Re: planned release of WDQMS 1.5.2

от Martina Suaya - пятница, 22 ноября 2024, 16:06

Dear Timo

Thank you for the update. It looks ok

The map with the hourly option I believe is going to be very useful, and not showing the greys and yellows provides a clearer view on how many observations there are.

Thank you and congratulations to you all

Martina