Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

segment-gatherer does not respect critical_segments in config #99

Open
TAlonglong opened this issue Dec 10, 2021 · 1 comment
Open

segment-gatherer does not respect critical_segments in config #99

TAlonglong opened this issue Dec 10, 2021 · 1 comment

Comments

@TAlonglong
Copy link
Collaborator

As far as I can see segment gatherer does not respect critical_segments in config using ini config

[metopc-ears-avhrr-pps-hrw]
pattern = S_NWC_{segment}_metopc_{orbit_number:05d}_{start_time:%Y%m%dT%H%M%S}{start_frac:1s}Z_{end_time_}Z.nc
critical_files = :CT,:CMA,:CTTH,:avhrr
wanted_files = :CT,:CMA,:CTTH,:avhrr
all_files = :CT,:CMA,:CTTH,:avhrr
topics = /tpcs
publish_topic = /PT
timeliness = 1200
time_name = start_time

Example from log file:

[WARNING: 2021-12-10 12:42:18 : segment_gatherer] Missing files: S_NWC_avhrr_metopc_00000_20211210T1133000Z_20211210T1133413Z.nc
[INFO: 2021-12-10 12:42:18 : segment_gatherer] Sending: pytroll://PPSV2021/EARS/GATHERED/AVHRR/PPS/HRW dataset ubuntu@pps-v2021-a 2021-12-10T12:42:18.808006 v1.01 application/json {"orbit_number": 0, "start_time"
: "2021-12-10T11:33:00", "start_frac": "0", "end_time_": "20211210T1133413", "platform_name_number": 3, "process_level": "l1b", "antenna": "ears", "platform_name": "Metop-C", "end_time": "2021-12-10T11:34:00", "d
isposition_mode": "O", "process_time": "2021-12-10T11:34:31", "origin": "157.249.97.83:9152", "format": "CF", "type": "netCDF4", "data_processing_level": "2", "station": "oslo", "env": "ears", "orig_orbit_number"
: 16046, "dataset": [{"uri": "/data/pytroll/nwcsaf/export/S_NWC_CMA_metopc_00000_20211210T1133000Z_20211210T1133413Z.nc", "uid": "S_NWC_CMA_metopc_00000_20211210T1133000Z_20211210T1133413Z.nc"}, {"uri": "/data/py
troll/nwcsaf/export/S_NWC_CT_metopc_00000_20211210T1133000Z_20211210T1133413Z.nc", "uid": "S_NWC_CT_metopc_00000_20211210T1133000Z_20211210T1133413Z.nc"}, {"uri": "/data/pytroll/nwcsaf/export/S_NWC_CTTH_metopc_00
000_20211210T1133000Z_20211210T1133413Z.nc", "uid": "S_NWC_CTTH_metopc_00000_20211210T1133000Z_20211210T1133413Z.nc"}], "sensor": ["avhrr/3"]}

So I get this message when the timeout is reached. But I'm sure sure what the actual meaning if critical_segments is. Should this never be sendt?

python                    3.9.7           hb7a2778_3_cpython    conda-forge
pytroll-collectors        0.11.1                   pypi_0    pypi
@pnuu
Copy link
Member

pnuu commented Dec 10, 2021

The message should not be sent if any of the critical segments are missing. I think I've seen this too, but forgot to fix/create an issue, so thanks for the issue!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants