Discussion of cross-platform flight segment kinds

[RobertPincus]

This issue is to discuss the list of flight segment kinds used across platforms and listed in Readme.md.

Suggestions, comments, refinements welcome.

[leosaffin]

I think it's nice that the current primary segment kinds mostly all describe the style/pattern of flying and the secondary descriptors cover extra details like specifics of the pattern, instruments, and meteorology. There's two that maybe stand out from that.

1. Cloud. There was discussion on the mattermost as to whether this is distinct from other patterns that happen to pass through cloud. For the twinotter (and possibly the P3), I would say this is a distinct pattern because the flying is targeted at clouds as opposed to flying straight and level. On the other hand, it could be a second category under level, allowing us to say "level, cloud" or "profile, cloud" etc.
2. axbt. I don't know too much about how this works. Do you fly a level leg while dropping these? In which case I would say it could be a subcategory similar to dropsonde for circle. Or is it a specific pattern flown that means there's no point looking at other measurements. In which case, I would say keep it.

[leosaffin]

For the twinotter it may become useful to include very short segments within cloud legs for individual cloud penetrations. We have said we shouldn't have the same primary flight segment kind covering the same time though. Any good suggestions for how I should label these?

[RobertPincus]

@LSaffin Re "very short segments within cloud legs" one could define an "in cloud" segment kind. We might use this for the P3 as well.

[RobertPincus]

@LSaffin You're right that the "primary segment kinds mostly all describe the style/pattern of flying". The cloud and axbt kinds are based on P3 flight patterns; in our case cloud is normally stacked legs and axbt is a lawn-mower pattern (offset straight and level legs at FL9-FL10). Only the P3 will use axbt and that's ok; only the Twin Otter will use sawtooth. But the Twin Otter also flies cloud sampling legs. Should we use the same primary name cloud for the P3 and the TO to emphasize similar goals, or different names to emphasize different approaches? Maybe @gdeboer2 has thoughts too.

[RobertPincus]

@LSaffin @gdeboer2 Might we re-think level as a primary tag? It doesn't follow the idea of "describing the objective", also many of the other patterns (circle, axbt) are also level. Should we replace it with a transit kind? This would still meet the ATR's needs to denote ferry legs. Users could have the expectation that transit legs would normally be more-or-less straight and level. Your thoughts?

[d70-t]

I am not yet confident if a hierarchy of kinds or an exclusion between kinds or the separation into primary or secondary is really what we want. At least maybe not in a formal way. E.g. a calibration flight can deliberately be level in order to perform the calibration. So I'd argue that while it is true that some kinds are more specific and others are more general and some kinds will not mix in a physically sensible way with other kinds, enforcing that in a formal way seems to be problematic.

I think I am more a fan of suggesting to order the kinds roughly along something like specificity (is that even a word?) that allows to express kind of a hierarchy in a more ad-hoc manner and programs using the data (e.g. for displaying a report) could easily select something like the most specific kind which the programs knows how to display.

If we still want to do formal checks about the kinds, we could add and exclusion-table somewhere (e.g. ascending and descending maybe should never be mixed...).

[RobertPincus]

@d70-t You make a good point about why we might not want a hierarchy. My aim in a set of primary kinds was to be able to write codes like report.py that expect some set of segments to be non-overlapping, and also to have a common vocabulary across platforms. But perhaps this isn't needed?

The only formal validation I was imagining (#4) was that no segment has more than one primary kind and the start and end times of all segments other than ground lie between take-off and landing.

[RobertPincus]

... I was partly imagining that, if people keep adding to the YAML files, a given time window might belong to three or more kinds (level, cloud layer in cloud, below inversion ferry) and plots would be difficult to read

[leosaffin]

I feel like I should comment to say that I am still following but I haven't come up with any good suggestions yet. I think in_cloud is good but then that would be it's primary kind because it is already part of a level leg. I would vote against using transit because I think that might be confusing as I was using it to describe going to/from the circle. Maybe straight_and_level?

There was another question of which of

1. underscores
2. dashes
3. spaces
   to use in the naming. That would be my preference order.

[RobertPincus]

@LSaffin Thanks for the vote for underscores.

I'd understand in_cloud as being secondary, part of a cloud or cloud_sampling module.

My problem with level is that it doesn't describe the sampling objective, which all the other primary kinds do.

[leosaffin]

@RobertPincus
For cloud/cloud_sampling I was thinking I would use that to describe the longer period of time when we were flying at cloud level in and out of cloud. The issue is that that I can't reuse that to have a separate set of segments to describe the subset of the cloud/cloud_sampling segment which is in cloud because the kinds can't overlap.

I can see the issue with level. For the twinotter, the times we would be flying level (and straight) are: transits to/from the circle, boundary layer (subcloud layer) and cold pools. I would think of those as secondary kind but I can't think of a good primary kind that might cover those. Does that help?

[d70-t]

@d70-t You make a good point about why we might not want a hierarchy. My aim in a set of primary kinds was to be able to write codes like report.py that expect some set of segments to be non-overlapping, and also to have a common vocabulary across platforms. But perhaps this isn't needed?

The only formal validation I was imagining (#4) was that no segment has more than one primary kind and the start and end times of all segments other than ground lie between take-off and landing.

Within the HALO segmentation, our only rule has been that there may be no two segments overlapping segments of the same kind. But any segments of different kinds may overlap. I see the point that this could make plotting difficult, but in the past I've always made the experience that designing storage formats in a way that best describes the reality is far superior when compared to designing it such that it is perfect for a specific use case. A mutual exclusion list based on physical arguments would be in line with this argument. And I think would fit better than a separation into primary, secondary, n-ary kinds. We could do something like:

things_which_dont_belong_together.yaml

- [ground, level]
- [ascending, descending]
- [sawtooth, level, profile, circle]

The semantics would be that every sub-list (e.g. each line of the above) lists things which may not be used together.

Actually, that brings up another (subtle) question of what "may not be used together would mean". It could either be that

• any such pair may not be used together in the kinds attribute of one segment or it could mean that
• on one platform no point in time may be marked with any pair of those.

[d70-t]

@LSaffin I'd prefer spaces as they look nicer. But seemingly we didn't do that for the HALO segments... In the end, I think I would be fine with any of these choices.

[gdeboer2]

Trying to catch up on this discussion... This seems to be getting complicated. The thing that seems to make most sense to me is to first describe the aircraft attitude, and then secondarily describe the objective. For example, at level 1: - Is aircraft climbing or descending - Is aircraft turning Then, layer that with all of the objectives you want (e.g. cloud, dropsondes, axbt, etc.). FWIW, to me “level” is straight and level. Therefore, only parts of axbt are level, as well as only parts of circle (if flown as a polygon as was the case with the P3). This may just be my lack of understanding of YAML, but I don’t see the need for any hierarchy.

…

On Jul 31, 2020, at 9:05 AM, d70-t ***@***.***> wrote: @LSaffin <https://github.com/LSaffin> I'd prefer spaces as they look nicer. But seemingly we didn't do that for the HALO segments... In the end, I think I would be fine with any of these choices. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#2 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AO3B45WIGW74N5APFMPS4ATR6LMT3ANCNFSM4PONCV7Q>.

[RobertPincus]

@gdeboer2 @LSaffin @d70-t: The point of the flight segmentation files is to help uses identify broad times for analysis that might not be easily identified automatically. We would coordinate kinds to the extent that it helps communicate similar sampling goals.

There is nothing to keep particular platforms from including segments that define e.g. climbing, depending, turning at any desired level of granularity for that platform. Here we'd think of climbing as a state of the plane as different (and more frequent than) profile as a sampling strategy.

I thought that identifying a list of primary kinds would help make tools more precise but perhaps this knowledge can live with the tool. What if, instead if distinguishing primary/secondary, we took @d70-t suggestion and made lists of mutually exclusive types and maybe always-together types, i.e. profile ascending must also be a profile?

[gdeboer2]

Right, I understand. But it seems to me that there will be interest in finding all points that span some of these more specific descriptions (e.g. all flight times when the aircraft is “level”). I like the idea of starting with the most exclusive layer and then refining from there. For example, an aircraft can not be climbing and descending at the same time, but can be profiling for both. I don’t understand your last example — that seems redundant to me. If already labeled a profile, why would we need to also somehow show it is always a profile? Gijs

…

On Jul 31, 2020, at 10:11 AM, Robert Pincus ***@***.***> wrote: @gdeboer2 <https://github.com/gdeboer2> @LSaffin <https://github.com/LSaffin> @d70-t <https://github.com/d70-t>: The point of the flight segmentation files is to help uses identify broad times for analysis that might not be easily identified automatically. We would coordinate kinds to the extent that it helps communicate similar sampling goals. There is nothing to keep particular platforms from including segments that define e.g. climbing, depending, turning at any desired level of granularity for that platform. Here we'd think of climbing as a state of the plane as different (and more frequent than) profile as a sampling strategy. I thought that identifying a list of primary kinds would help make tools more precise but perhaps this knowledge can live with the tool. What if, instead if distinguishing primary/secondary, we took @d70-t <https://github.com/d70-t> suggestion and made lists of mutually exclusive types and maybe always-together types, i.e. profile ascending must also be a profile? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#2 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AO3B45SMQP2V7LGX6KGRY5DR6LUMLANCNFSM4PONCV7Q>.

[RobertPincus]

@LSaffin

1. Re clouds, isn't the idea to have one long cloud segment that might contain many shorter in_cloud segments?
2. Maybe subcloud layer could be the general term and and some of these would also be labeled cold pools?

[RobertPincus]

@gdeboer2 Not sure if this addresses the question but I was suggesting that every profile ascending (a kind in use by the ATR) should also be coded a profile which we'd use uniformly across platforms. For some applications one only wants to know that the plane is steadily changing altitude; for others it may be useful to know that it's going up. The same would apply to calibration: all specific types of calibration segments (calibration_lidar) would also be (calibration). This is something we can check ard/or fix automatically.

... although I guess a profile segment might also encompass a profile ascending and profile descending... though we have the same issue for the mutually exclusive case.

[d70-t]

@gdeboer2 yes, I thing this discussion can escalate quickly :-)

So the thing is, that a hard definition of primary, secondary is not really possible for the general idea of segment kinds. Maybe we are also talking about some slightly different things. If we explicitly want some variable which describes the general attitude, then there should be a scalar-valued (i.e. not a list) attitude attribute. This would then need to be defined exactly once and can only have a single value. The kinds are more like tags you would attach to the segments. They shouldn't have too much explicit structure.

@RobertPincus I am with you that it may be a good idea to express more complex relations between kinds for the verification. But maybe we should move the verification aspect to a separate issue in oder to reduce complexity in this one.

[RobertPincus]

@d70-t We do have an issue (#4) for validation. Agreed it can be separate.

If we have lists of mutually-exclusive segment types I'm not sure a hierarchy is needed.

[SBony]

I like the idea of having no hierarchy, and of defining a mutual exclusion list.
It would be very nice if the different platforms could use as much as possible (and as much as it makes sense) commonly defined kind names, and could follow a few common rules (such as no overlapping segments of the same kind as proposed), but otherwise we should try to keep as flexible as possible so that each platform can define its own additional sub segments as is most convenient for its own scientific objectives.