HDX-9984 Allow key specification to be nested

[IanHopkinson]

Purpose

Version for this PR: 2024.7.1

This PR provides functionality to access nested keys in the list command, typically the nested keys of the organization dictionary, or the keys of lists such as tags and groups.

#5

Major file changes

The major change is the introduction of a new function, query_dict which is added to utilities.py and tested in test_utilities.py. It is invoked in the "list" command in cli.py:

hdx-cli-toolkit/src/hdx_cli_toolkit/cli.py

Line 102 in 28fe9d8

@hdx_toolkit.command(name="list")

Minor file changes

As a result of changing the global git config handling line endings, all(?) many lines of code appeared to have changed but I believe this is just a line ending change.

Versioning

hdx-cli-toolkit uses the CalVer versioning scheme with format YYYY.MM.Micro i.e. 2022.12.1 which is updated manually in pyproject.toml. The "Micro" component is simply an integer increased by 1 at each version, starting from 0.

• Version updated in pyproject.toml and PR description
• Update README.md and DEMO.md with any new CLI commands

[IanHopkinson]

Pipeline failures are mainly down to the stage API key changing a couple of times!

[turnerm]

I'm glad I got to do this review, it was my first time using the hdx toolkit, and it's really slick, I'll definitely be keeping it in my back pocket!

Overall I think this change is a tricky one because it opens up a lot of different possibilities for queries. Below are some ones that I tried:

First, there appears to be a duplication bug, e.g. the following returns two rows:

hdx-toolkit list \
  --dataset_filter=gibraltar-healthsites --hdx_site=stage \
  --key=archived,batch

I suspect it comes from using both output and output_row in the query_dict function.

This returns a dictionary that doesn't fit into the table:

hdx-toolkit list \
  --dataset_filter=gibraltar-healthsites --hdx_site=stage \
  --key=resources \
  --with_extras

This returns a too many values error that should perhaps be caught or otherwise handled:

hdx-toolkit list \
  --dataset_filter=gibraltar-healthsites \
  --hdx_site=stage \
  --key=resources.fs_check_info.state \
  --with_extras

This returns 5 rows of absent key:

hdx-toolkit list \
  --dataset_filter=gibraltar-healthsites \
  --hdx_site=stage \
  --key=resources.nonsense \
  --with_extras

[IanHopkinson]

@turnerm here's my response to your test cases:

1. archived,batch – produced two lines a bug – FIXED
2. resources – produced a dictionary that didn’t fit in the table – it will output correctly to file – FIXED by providing an information message:
   Field 'resources' is list or dict type, use --output_path to see full output
3. resources.fs_check_info.state – fails because it is 3 keys deep and first two are lists – FIXED by providing message:
   resources.fs_check_info.state is nested to depth 3, maximum depth is 2
4. resources.nonsense – returns five rows with “absent key” – I think this behaviour is correct, we may have situations where some items in the list have the key but others don't - WONTFIX