Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Enable exporting (registered) submission metadata #4962

Open
wants to merge 4 commits into
base: master
Choose a base branch
from

Conversation

sergei-maertens
Copy link
Member

@sergei-maertens sergei-maertens commented Dec 20, 2024

Closes #4930

Changes

  • Added custom admin page and link to it from the form statistics
  • Implemented exporting submission statistics based on timeline logs

Checklist

Check off the items that are completed or not relevant.

  • Impact on features

    • Checked copying a form
    • Checked import/export of a form
    • Config checks in the configuration overview admin page
    • Problem detection in the admin email digest is handled
  • Release management

    • I have labelled the PR as "needs-backport" accordingly
  • I have updated the translations assets (you do NOT need to provide translations)

    • Ran ./bin/makemessages_js.sh
    • Ran ./bin/compilemessages_js.sh
  • Dockerfile/scripts

    • Updated the Dockerfile with the necessary scripts from the ./bin folder
  • Commit hygiene

    • Commit messages refer to the relevant Github issue
    • Commit messages explain the "why" of change, not the how

Copy link

codecov bot commented Dec 20, 2024

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 96.63%. Comparing base (9c6c899) to head (5627a76).

Additional details and impacted files
@@            Coverage Diff             @@
##           master    #4962      +/-   ##
==========================================
+ Coverage   96.62%   96.63%   +0.01%     
==========================================
  Files         760      762       +2     
  Lines       25837    25925      +88     
  Branches     3385     3389       +4     
==========================================
+ Hits        24964    25052      +88     
  Misses        608      608              
  Partials      265      265              

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

Export submission statistics based on the timeline logs.

Note: if the timeline logs are pruned, this affects the exports. It's
up to the users to periodically create these exports and save them
somewhere if they periodically prune log records.

Note 2: filtering on forms only works on new log records, as existing
log records don't have the form ID stored in the structured data.

Submissions that were deleted for which existing log records are
present will display 'unknown' for some columns because the relevant
information has been deleted. Only from 3.0 onwards are we snapshotting
the data required for the exports.
Ensures that the filters are correctly implemented.
@sergei-maertens sergei-maertens marked this pull request as ready for review December 20, 2024 16:09
end_date: date = form.cleaned_data["end_date"]
dataset = form.export()
format = XLSX()
filename = f"submissions_{start_date.isoformat()}_{end_date.isoformat()}.xlsx"
Copy link
Contributor

@robinmolen robinmolen Dec 24, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think it would be nice to add the form names, if the limit is used

def export(self) -> Dataset:
start_date: date = self.cleaned_data["start_date"]
end_date: date = self.cleaned_data["end_date"]
return export_registration_statistics(
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe a bit picky, but why do you define the dates as variables? Is it (only) to specify the date typing?
Otherwise, why not just pass the cleaned_data directly?

@@ -34,7 +34,7 @@ def export_registration_statistics(
)

_start_date = make_aware(datetime.combine(start_date, time.min))
_end_date = make_aware(datetime.combine(end_date, time.max))
_end_date = make_aware(datetime.combine(end_date, time.min))
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why not till the end of the day?

@robinmolen
Copy link
Contributor

robinmolen commented Dec 24, 2024

I've tried the export, and was missing some submissions.. I'm not yet sure why, i'll take another look later today..

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

As admin, I want to create an export of all submissions to compare them with registration systems
2 participants