Findings dashboard for all organizations

[madelondohmen]

Changes

This PR adds dashboards to OpenKAT.

In the frontend we've only added the Findings Dashboard to the Crisis Room of all organizations, but the backend is already ready to add all kinds of dashboards to both Crisis Rooms.

This PR adds:

• DashboardData model
• Automatically add a findings dashboard to a new organization
  • Including a ReportRecipe, based on a recipe_seeder
  • This ReportRecipe makes sure the Findings Dashboard is updated regularly
• Use "make dashboard" to add dashboards to all existing organizations
• Findings Report in Aggregate Report
• Findings Report visible in Multi Report
• Make template of Findings Report more generic
• Only show the 5 most recent reports that have been generated when expanding the Scheduled Reports table

Issue link

Closes #3881

Demo

Opname.2025-01-07.102915.mp4

Aggregate Report:

QA notes

New organization

Create a new organization. Creating a new organization would automatically add this organization to the Findings Dashboard (which is shown in the Crisis Room for all organizations). Besides this, on the Scheduled Reports page there should be a ReportRecipe called "Crisis Room Aggregate Report". This should also create a report, which can be seen on the Report History page (this may take a minute). After the report has been created, the results of the report should be visible on the Crisis Room for all organizations. Please make sure that these two matches.

Existing organizations

Existing organizations also need their own dashboard. This can be done by using the command "make dashboard" in rocky. Please check if this adds the existing organizations to the Findings Dashboard (which is shown in the Crisis Room for all organizations).

Findings Report

The Findings Report should work as expected. Also, it should be possible to create a Aggregate and Multi Findings Report. Please check if this work as expected.

---

Code Checklist

• All the commits in this PR are properly PGP-signed and verified.
• This PR only contains functionality relevant to the issue.
• I have written unit tests for the changes or fixes I made.
• I have checked the documentation and made changes where necessary.
• I have performed a self-review of my code and refactored it to the best of my abilities.

• Tickets have been created for newly discovered issues.
• For any non-trivial functionality, I have added integration and/or end-to-end tests.
• I have informed others of any required .env changes files if required and changed the .env-dist accordingly.
• I have included comments in the code to elaborate on what is not self-evident from the code itself, including references to issues and discussions online, or implicit behavior of an interface.

---

Checklist for code reviewers:

Copy-paste the checklist from the docs/source/templates folder into your comment.

---

Checklist for QA:

Copy-paste the checklist from the docs/source/templates folder into your comment.

[stephanie0x00]

QA suggestion before I forget it completely. The QA notes mention the following:

Existing organizations

Existing organizations also need their own dashboard. This can be done by using the command "make dashboard" in rocky. Please check if this adds the existing organizations to the Findings Dashboard (which is shown in the Crisis Room for all organizations)

I think it is a good practice to add a little 'howto' with all the copy paste steps to do this, to the docs. As to make it easier for user to migrate and activate this.

[stephanie0x00]

Found the following 4 things for a clean install on this PR.

1. Crisis room says there is no organisation, while an organisation is present (with a 'make reset').
2. The navigation bar also says "CRISIS ROOM" instead of 'Crisis room' for the overview page of all organisations.
3. Newly added organisations don't show up on the crisis room page (thus an organisation created after completing the onboarding).

4. Running 'make dashboards' doesnt show the dashboards for the organisations (for a clean install on this PR with 2 already added organisations). (Corresponding logs are below)

$ make dashboards 
docker compose run --rm rocky python3 manage.py dashboard
[+] Creating 1/0
 ✔ Container nl-kat-coordination-postgres-1  Running                                                                                                                                                                                   0.0s 
System check identified some issues:

WARNINGS:
crisis_room.DashboardData.position: (fields.W122) 'max_length' is ignored when used with IntegerField.
        HINT: Remove 'max_length' from field
Operations to perform:
  Apply all migrations: account, admin, auth, contenttypes, crisis_room, fmea, knox, otp_static, otp_totp, password_history, sessions, tools, two_factor
Running migrations:
  No migrations to apply.
System check identified some issues:

WARNINGS:
crisis_room.DashboardData.position: (fields.W122) 'max_length' is ignored when used with IntegerField.
        HINT: Remove 'max_length' from field
HTTP Request: GET http://scheduler:8000/schedulers/report-aa "HTTP/1.1 200 OK"
2025-01-13T10:28:05.852708 [info] Dashboard Crisis Room Findings Dashboard for organization aa created object=Crisis Room Findings Dashboard for organization aa object_type=Dashboard
2025-01-13T10:28:05.856813 [info] DashboardData Crisis Room Findings Dashboard for organization aa created object=Crisis Room Findings Dashboard for organization aa object_type=DashboardData
HTTP Request: POST http://bytes:8000/token "HTTP/1.1 200 OK"
HTTP Request: POST http://bytes:8000/bytes/boefje_meta "HTTP/1.1 201 Created"
HTTP Request: POST http://bytes:8000/bytes/raw?boefje_meta_id=b37c6228-e55d-4c82-86e1-977f3402d0be "HTTP/1.1 200 OK"
HTTP Request: POST http://bytes:8000/bytes/normalizer_meta "HTTP/1.1 201 Created"
HTTP Request: POST http://octopoes_api/aa/declarations "HTTP/1.1 200 OK"
2025-01-13T10:28:06.451276 [info] Saved declaration              declaration=Declaration(ooi=ReportRecipe(object_type='ReportRecipe', scan_profile=None, user_id=None, primary_key='ReportRecipe|e32f168e-58a5-4d6a-84dd-accb6fcc326c', recipe_id=UUID('e32f168e-58a5-4d6a-84dd-accb6fcc326c'), report_name_format='Crisis Room Aggregate Report', subreport_name_format='Findings Report for ${ooi}', input_recipe={'query': {'ooi_types': ['IPAddressV6', 'Hostname', 'IPAddressV4', 'URL'], 'scan_level': [1, 2, 3, 4], 'scan_type': ['declared'], 'search_string': '', 'order_by': 'object_type', 'asc_desc': 'desc'}}, parent_report_type='aggregate-organisation-report', report_types=['systems-report', 'findings-report'], cron_expression='0 * * * *'), valid_time=datetime.datetime(2025, 1, 13, 10, 28, 5, 814396, tzinfo=datetime.timezone.utc), end_valid_time=None, method=None, source_method=None, task_id=UUID('4f42103d-31e9-4459-88b1-a9730282fda0')) event_code=100201 organisation_code=aa
HTTP Request: POST http://scheduler:8000/schedules "HTTP/1.1 201 Created"
2025-01-13T10:28:06.478453 [info] Schedule created               event_code=800081 schedule=ScheduleRequest(scheduler_id='report-aa', data={'type': 'report', 'organisation_id': 'aa', 'report_recipe_id': 'e32f168e-58a5-4d6a-84dd-accb6fcc326c'}, schedule='0 * * * *', deadline_at='2025-01-13T10:28:05.814396+00:00')
2025-01-13T10:28:06.483549 [info] DashboardData Crisis Room Findings Dashboard for organization aa updated object=Crisis Room Findings Dashboard for organization aa object_type=DashboardData
HTTP Request: GET http://scheduler:8000/schedulers/report-bb "HTTP/1.1 200 OK"
2025-01-13T10:28:06.527447 [info] Dashboard Crisis Room Findings Dashboard for organization bb created object=Crisis Room Findings Dashboard for organization bb object_type=Dashboard
2025-01-13T10:28:06.530188 [info] DashboardData Crisis Room Findings Dashboard for organization bb created object=Crisis Room Findings Dashboard for organization bb object_type=DashboardData
HTTP Request: POST http://bytes:8000/token "HTTP/1.1 200 OK"
HTTP Request: POST http://bytes:8000/bytes/boefje_meta "HTTP/1.1 201 Created"
HTTP Request: POST http://bytes:8000/bytes/raw?boefje_meta_id=418f62c0-3bc6-46c5-9955-73c39bc03b98 "HTTP/1.1 200 OK"
HTTP Request: POST http://bytes:8000/bytes/normalizer_meta "HTTP/1.1 201 Created"
HTTP Request: POST http://octopoes_api/bb/declarations "HTTP/1.1 200 OK"
2025-01-13T10:28:07.135650 [info] Saved declaration              declaration=Declaration(ooi=ReportRecipe(object_type='ReportRecipe', scan_profile=None, user_id=None, primary_key='ReportRecipe|d761caf6-e0a2-4fe7-8edf-4de548589cdf', recipe_id=UUID('d761caf6-e0a2-4fe7-8edf-4de548589cdf'), report_name_format='Crisis Room Aggregate Report', subreport_name_format='Findings Report for ${ooi}', input_recipe={'query': {'ooi_types': ['IPAddressV6', 'Hostname', 'IPAddressV4', 'URL'], 'scan_level': [1, 2, 3, 4], 'scan_type': ['declared'], 'search_string': '', 'order_by': 'object_type', 'asc_desc': 'desc'}}, parent_report_type='aggregate-organisation-report', report_types=['systems-report', 'findings-report'], cron_expression='0 * * * *'), valid_time=datetime.datetime(2025, 1, 13, 10, 28, 6, 484379, tzinfo=datetime.timezone.utc), end_valid_time=None, method=None, source_method=None, task_id=UUID('d10b99bb-900d-4cf1-ad49-ffd45e9c01cc')) event_code=100201 organisation_code=bb
HTTP Request: POST http://scheduler:8000/schedules "HTTP/1.1 201 Created"
2025-01-13T10:28:07.176510 [info] Schedule created               event_code=800081 schedule=ScheduleRequest(scheduler_id='report-bb', data={'type': 'report', 'organisation_id': 'bb', 'report_recipe_id': 'd761caf6-e0a2-4fe7-8edf-4de548589cdf'}, schedule='0 * * * *', deadline_at='2025-01-13T10:28:06.484379+00:00')
2025-01-13T10:28:07.178780 [info] DashboardData Crisis Room Findings Dashboard for organization bb updated object=Crisis Room Findings Dashboard for organization bb object_type=DashboardData

[ammar92]

Just a few suggestions:

• Based on the dashboards make target and the output of this command, dashboards would be a better command name. I would suggest renaming this module to dashboards.py.
• There's a lot happening here when the command is run, but there are no specific logs or prints that give some feedback to the user. I would expect at least something like "Creating dashboard for organization X..." when the command is creating all the dashboards

[ammar92]

Return type indicates that this is a date time type, not a date.

Suggested change

	@register.filter
	def get_date(date_str: str) -> datetime:
	return datetime.fromisoformat(date_str)
	@register.filter
	def get_date(date_str: str) -> date:
	return datetime.fromisoformat(date_str).date

[Rieven]

@madelondohmen is the idea here to return a datetime object? if yes, we should make this function name: get_datetime instead. The function name says to return a date only object as that was the intention or renaming the function to return indeed what it was for .

[Rieven]

QA suggestion before I forget it completely. The QA notes mention the following:

Existing organizations

Existing organizations also need their own dashboard. This can be done by using the command "make dashboard" in rocky. Please check if this adds the existing organizations to the Findings Dashboard (which is shown in the Crisis Room for all organizations)

I think it is a good practice to add a little 'howto' with all the copy paste steps to do this, to the docs. As to make it easier for user to migrate and activate this.

The dos has been updated

[Rieven]

3. Newly added organisations don't show up on the crisis room page (thus an organisation created after completing the onboarding).

You have to run make dashboards again after adding the new organization. What we did before is that when a new organization is added the dashboard will be created, but just after creating a new dashboard no objects are added yet, the schedule will create empty reports with 0 objects. Then it came to be that the user decides when to create the dashboard.

For already existing organization it will update it:

[Rieven]

4. Running 'make dashboards' doesn't show the dashboards for the organisations (for a clean install on this PR with 2 already added organisations). (Corresponding logs are below)

Running make dashboards does not ensure that findings data is available for the organization. The recipe and schedule for every hour must have also input objects to generate findings when creating the findings report through the scheduler. We show only findings data that is available for the organization.

[sonarqubecloud]

Quality Gate failed

Failed conditions
67.4% Coverage on New Code (required ≥ 80%)
8.5% Duplication on New Code (required ≤ 3%)

See analysis details on SonarQube Cloud