MVP ready

[tsundvoll]

Main goal:

• Ability to show an image in Flotilla. Provide the Flotilla backend with a link to an inspection data (in a blob container), which it can then serve to the Flotilla frontend. The image needs to have been anonymized beforehand.

What is required:

• Storage accounts
  • Raw data blob; configure isar-anymal to upload to these containers (nls)
  • Long term storage (anonymized)
• Database for IDA
  • inspection_id (from ISAR and Flotilla)
  • link to raw data (short term, to be deleted)
  • boolean: has_raw_data_been_deleted
  • link to actual data (anonymized long term storage)
  • (analysis to be run)
  • (performed analysis)
  • (list of findings (with link to findings visualization))
• .NET application
  • Endpoints (.NET) (Create .NET backend #10)
    • GET anonymized-inspection-data/{inspection_id} -> Link to anonymized data; to be used by Flotilla backend
    • POST workflows_started (to be used by the workflows [This job has started], refers to the inspection_id)
    • POST workflows_finished (to be used by the workflows [Successful / Failed], refers to the inspection_id)
    • (POST new-analysis/{inspection_id})
  • MQTT service
    • Listening for messages to the /inspection_result topic (when something is uploaded to raw; with info about filepath)
  • Talking to the database
• Automatic jobs (flows)
  • Anonymizer (Python)
    • Trigger: IDA listens to MQTT message and trigger a workflow. Whenever a new inspection data is uploaded to raw_data, run the anonymizer job. IDA decides / generates the location of the result after the anonymization and sends this as a parameter to the workflow. In this way IDA will have the overview of ongoing / finished workflows and data location.
    • The workflow
      • Parameters (source_path, destination_path, inspection_id)
      • POST to Flotilla: workflows_started
      • Looks for people in images and mark them with a black box
      • Stores the result in long_term_storage
      • POST to Flotilla: workflows_done
    • (Delete the inspection data from raw_data and delete the link to raw data from the database)
  • Raw data clean up
    • Batch job cleaning up the raw data that has not been deleted automatically after 7 days. Log this as a mistake, as these data should have been deleted. (or flag it for deletion)

Unknows:

• Secrets in cluster
• Is the .NET deployed successfully now?
• How the MQTT message is sent?
• Push content til auroradevacr (from GitHub Actions)
• Fjerne Ouath fra IDA og set network policy

TODOs:

• Infrastructure as code for the database, storage accounts etc.

The flow:

• isar-anymal uploads data to blob storage (raw data storage account; nls container)
• isar publishes message to MQTT channel
• IDA subscribes to this message over MQTT
• workflow for anonymization is triggered
• anonymizer reads raw image, anonymize it and stores to the anonymized data storage account; nls container)
• IDA stores the link to the anonymized image in its database
• IDA can respond to the GET anonymized-inspection-data/{inspection_id} with a link to the relevant blob
• Flotilla request this and displays the image when the user click on the inspection task