Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat/HIT26_GDPR-anonymity-of-beneficiaries #23

Open
wants to merge 6 commits into
base: hit-oort
Choose a base branch
from
Open

feat/HIT26_GDPR-anonymity-of-beneficiaries #23

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
1146746
add: lastLogin in model and update when user login
GuilhermeGabriel Jan 15, 2024
a7c0ae4
feat: add cron job that delete unwanted data
GuilhermeGabriel Jan 16, 2024
8e7a8b2
refactor + few improvements
matheus-relief Jan 21, 2024
b915416
minor fixes
matheus-relief Jan 21, 2024
b82e95b
fix: beneficiaries logic
GuilhermeGabriel Jan 23, 2024
d93ec24
fix: description Anonymize beneficiaries cronjob
GuilhermeGabriel Jan 23, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
101 changes: 55 additions & 46 deletions src/jobs/anonymizeBeneficiaries.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,63 +3,72 @@ import { Types } from 'mongoose';

/** Staff resource ID */
const AID_RESOURCE_ID = new Types.ObjectId('64e6e0933c7bf3962bf4f04c');
const FAMILY_RESOURCE_ID = new Types.ObjectId('64de75fd3fb2a11c988dddb2');
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

missing jsdoc, don't forget to always run npm run lint before marking the PR as ready for review


/** Anonymizes the beneficiary data, if didn't log in for more than 18 months */
export const anonymizeBeneficiaries = async () => {
// Find all the records of Ais was given more than 18 months ago
const allAids = await Record.find({
resource: AID_RESOURCE_ID,
createdAt: {
$lt: new Date(Date.now() - 18 * 30 * 24 * 60 * 60 * 1000),
}, // 18 months ago
});
// For all family records, check if
// in the last 18 months they received aid

// Anonymize all members of that family
for (let i = 0; i < allAids.length; i++) {
const aidRecord = allAids[i];
// Get all the family records
const allFamilies = await Record.find({
resource: FAMILY_RESOURCE_ID,
});

// Get Family record
const familyRecord = await Record.findOne({
_id: aidRecord?.data?.owner_resource,
// For each family record, check if exists
// an aid record in the last 18 months
for (const family of allFamilies) {
const aidGivenToFamily = await Record.exists({
resource: AID_RESOURCE_ID,
createdAt: {
$gt: new Date(Date.now() - 18 * 30 * 24 * 60 * 60 * 1000),
}, // 18 months ago
'data.owner_resource': family._id.toString(),
});

// Find all members of the family
const members = await Record.find({
_id: { $in: familyRecord?.data?.members },
});
// If no aid was given to the family in the last 18 months
if (!aidGivenToFamily) {
// Find all members of the family
const members = await Record.find({
_id: { $in: family?.data?.members },
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This would throw an error if family?.data?.members resolves to undefined, as the $in operator requires an array as value

Copy link
Collaborator

@matheus-relief matheus-relief Jan 30, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Would also be nice to check if the record is in the Person form, but no big deal

});

// Anonymize all members of that family
for (let j = 0; j < members.length; j++) {
const member = members[j];
// Anonymize all the members
members.forEach((member) => {
if (!member.data) {
return;
}
// Anonymize the member
member._createdBy = new User({
name: 'ANONYMOUS',
username: `${member._id.toString()}@oort-anonymous.com`,
});
Comment on lines +41 to +45
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think we need this, the createdBy would be a staff member, and their anonymization is already handled in the other function


// Anonymize the member
member._createdBy = new User({
name: 'ANONYMOUS',
username: `${Math.random()
.toString(36)
.substring(2, 15)}@anonymus-oort.com`,
});
member.data.location = 'ANONYMOUS';
member.data.surname = 'ANONYMOUS';
member.data.firstname = 'ANONYMOUS';
member.data.phone = 'ANONYMOUS';
member.data.nom_employes = 'ANONYMOUS';
member.data.gender = 'ANONYMOUS';
member.data.birthdate = 'ANONYMOUS';
member.data.prenom_employes = 'ANONYMOUS';
member.data.nom_prenom_employes = 'ANONYMOUS';
member.data.tel_staff = 'ANONYMOUS';
member.data.email_staff = 'ANONYMOUS';
member.data.birthdate_employes = 'ANONYMOUS';
member._lastUpdatedBy = new User({
name: 'ANONYMOUS',
username: `${Math.random()
.toString(36)
.substring(2, 15)}@anonymus-oort.com`,
member.data = {
...member.data,
location: 'ANONYMOUS',
surname: 'ANONYMOUS',
firstname: 'ANONYMOUS',
phone: 'ANONYMOUS',
nom_employes: 'ANONYMOUS',
gender: 'ANONYMOUS',
birthdate: 'ANONYMOUS',
prenom_employes: 'ANONYMOUS',
nom_prenom_employes: 'ANONYMOUS',
tel_staff: 'ANONYMOUS',
email_staff: 'ANONYMOUS',
birthdate_employes: 'ANONYMOUS',
file_gdpr_staff: [],
};
Comment on lines +47 to +62
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

When anonymizing the data, we don't need to remove fields that are not traceable back to the beneficiary. e.g: their location and gender.

Also, make sure to only save valid data. For the birthdate for example, it should always be a valid date, check the other anonymization function to see how it should be dealt with. The email should also be a valid email, you can use the same logic of [id]@oort-anonymous.com. To make sure that all fields have valid data, you can create a record and see how the data is stored in the db.

And finally, it looks like there are fields here that are not part of the form (and it's possibly missing some that should be there, but I didn't check)


member._lastUpdatedBy = new User({
name: 'ANONYMOUS',
username: `${member._id.toString()}@oort-anonymous.com`,
});
Comment on lines +64 to +67
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same here, can be removed. And actually, another change you should make is doing this but in the staff anonymization, so when anonymizing a staff, you have to check all records created or last updated by them and updated the _lastUpdatedby and the _createdBy. However, do not create a new user, as that creates a new id, and we do not want that.

});
member.data.file_gdpr_staff = [];

await member.save();
// Save all the records
await Record.bulkSave(members);
}
}
};
9 changes: 9 additions & 0 deletions src/jobs/index.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,5 +1,6 @@
import { CronJob } from 'cron';
import { anonymizeStaff } from './anonymizeStaff';
import { anonymizeBeneficiaries } from './anonymizeBeneficiaries';
import { logger } from '@services/logger.service';
import config from 'config';

Expand All @@ -21,6 +22,14 @@ const JOBS: {
fn: anonymizeStaff,
envs: ['alimentaide'],
},
{
name: 'Anonymize beneficiaries',
description: '',
// Every week
schedule: '0 0 * * 0',
fn: anonymizeBeneficiaries,
envs: ['alimentaide'],
},
];

/** Starts all the jobs */
Expand Down