Skip to content

Commit

Permalink
Merge branch 'main' into andres/HJ-250
Browse files Browse the repository at this point in the history
  • Loading branch information
andres-torres-marroquin authored Dec 11, 2024
2 parents d047957 + e6609fa commit 5c87f32
Show file tree
Hide file tree
Showing 159 changed files with 9,539 additions and 4,634 deletions.
6 changes: 3 additions & 3 deletions .github/workflows/backend_checks.yml
Original file line number Diff line number Diff line change
Expand Up @@ -244,6 +244,7 @@ jobs:
Safe-Tests:
needs: Check-Container-Startup
strategy:
fail-fast: false
matrix:
python_version: ["3.9.18", "3.10.13"]
test_selection:
Expand All @@ -255,7 +256,6 @@ jobs:

runs-on: ubuntu-latest
timeout-minutes: 45
continue-on-error: true
steps:
- name: Download container
uses: actions/download-artifact@v4
Expand Down Expand Up @@ -397,8 +397,8 @@ jobs:
# Secrets to pull from 1Password
BIGQUERY_DATASET: op://github-actions/bigquery/BIGQUERY_DATASET
BIGQUERY_KEYFILE_CREDS: op://github-actions/bigquery/BIGQUERY_KEYFILE_CREDS
BIGQUERY_ENTERPRISE_DATASET: op://github-actions/bigquery-enterprise/BIGQUERY_DATASET
BIGQUERY_ENTERPRISE_KEYFILE_CREDS: op://github-actions/bigquery-enterprise/BIGQUERY_KEYFILE_CREDS
BIGQUERY_ENTERPRISE_DATASET: op://github-actions/bigquery-enterprise/BIGQUERY_ENTERPRISE_DATASET
BIGQUERY_ENTERPRISE_KEYFILE_CREDS: op://github-actions/bigquery-enterprise/BIGQUERY_ENTERPRISE_KEYFILE_CREDS
DYNAMODB_ACCESS_KEY_ID: op://github-actions/dynamodb/DYNAMODB_ACCESS_KEY_ID
DYNAMODB_ACCESS_KEY: op://github-actions/dynamodb/DYNAMODB_ACCESS_KEY
DYNAMODB_ASSUME_ROLE_ARN: op://github-actions/dynamodb/DYNAMODB_ASSUME_ROLE_ARN
Expand Down
40 changes: 36 additions & 4 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -15,12 +15,47 @@ The types of changes are:
- `Fixed` for any bug fixes.
- `Security` in case of vulnerabilities.

## [Unreleased](https://github.com/ethyca/fides/compare/2.50.0...main)
## [Unreleased](https://github.com/ethyca/fides/compare/2.51.1...main)

### Added
- New page in the Cookie House sample app to demonstrate the use of embedding the FidesJS SDK on the page [#5564](https://github.com/ethyca/fides/pull/5564)
- Added new erasure tests for BigQuery Enterprise [#5554](https://github.com/ethyca/fides/pull/5554)

### Fixed
- Fixing quickstart.py script [#5585](https://github.com/ethyca/fides/pull/5585)

## [2.51.1](https://github.com/ethyca/fides/compare/2.51.0...2.51.1)

### Fixed
- SaaS integrations using `oauth_client_credentials` now properly update their access token when editing the secrets. [#5548](https://github.com/ethyca/fides/pull/5548)
- Saas integrations using `oauth_client_credentials` now properly refresh their access token when the current token expires [#5569](https://github.com/ethyca/fides/pull/5569)
- Adding `dsr_testing_tools_enabled` security setting [#5573](https://github.com/ethyca/fides/pull/5573)

## [2.51.0](https://github.com/ethyca/fides/compare/2.50.0...2.51.0)

### Added
- Added new column for Action Type in privacy request event logs [#5546](https://github.com/ethyca/fides/pull/5546)
- Added `fides_consent_override` option in FidesJS SDK [#5541](https://github.com/ethyca/fides/pull/5541)
- Added new `script` ConsentMethod in FidesJS SDK for tracking automated consent [#5541](https://github.com/ethyca/fides/pull/5541)
- Added a new page under system integrations to run standalone dataset tests (Fidesplus) [#5549](https://github.com/ethyca/fides/pull/5549)

### Changed
- Adding hashes to system tab URLs [#5535](https://github.com/ethyca/fides/pull/5535)
- Boolean inputs will now show as a select with true/false values in the connection form [#5555](https://github.com/ethyca/fides/pull/5555)
- Updated Cookie House to be responsive [#5541](https://github.com/ethyca/fides/pull/5541)
- Updated `/system` endpoint to filter vendor deleted systems [#5553](https://github.com/ethyca/fides/pull/5553)

### Developer Experience
- Migrated remaining instances of Chakra's Select component to use Ant's Select component [#5502](https://github.com/ethyca/fides/pull/5502)

### Fixed
- Updating dataset PUT to allow deleting all datasets [#5524](https://github.com/ethyca/fides/pull/5524)
- Adds support for fides_key generation when parent_key is provided in Taxonomy create endpoints [#5542](https://github.com/ethyca/fides/pull/5542)
- An integration will no longer re-enable after saving the connection form [#5555](https://github.com/ethyca/fides/pull/5555)
- Fixed positioning of Fides brand link in privacy center [#5572](https://github.com/ethyca/fides/pull/5572)

### Removed
- Removed unnecessary debug logging from the load_file config helper [#5544](https://github.com/ethyca/fides/pull/5544)


## [2.50.0](https://github.com/ethyca/fides/compare/2.49.1...2.50.0)
Expand Down Expand Up @@ -58,9 +93,6 @@ The types of changes are:
- Fixed missing "Manage privacy preferences" button label option in TCF experience translations [#5528](https://github.com/ethyca/fides/pull/5528)
- Fixed privacy center not fetching the correct experience when using custom property paths [#5532](https://github.com/ethyca/fides/pull/5532)

### Docs
- Added docs for PrivacyNoticeRegion type [#5488](https://github.com/ethyca/fides/pull/5488)

### Security
- Password Policy is now Enforced in Accept Invite API [CVE-2024-52008](https://github.com/ethyca/fides/security/advisories/GHSA-v7vm-rhmg-8j2r)

Expand Down
2 changes: 0 additions & 2 deletions clients/README.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,3 @@
# Clients

The clients directory houses all front-end packages and shared code amongst clients, and also includes e2e tests.

See the [UI Contribution Guide](http://localhost:8000/fides/development/ui/overview/) for more information
39 changes: 38 additions & 1 deletion clients/admin-ui/cypress/e2e/system-integrations.cy.ts
Original file line number Diff line number Diff line change
@@ -1,12 +1,17 @@
import {
stubDatasetCrud,
stubDisabledIntegrationSystemCrud,
stubPlus,
stubPrivacyNoticesCrud,
stubSystemCrud,
stubSystemIntegrations,
stubTaxonomyEntities,
} from "cypress/support/stubs";

import { SYSTEM_ROUTE } from "~/features/common/nav/v2/routes";
import {
EDIT_SYSTEM_ROUTE,
SYSTEM_ROUTE,
} from "~/features/common/nav/v2/routes";

describe("System integrations", () => {
beforeEach(() => {
Expand Down Expand Up @@ -69,4 +74,36 @@ describe("System integrations", () => {
cy.getByTestId("enabled-actions").should("not.exist");
});
});

describe("Loading existing integration", () => {
beforeEach(() => {
cy.login();
stubPlus(false);
stubSystemIntegrations();
stubSystemCrud();
stubDatasetCrud();
stubTaxonomyEntities();
stubPrivacyNoticesCrud();
stubDisabledIntegrationSystemCrud();

cy.visit(EDIT_SYSTEM_ROUTE.replace("[id]", "disabled_postgres_system"));
cy.getByTestId("tab-Integrations").click();
});

it("when saving the form it shouldn't re-enable the integration", () => {
cy.get("form").within(() => {
cy.get("button[type=submit]").click();
});
cy.wait("@patchConnection").then(({ request }) => {
expect(request.body[0]).to.deep.equal({
access: "write",
connection_type: "postgres",
description: "",
key: "asdasd_postgres",
enabled_actions: [],
});
expect(request.body[0].disabled).to.be.undefined;
});
});
});
});
44 changes: 44 additions & 0 deletions clients/admin-ui/cypress/e2e/systems-plus.cy.ts
Original file line number Diff line number Diff line change
Expand Up @@ -123,7 +123,11 @@ describe("System management with Plus features", () => {
cy.wait("@getDictSystem");
cy.getByTestId("input-dpo").should("have.value", "[email protected]");
cy.getByTestId("tab-Data uses").click();
// eslint-disable-next-line cypress/no-unnecessary-waiting
cy.wait(500);
cy.getByTestId("tab-Information").click();
// eslint-disable-next-line cypress/no-unnecessary-waiting
cy.wait(500);
cy.getByTestId("tab-Data uses").click();
cy.getByTestId("confirmation-modal").should("not.exist");
});
Expand Down Expand Up @@ -460,4 +464,44 @@ describe("System management with Plus features", () => {
});
});
});

describe("tab navigation", () => {
it("updates URL hash when switching tabs", () => {
cy.visit(`${SYSTEM_ROUTE}/configure/demo_analytics_system#information`);
cy.location("hash").should("eq", "#information");

// eslint-disable-next-line cypress/no-unnecessary-waiting
cy.wait(500);
cy.getByTestId("tab-Data uses").click();
cy.location("hash").should("eq", "#data-uses");

// eslint-disable-next-line cypress/no-unnecessary-waiting
cy.wait(500);
cy.getByTestId("tab-Data flow").click();
cy.location("hash").should("eq", "#data-flow");

// eslint-disable-next-line cypress/no-unnecessary-waiting
cy.wait(500);
cy.getByTestId("tab-Integrations").click();
cy.location("hash").should("eq", "#integrations");

// eslint-disable-next-line cypress/no-unnecessary-waiting
cy.wait(500);
cy.getByTestId("tab-History").click();
cy.location("hash").should("eq", "#history");
});

it("loads correct tab directly based on URL hash", () => {
// Visit page with specific hash
cy.visit(`${SYSTEM_ROUTE}/configure/demo_analytics_system#data-uses`);

// Verify correct tab is active
cy.getByTestId("tab-Data uses").should(
"have.attr",
"aria-selected",
"true",
);
cy.location("hash").should("eq", "#data-uses");
});
});
});
Original file line number Diff line number Diff line change
@@ -0,0 +1,67 @@
{
"fides_key": "disabled_postgres_system",
"organization_fides_key": "default_organization",
"tags": [],
"name": "A system with adisabled Postgres integration",
"description": "",
"meta": null,
"fidesctl_meta": null,
"system_type": "",
"egress": null,
"ingress": null,
"privacy_declarations": [],
"administrating_department": "Not defined",
"vendor_id": null,
"previous_vendor_id": null,
"vendor_deleted_date": null,
"dataset_references": [],
"processes_personal_data": true,
"exempt_from_privacy_regulations": false,
"reason_for_exemption": null,
"uses_profiling": false,
"legal_basis_for_profiling": [],
"does_international_transfers": false,
"legal_basis_for_transfers": [],
"requires_data_protection_assessments": false,
"dpa_location": null,
"dpa_progress": null,
"privacy_policy": null,
"legal_name": "",
"legal_address": "",
"responsibility": [],
"dpo": "",
"joint_controller_info": null,
"data_security_practices": "",
"cookie_max_age_seconds": null,
"uses_cookies": false,
"cookie_refresh": false,
"uses_non_cookie_access": false,
"legitimate_interest_disclosure_url": null,
"cookies": [],
"created_at": "2024-12-03T15:21:25.496095Z",
"connection_configs": {
"name": "asdasd_postgres",
"key": "asdasd_postgres",
"description": "",
"connection_type": "postgres",
"access": "write",
"created_at": "2024-12-03T15:22:04.816975Z",
"updated_at": "2024-12-03T15:33:16.990741Z",
"disabled": true,
"last_test_timestamp": null,
"last_test_succeeded": null,
"saas_config": null,
"secrets": {
"host": "host.docker.internal",
"port": 6432,
"username": "asdasd",
"password": "**********",
"dbname": "postgres_example",
"db_schema": "",
"ssh_required": false
},
"authorized": false,
"enabled_actions": null
},
"data_stewards": []
}
29 changes: 29 additions & 0 deletions clients/admin-ui/cypress/support/stubs.ts
Original file line number Diff line number Diff line change
Expand Up @@ -240,6 +240,10 @@ export const stubPlus = (available: boolean, options?: HealthCheck) => {
statusCode: 400,
body: {},
}).as("getPlusHealth");
cy.intercept("GET", "/api/v1/plus/*", {
statusCode: 404,
body: {},
}).as("getNoPlusAvailable");
}
};

Expand Down Expand Up @@ -433,6 +437,31 @@ export const stubSystemIntegrations = () => {
).as("getConnections");
};

export const stubDisabledIntegrationSystemCrud = () => {
cy.intercept("GET", "/api/v1/system/disabled_postgres_system", {
fixture: "systems/system_disabled_integration.json",
}).as("getDisabledSystemIntegration");

cy.intercept("PATCH", "/api/v1/system/disabled_postgres_system/connection", {
statusCode: 200,
body: {},
}).as("patchConnection");

cy.intercept("PUT", "/api/v1/connection/asdasd_postgres/datasetconfig", {
statusCode: 200,
body: {},
}).as("putDatasetConfig");

cy.intercept(
"PATCH",
"/api/v1/system/disabled_postgres_system/connection/secrets*",
{
statusCode: 200,
body: {},
},
).as("patchConnectionSecret");
};

export const stubUserManagement = () => {
cy.intercept("/api/v1/user?*", {
fixture: "user-management/users.json",
Expand Down
2 changes: 2 additions & 0 deletions clients/admin-ui/src/app/store.ts
Original file line number Diff line number Diff line change
Expand Up @@ -41,6 +41,7 @@ import { propertySlice } from "~/features/properties";
import { systemSlice } from "~/features/system";
import { dictSuggestionsSlice } from "~/features/system/dictionary-form/dict-suggestion.slice";
import { taxonomySlice } from "~/features/taxonomy";
import { datasetTestSlice } from "~/features/test-datasets";
import { userManagementSlice } from "~/features/user-management";

/**
Expand Down Expand Up @@ -79,6 +80,7 @@ const reducer = {
[dataSubjectsSlice.name]: dataSubjectsSlice.reducer,
[dataUseSlice.name]: dataUseSlice.reducer,
[datasetSlice.name]: datasetSlice.reducer,
[datasetTestSlice.name]: datasetTestSlice.reducer,
[datastoreConnectionSlice.name]: datastoreConnectionSlice.reducer,
[discoveryDetectionSlice.name]: discoveryDetectionSlice.reducer,
[featuresSlice.name]: featuresSlice.reducer,
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -63,7 +63,7 @@ const ConnectorParametersForm = ({
isSubmitting = false,
onSaveClick,
onTestConnectionClick,
testButtonLabel = "Test connection",
testButtonLabel = "Test integration",
}: ConnectorParametersFormProps) => {
const mounted = useRef(false);
const { handleError } = useAPIHelper();
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -380,6 +380,40 @@ export const datastoreConnectionApi = baseApi.injectEndpoints({
}),
invalidatesTags: () => ["Datastore Connection"],
}),
testDatastoreConnectionDatasets: build.mutation<
{ privacy_request_id: string },
{
connection_key: string;
dataset_key: string;
input_data: Record<string, any>;
}
>({
query: (params) => ({
url: `${CONNECTION_ROUTE}/${params.connection_key}/dataset/${params.dataset_key}/test`,
method: "POST",
body: params.input_data,
}),
}),
getDatasetInputs: build.query<
any,
{ connectionKey: string; datasetKey: string }
>({
query: ({ connectionKey, datasetKey }) => ({
url: `${CONNECTION_ROUTE}/${connectionKey}/dataset/${datasetKey}/inputs`,
method: "GET",
}),
providesTags: () => ["Datastore Connection"],
}),
getDatasetReachability: build.query<
{ reachable: boolean; details: string },
{ connectionKey: string; datasetKey: string }
>({
query: ({ connectionKey, datasetKey }) => ({
url: `${CONNECTION_ROUTE}/${connectionKey}/dataset/${datasetKey}/reachability`,
method: "GET",
}),
providesTags: () => ["Datastore Connection"],
}),
}),
});

Expand All @@ -402,6 +436,9 @@ export const {
usePatchDatastoreConnectionsMutation,
useUpdateDatastoreConnectionSecretsMutation,
usePatchDatastoreConnectionSecretsMutation,
useTestDatastoreConnectionDatasetsMutation,
useGetDatasetInputsQuery,
useGetDatasetReachabilityQuery,
} = datastoreConnectionApi;

/**
Expand Down
Loading

0 comments on commit 5c87f32

Please sign in to comment.