Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

wip: feat: collection service & controller #3971

Draft
wants to merge 46 commits into
base: main
Choose a base branch
from
Draft
Show file tree
Hide file tree
Changes from 15 commits
Commits
Show all changes
46 commits
Select commit Hold shift + click to select a range
9f469cb
feat: create type and validator
zz-hh-aa Nov 18, 2024
164798f
chore: change 'council' to 'team'
zz-hh-aa Nov 18, 2024
dc22f0b
feat: setup collection service
zz-hh-aa Nov 18, 2024
a957c1a
feat: create new collection type
zz-hh-aa Nov 18, 2024
9739d6b
feat: create collection controller
zz-hh-aa Nov 18, 2024
16a02b3
chore: change id type to number
zz-hh-aa Nov 18, 2024
165eca6
feat: use new MetabaseError type
zz-hh-aa Nov 18, 2024
a4d6b5f
chore: tidy error handling in controller
zz-hh-aa Nov 18, 2024
62836ba
chore: move collection types to types.ts
zz-hh-aa Nov 18, 2024
73ba1e9
feat: remove unnecessary checkCollection controller
zz-hh-aa Dec 2, 2024
22d3ee4
chore: simplify error handling to just throw
zz-hh-aa Dec 2, 2024
e19ce99
nit: correctly rename type as handler
zz-hh-aa Dec 2, 2024
bc13342
chore: tidy data types and undefined values
zz-hh-aa Dec 2, 2024
97ee946
chore: add snake_case and camelCase conversions
zz-hh-aa Dec 2, 2024
1a4ca85
chore: re-import createMetabaseClient after rebase
zz-hh-aa Dec 2, 2024
6ac50e7
feat: remove manual toSnakeCase function
zz-hh-aa Dec 3, 2024
1d8dfb4
feat: add metabase collection route
zz-hh-aa Dec 3, 2024
ae037aa
feat: remove unnecessary authentication and transformation
zz-hh-aa Dec 3, 2024
9d6878c
feat: transform request in newCollectionSchema
zz-hh-aa Dec 3, 2024
cea1b05
feat: add metabase_id column to teams table migration
zz-hh-aa Dec 9, 2024
df3345e
chore: add assert import
zz-hh-aa Dec 9, 2024
628c151
wip: store metabase id to planx db
zz-hh-aa Dec 9, 2024
3974b2c
feat: update Hasura api permissions
zz-hh-aa Dec 10, 2024
1d885e0
test: write tests for collections service
zz-hh-aa Dec 2, 2024
62ca295
chore: update user role to api
zz-hh-aa Dec 10, 2024
052b0f3
feat: change string matching to exact
zz-hh-aa Dec 10, 2024
bb3c642
test: add tsdoc and getCollection function for tests
zz-hh-aa Dec 2, 2024
d88d118
chore: rename updateMetabaseId file
zz-hh-aa Dec 10, 2024
0b6153c
test: add new test for updateMetabaseId
zz-hh-aa Dec 10, 2024
b76bde1
chore: throw error in function
zz-hh-aa Dec 10, 2024
0b33f0b
chore: update import after updateMetabaseId name change
zz-hh-aa Dec 10, 2024
7ab0398
chore: remove console.log
zz-hh-aa Dec 10, 2024
110ce9c
chore: remove old checkCollection
zz-hh-aa Dec 10, 2024
5ec8b7a
feat: getTeamAndMetabaseId function
zz-hh-aa Dec 10, 2024
f2c5ba6
feat: separate newCollection into own function
zz-hh-aa Dec 10, 2024
a6f005d
test: test for collection services
zz-hh-aa Dec 10, 2024
e234edc
chore: tidy function names
zz-hh-aa Dec 10, 2024
c384fee
chore: move createCollection to own file
zz-hh-aa Dec 10, 2024
8378601
chore: tidy comments and console.logs
zz-hh-aa Dec 11, 2024
1aa5f1c
feat: move getCollection to own file
zz-hh-aa Dec 11, 2024
08f2402
feat: type getCollection response
zz-hh-aa Dec 11, 2024
26f9ec2
chore: type checkCollections response
zz-hh-aa Dec 11, 2024
ee3a730
chore: rename to getTeamIdAndMetabaseId for clarity
zz-hh-aa Dec 11, 2024
ecca6d6
chore: tidy destructuring
zz-hh-aa Dec 11, 2024
7203db0
chore: rename to createCollectionIfDoesNotExist for clarity
zz-hh-aa Dec 11, 2024
5248d51
chore: rename to MetabaseCollectionsController for clarity
zz-hh-aa Dec 11, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
18 changes: 18 additions & 0 deletions api.planx.uk/modules/analytics/metabase/collection/controller.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,18 @@
import { newCollection } from "./service.js";
import type { NewCollectionRequestHandler } from "./types.js";

export const newCollectionController: NewCollectionRequestHandler = async (
_req,
res,
) => {
try {
const params = res.locals.parsedReq.body;
const collection = await newCollection(params);
res.status(201).json({ data: collection });
} catch (error) {
res.status(400).json({
error:
error instanceof Error ? error.message : "An unexpected error occurred",
});
}
};
90 changes: 90 additions & 0 deletions api.planx.uk/modules/analytics/metabase/collection/service.ts
Original file line number Diff line number Diff line change
@@ -1,0 +1,90 @@
import { MetabaseError, createMetabaseClient } from "../shared/client.js";
import type { NewCollectionParams } from "./types.js";
import { toSnakeCase } from "../shared/utils.js";

const client = createMetabaseClient();

export async function authentication(): Promise<boolean> {
try {
const response = await client.get("/user/current");
return response.status === 200;
} catch (error) {
console.error("Error testing Metabase connection:", error);
return false;
}
}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This also looks unused currently.

I think this should be a middleware added to the route - before proceeding to make additional requests to Metabase, we can first check the API key is valid.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I had a quick chat with Jo, who pointed out that it might not be necessary as the authentication function was basically checking that we could get before actually get-ing (which happens in service.ts, in checkCollections():

const response = await client.get(`/api/collection/`);

Between validateConfig() in client.ts and running the get above, does that mean authentication is redundant? Or do we still need another check for the valid API key? I believe the first get would just throw a MetabaseError if it was invalid, is that sufficient?


export async function newCollection({
name,
description,
parent_id,
}: NewCollectionParams): Promise<any> {

Check warning on line 21 in api.planx.uk/modules/analytics/metabase/collection/service.ts

View workflow job for this annotation

GitHub Actions / Run API Tests

Unexpected any. Specify a different type
try {
// Check if collection exists
const existingCollectionId = await checkCollections(name);
if (existingCollectionId) {
console.log(
`Collection "${name}" already exists with ID: ${existingCollectionId}`,
);
return existingCollectionId;
}

// If no existing collection, create new one
const requestBody = toSnakeCase({
name,
description,
parent_id,
});
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Rather than casting to snake_case, the API should accept snake_case, and then we can cast or transform as necessary for the Metabase request.

Incoming request (camelCase) → Validate → Transform to snake_case → Make request to Metabase

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ended up using transform instead of importing the Lodash method.


// Remove undefined properties
Object.keys(requestBody).forEach(
(key) => requestBody[key] === undefined && delete requestBody[key],
);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Passing through the validate() middleware should handle this step already - no additional keys would be present.


const response = await client.post(`/api/collection/`, {
name,
description,
parent_id,
});
console.log(
`New collection: ${response.data.name}, new collection ID: ${response.data.id}`,
);
return response.data.id;
} catch (error) {
console.error("Error in newCollection:");
throw error;
}
}

/**
* Checks if a collection exists with name matching `teamName` exists.
* Returns the matching collection ID if exists, otherwise false. */
export async function checkCollections(teamName: string): Promise<any> {

Check warning on line 62 in api.planx.uk/modules/analytics/metabase/collection/service.ts

View workflow job for this annotation

GitHub Actions / Run API Tests

Unexpected any. Specify a different type
try {
console.log("Checking for collection: ", teamName);

// Get collections from Metabase
const response = await client.get(`/api/collection/`);

const matchingCollection = response.data.find((collection: any) =>

Check warning on line 69 in api.planx.uk/modules/analytics/metabase/collection/service.ts

View workflow job for this annotation

GitHub Actions / Run API Tests

Unexpected any. Specify a different type
collection.name.toLowerCase().includes(teamName.toLowerCase()),
);

if (matchingCollection) {
console.log("Matching collection found with ID: ", matchingCollection.id);
return matchingCollection.id;
} else {
console.log("No matching collection found");
return undefined;
}
} catch (error) {
console.error("Error: ", error);
if (error instanceof MetabaseError) {
console.error("Metabase API error:", {
message: error.message,
statusCode: error.statusCode,
});
}
throw error;
}
}
37 changes: 37 additions & 0 deletions api.planx.uk/modules/analytics/metabase/collection/types.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,37 @@
import { z } from "zod";
import type { ValidatedRequestHandler } from "../../../../shared/middleware/validate.js";

type ApiResponse<T> = {
data?: T;
error?: string;
};

export interface NewCollectionParams {
jamdelion marked this conversation as resolved.
Show resolved Hide resolved
name: string;
description?: string;
/** Optional; if the collection is a child of a parent, specify parent ID here
* For council teams, parent collection should be 58
*/
parent_id?: number;
jamdelion marked this conversation as resolved.
Show resolved Hide resolved
}

/** Metbase collection ID for the the "Council" collection **/
const COUNCIL_COLLECTION_ID = 58;

export const newCollectionSchema = z.object({
body: z.object({
name: z.string(),
description: z.string().optional(),
parent_id: z.number().default(COUNCIL_COLLECTION_ID),
}),
});

export type NewCollectionRequestHandler = ValidatedRequestHandler<
typeof newCollectionSchema,
ApiResponse<NewCollectionResponse>
>;

export interface NewCollectionResponse {
id: number;
name: string;
}
40 changes: 40 additions & 0 deletions api.planx.uk/modules/analytics/metabase/shared/types.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,40 @@
export interface Input {
teamName: string;
originalDashboardId: number;
/** The name of the filter to be updated */
filter: string;
/** The value of the new filter; updateFilter() only supports strings right now */
filterValue: string;
}

export function validateInput(input: unknown): input is Input {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This looks like it's currently unused - is that right?

In terms of validation, it would be best to follow the existing pattern in the API.

→ Incoming request of type unknown
→ Passes through existing validate() middleware in route →
→ Using Zod, is checked against a provided schema →
→ Validated payload is automatically passed along as res.locals.parsedReq
→ Controller and Service have both type safety and a validated payload

// check that input type is object
if (typeof input !== "object" || input === null) {
return false;
}

// check that input object is same shape as Input type with same properties
const { teamName, originalDashboardId, filter, filterValue } = input as Input;

if (typeof teamName !== "string" || teamName.trim() === "") {
console.error("Invalid teamName: must be a non-empty string");
return false;
}

if (!Number.isInteger(originalDashboardId) || originalDashboardId <= 0) {
console.error("Invalid originalDashboardId: must be a positive integer");
return false;
}

if (typeof filter !== "string" || filter.trim() === "") {
console.error("Invalid filter: must be a non-empty string");
return false;
}

if (typeof filterValue !== "string") {
console.error("Invalid filterValue: must be a string");
return false;
}
console.log("Input valid");
return true;
}
34 changes: 34 additions & 0 deletions api.planx.uk/modules/analytics/metabase/shared/utils.ts
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For functions like this, I'd suggest importing from lodash (which we already use in the API) instead of rolling our own. It means we're using well tested and efficient utility functions, and there's not much overhead in terms of package size.

Original file line number Diff line number Diff line change
@@ -0,0 +1,34 @@
/**
* Converts object keys from camelCase to snake_case
*/
export function toSnakeCase<T extends object>(obj: T): Record<string, unknown> {
return Object.entries(obj).reduce(
(acc, [key, value]) => {
// Convert camelCase to snake_case
const snakeKey = key.replace(
/[A-Z]/g,
(letter) => `_${letter.toLowerCase()}`,
);
acc[snakeKey] = value;
return acc;
},
{} as Record<string, unknown>,
);
}

/**
* Converts object keys from snake_case to camelCase
*/
export function toCamelCase<T extends object>(obj: T): Record<string, unknown> {
return Object.entries(obj).reduce(
(acc, [key, value]) => {
// Convert snake_case to camelCase
const camelKey = key.replace(/_([a-z])/g, (_, letter) =>
letter.toUpperCase(),
);
acc[camelKey] = value;
return acc;
},
{} as Record<string, unknown>,
);
}
Loading