Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Catalog: Replace S3 Select with GQL #4218

Merged
merged 34 commits into from
Nov 19, 2024
Merged

Catalog: Replace S3 Select with GQL #4218

merged 34 commits into from
Nov 19, 2024

Conversation

nl0
Copy link
Member

@nl0 nl0 commented Nov 15, 2024

Changes

  • Move bucket overview components into dedicated Overview/ directory
  • Convert bucket and object access counts to use GraphQL API instead of S3 Select
  • Add new GraphQL types and queries for access counts data
  • Remove unused S3 Select code from AWS S3 client
  • Update file analytics components to use new GraphQL queries
  • Clean up and modernize code with TypeScript

Checklist

  • Confirm that this change meets security best practices and does not violate the security model
  • revert deployment change
  • Changelog entry (skip if change is not significant to end users, e.g. docs only)

Copy link

codecov bot commented Nov 15, 2024

Codecov Report

Attention: Patch coverage is 10.58496% with 321 lines in your changes missing coverage. Please review.

Project coverage is 39.22%. Comparing base (e4d9590) to head (09eb8bb).
Report is 6 commits behind head on master.

Files with missing lines Patch % Lines
...talog/app/containers/Bucket/Overview/Downloads.tsx 25.37% 93 Missing and 7 partials ⚠️
catalog/app/containers/Bucket/Overview/Header.tsx 0.00% 71 Missing ⚠️
...atalog/app/containers/Bucket/Overview/Overview.tsx 0.00% 62 Missing and 1 partial ⚠️
catalog/app/containers/Bucket/File/Analytics.tsx 0.00% 35 Missing and 2 partials ⚠️
catalog/app/utils/AWS/S3.js 0.00% 9 Missing and 3 partials ⚠️
catalog/app/containers/Bucket/File/File.js 0.00% 11 Missing ⚠️
...atalog/app/containers/Bucket/Overview/ColorPool.ts 0.00% 7 Missing and 1 partial ⚠️
.../app/containers/Bucket/requests/requestsUntyped.js 14.28% 6 Missing ⚠️
...rs/Bucket/File/gql/ObjectAccessCounts.generated.ts 0.00% 2 Missing ⚠️
...ners/Bucket/Overview/gql/BucketConfig.generated.ts 0.00% 2 Missing ⚠️
... and 7 more
Additional details and impacted files
@@            Coverage Diff             @@
##           master    #4218      +/-   ##
==========================================
+ Coverage   39.02%   39.22%   +0.19%     
==========================================
  Files         772      780       +8     
  Lines       35540    35493      -47     
  Branches     5111     5293     +182     
==========================================
+ Hits        13870    13922      +52     
+ Misses      21063    20375     -688     
- Partials      607     1196     +589     
Flag Coverage Δ
api-python 91.09% <ø> (ø)
catalog 14.54% <10.58%> (+0.24%) ⬆️
lambda 87.95% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.


🚨 Try these New Features:

@nl0 nl0 changed the title Replace S3 Select with GQL Catalog: Replace S3 Select with GQL Nov 15, 2024
@nl0 nl0 requested a review from fiskus November 15, 2024 12:25
@nl0
Copy link
Member Author

nl0 commented Nov 15, 2024

@fiskus this is deployed on the no-select dev stack in case you'd like to test it manually

fiskus
fiskus previously approved these changes Nov 15, 2024
Copy link
Member

@fiskus fiskus left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good, and I'm glad gql and typescript coverage is increasing.

I still have some concerns over using Effect. Sometimes it seems that code with Effect is more complicated than without. I experimented with it a little, and will show you what I found later.

catalog/app/embed/File.js Outdated Show resolved Hide resolved
catalog/CHANGELOG.md Show resolved Hide resolved
catalog/app/containers/Bucket/Overview/Downloads.tsx Outdated Show resolved Hide resolved
catalog/app/containers/Bucket/Overview/Downloads.tsx Outdated Show resolved Hide resolved
@nl0
Copy link
Member Author

nl0 commented Nov 18, 2024

I still have some concerns over using Effect. Sometimes it seems that code with Effect is more complicated than without.

for trivial cases -- maybe.
the most compelling thing about effect is that it is a system of primitives/abstractions and operations over/between them that cover a lot of use cases, from simple to complex, not just a collection of disparate "utilities".
so when i start doing everything with it, things start looking more similar, and, with time, familiar.
for trivial cases that may look like unnecessary complexity, but it actually helps see (and highlight) the patterns, and makes the code easier to modify, so i think it's a win in the long term.

I experimented with it a little, and will show you what I found later.

i'm curious to see your findings

This reverts commit 7b57360.
@nl0 nl0 requested a review from fiskus November 18, 2024 12:10
@nl0 nl0 marked this pull request as ready for review November 18, 2024 12:14
@nl0 nl0 requested a review from Copilot November 18, 2024 12:17
Copy link
Contributor

@Copilot Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot reviewed 14 out of 27 changed files in this pull request and generated 1 suggestion.

Files not reviewed (13)
  • catalog/app/containers/Bucket/File/gql/ObjectAccessCounts.graphql: Language not supported
  • catalog/app/containers/Bucket/Overview/gql/BucketAccessCounts.graphql: Language not supported
  • catalog/app/embed/File.js: Evaluated as low risk
  • catalog/app/containers/Bucket/File/AssistantContext.ts: Evaluated as low risk
  • catalog/app/containers/Bucket/File/File.js: Evaluated as low risk
  • catalog/CHANGELOG.md: Evaluated as low risk
  • catalog/app/containers/Bucket/Overview/gql/BucketConfig.generated.ts: Evaluated as low risk
  • catalog/app/containers/Bucket/Summarize.tsx: Evaluated as low risk
  • catalog/app/containers/Bucket/requests/requestsUntyped.js: Evaluated as low risk
  • catalog/app/model/graphql/types.generated.ts: Evaluated as low risk
  • catalog/app/model/graphql/schema.generated.ts: Evaluated as low risk
  • catalog/app/containers/Bucket/File/index.ts: Evaluated as low risk
  • catalog/app/containers/Bucket/Overview/index.tsx: Evaluated as low risk

@nl0 nl0 added this pull request to the merge queue Nov 19, 2024
Merged via the queue into master with commit 8dd6815 Nov 19, 2024
39 of 41 checks passed
@nl0 nl0 deleted the replace-select branch November 19, 2024 12:34
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants