Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Catalog: Replace S3 Select with GQL #4218

Merged
merged 34 commits into from
Nov 19, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
34 commits
Select commit Hold shift + click to select a range
773e91e
gql schema: add access counts queries for buckets and objects
nl0 Nov 8, 2024
ca318e8
overview: js -> tsx
nl0 Nov 8, 2024
a2a7f93
Overview: TSify
nl0 Nov 8, 2024
5f07f8f
refactor: update GraphQL import to use namespace import in Overview
nl0 Nov 9, 2024
367569d
move stuff around
nl0 Nov 9, 2024
ab06706
feat: add GraphQL query for bucket access counts
nl0 Nov 9, 2024
a524b0d
refactor: migrate bucket downloads chart to use GraphQL data
nl0 Nov 11, 2024
e48e241
adjust gql schema
nl0 Nov 12, 2024
7b2243a
gql: keys for newly added types
nl0 Nov 12, 2024
1bf1f4f
gql: effect interop
nl0 Nov 13, 2024
81b3f25
move Header out
nl0 Nov 13, 2024
cdd1ea4
move ColorPool out
nl0 Nov 13, 2024
0b54c8d
move Downloads out
nl0 Nov 13, 2024
e1c1d0e
Header: ramda -> effect
nl0 Nov 13, 2024
4481a56
Downloads: complete the migration / refactoring
nl0 Nov 13, 2024
b0b5f96
requestsUntyped: rm bucketAccessCounts
nl0 Nov 14, 2024
2d95c42
progress when changing window
nl0 Nov 14, 2024
012e2a8
move FileAnalytics out
nl0 Nov 14, 2024
cb80994
move Bucket/File* into a folder
nl0 Nov 15, 2024
7f934fb
migrate file analytics to GraphQL and Effect
nl0 Nov 15, 2024
664342a
requestsUntyped: rm obsolete code
nl0 Nov 15, 2024
165900f
refactor: migrate analytics to GraphQL and Effect
nl0 Nov 15, 2024
bc77833
refactor: remove unused getDataOption function and Effect import
nl0 Nov 15, 2024
6474084
refactor: replace fp-ts with effect library for pipe operations
nl0 Nov 15, 2024
c5b050b
refactor: simplify S3 request signing logic and remove analytics bucket
nl0 Nov 15, 2024
7b57360
deploy
nl0 Nov 15, 2024
88c4949
Merge branch 'master' into replace-select
nl0 Nov 15, 2024
f59ff75
cl
nl0 Nov 15, 2024
3e45dd5
embed/File: reuse Analytics component
nl0 Nov 18, 2024
623fb74
Effect.Array.map -> array.map
nl0 Nov 18, 2024
3fbcee7
refactor: simplify FilePreview component and fix type errors
nl0 Nov 18, 2024
68581ff
Revert "deploy"
nl0 Nov 18, 2024
a4a2afc
Merge branch 'master' into replace-select
nl0 Nov 18, 2024
09eb8bb
test: add unit tests for Downloads data processing
nl0 Nov 18, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions catalog/CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -17,6 +17,7 @@ where verb is one of

## Changes

- [Changed] S3 Select -> GQL API calls for getting access counts ([#4218](https://github.com/quiltdata/quilt/pull/4218))
nl0 marked this conversation as resolved.
Show resolved Hide resolved
- [Changed] Athena: improve loading state and errors visuals; fix minor bugs; alphabetize and persist selection in workgroups, catalog names and databases ([#4208](https://github.com/quiltdata/quilt/pull/4208))
- [Changed] Show stack release version in footer ([#4200](https://github.com/quiltdata/quilt/pull/4200))
- [Added] Selective package downloading ([#4173](https://github.com/quiltdata/quilt/pull/4173))
Expand Down
92 changes: 92 additions & 0 deletions catalog/app/containers/Bucket/File/Analytics.tsx
Original file line number Diff line number Diff line change
@@ -0,0 +1,92 @@
import * as dateFns from 'date-fns'
import * as Eff from 'effect'
import * as React from 'react'
import * as M from '@material-ui/core'

Check warning on line 4 in catalog/app/containers/Bucket/File/Analytics.tsx

View check run for this annotation

Codecov / codecov/patch/informational

catalog/app/containers/Bucket/File/Analytics.tsx#L1-L4

Added lines #L1 - L4 were not covered by tests

import Sparkline from 'components/Sparkline'
import * as GQL from 'utils/GraphQL'
import log from 'utils/Logging'
import * as SVG from 'utils/SVG'
import { readableQuantity } from 'utils/string'

Check warning on line 10 in catalog/app/containers/Bucket/File/Analytics.tsx

View check run for this annotation

Codecov / codecov/patch/informational

catalog/app/containers/Bucket/File/Analytics.tsx#L6-L10

Added lines #L6 - L10 were not covered by tests

import Section from '../Section'

Check warning on line 12 in catalog/app/containers/Bucket/File/Analytics.tsx

View check run for this annotation

Codecov / codecov/patch/informational

catalog/app/containers/Bucket/File/Analytics.tsx#L12

Added line #L12 was not covered by tests

import ACCESS_COUNTS_QUERY from './gql/ObjectAccessCounts.generated'

Check warning on line 14 in catalog/app/containers/Bucket/File/Analytics.tsx

View check run for this annotation

Codecov / codecov/patch/informational

catalog/app/containers/Bucket/File/Analytics.tsx#L14

Added line #L14 was not covered by tests

const currentYear = new Date().getFullYear()

Check warning on line 16 in catalog/app/containers/Bucket/File/Analytics.tsx

View check run for this annotation

Codecov / codecov/patch/informational

catalog/app/containers/Bucket/File/Analytics.tsx#L16

Added line #L16 was not covered by tests

const formatDate = (date: Date) =>

Check warning on line 18 in catalog/app/containers/Bucket/File/Analytics.tsx

View check run for this annotation

Codecov / codecov/patch/informational

catalog/app/containers/Bucket/File/Analytics.tsx#L18

Added line #L18 was not covered by tests
dateFns.format(date, currentYear === date.getFullYear() ? 'd MMM' : 'd MMM yyyy')

interface AnalyticsProps {
bucket: string
path: string
}

export default function Analytics({ bucket, path }: AnalyticsProps) {
const [cursor, setCursor] = React.useState<number | null>(null)

Check warning on line 27 in catalog/app/containers/Bucket/File/Analytics.tsx

View check run for this annotation

Codecov / codecov/patch/informational

catalog/app/containers/Bucket/File/Analytics.tsx#L26-L27

Added lines #L26 - L27 were not covered by tests

const result = GQL.useQuery(ACCESS_COUNTS_QUERY, { bucket, key: path })

Check warning on line 29 in catalog/app/containers/Bucket/File/Analytics.tsx

View check run for this annotation

Codecov / codecov/patch/informational

catalog/app/containers/Bucket/File/Analytics.tsx#L29

Added line #L29 was not covered by tests

const data = React.useMemo(() => {

Check warning on line 31 in catalog/app/containers/Bucket/File/Analytics.tsx

View check run for this annotation

Codecov / codecov/patch/informational

catalog/app/containers/Bucket/File/Analytics.tsx#L31

Added line #L31 was not covered by tests
if (result.fetching) return Eff.Option.none()
if (result.error) log.error('Error fetching object access counts:', result.error)
return Eff.Option.some(Eff.Option.fromNullable(result.data?.objectAccessCounts))
}, [result.fetching, result.error, result.data])

const defaultExpanded = Eff.Option.match(data, {
onNone: () => false,

Check warning on line 38 in catalog/app/containers/Bucket/File/Analytics.tsx

View check run for this annotation

Codecov / codecov/patch/informational

catalog/app/containers/Bucket/File/Analytics.tsx#L37-L38

Added lines #L37 - L38 were not covered by tests
onSome: Eff.Option.match({
onNone: () => false,
onSome: ({ total }) => !!total,

Check warning on line 41 in catalog/app/containers/Bucket/File/Analytics.tsx

View check run for this annotation

Codecov / codecov/patch/informational

catalog/app/containers/Bucket/File/Analytics.tsx#L40-L41

Added lines #L40 - L41 were not covered by tests
}),
})

return (

Check warning on line 45 in catalog/app/containers/Bucket/File/Analytics.tsx

View check run for this annotation

Codecov / codecov/patch/informational

catalog/app/containers/Bucket/File/Analytics.tsx#L45

Added line #L45 was not covered by tests
<Section icon="bar_charts" heading="Analytics" defaultExpanded={defaultExpanded}>
{Eff.Option.match(data, {
onNone: () => <M.CircularProgress />,

Check warning on line 48 in catalog/app/containers/Bucket/File/Analytics.tsx

View check run for this annotation

Codecov / codecov/patch/informational

catalog/app/containers/Bucket/File/Analytics.tsx#L48

Added line #L48 was not covered by tests
onSome: Eff.Option.match({
onNone: () => <M.Typography>No analytics available</M.Typography>,
onSome: ({ counts, total }) =>

Check warning on line 51 in catalog/app/containers/Bucket/File/Analytics.tsx

View check run for this annotation

Codecov / codecov/patch/informational

catalog/app/containers/Bucket/File/Analytics.tsx#L50-L51

Added lines #L50 - L51 were not covered by tests
total ? (
<M.Box

Check warning on line 53 in catalog/app/containers/Bucket/File/Analytics.tsx

View check run for this annotation

Codecov / codecov/patch/informational

catalog/app/containers/Bucket/File/Analytics.tsx#L53

Added line #L53 was not covered by tests
display="flex"
width="100%"
justifyContent="space-between"
alignItems="center"
>
<M.Box>
<M.Typography variant="h5">Downloads</M.Typography>
<M.Typography variant="h4" component="div">
{readableQuantity(cursor === null ? total : counts[cursor].value)}
</M.Typography>
<M.Typography variant="overline" component="span">
{cursor === null
? `${counts.length} days`
: formatDate(counts[cursor].date)}

Check warning on line 67 in catalog/app/containers/Bucket/File/Analytics.tsx

View check run for this annotation

Codecov / codecov/patch/informational

catalog/app/containers/Bucket/File/Analytics.tsx#L66-L67

Added lines #L66 - L67 were not covered by tests
</M.Typography>
</M.Box>
<M.Box width="calc(100% - 7rem)">
<Sparkline
data={counts.map((c) => c.value)}

Check warning on line 72 in catalog/app/containers/Bucket/File/Analytics.tsx

View check run for this annotation

Codecov / codecov/patch/informational

catalog/app/containers/Bucket/File/Analytics.tsx#L72

Added line #L72 was not covered by tests
onCursor={setCursor}
width={1000}
height={60}
stroke={SVG.Paint.Server(
<linearGradient x2="0" y2="100%" gradientUnits="userSpaceOnUse">
<stop offset="0" stopColor={M.colors.blueGrey[800]} />
<stop offset="100%" stopColor={M.colors.blueGrey[100]} />
</linearGradient>,
)}
/>
</M.Box>
</M.Box>
) : (
<M.Typography>No analytics available</M.Typography>

Check warning on line 86 in catalog/app/containers/Bucket/File/Analytics.tsx

View check run for this annotation

Codecov / codecov/patch/informational

catalog/app/containers/Bucket/File/Analytics.tsx#L86

Added line #L86 was not covered by tests
),
}),
})}
</Section>
)
}
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@
import * as Assistant from 'components/Assistant'
import * as XML from 'utils/XML'

import { ObjectExistence } from './requests'
import { ObjectExistence } from '../requests'

Check warning on line 7 in catalog/app/containers/Bucket/File/AssistantContext.ts

View check run for this annotation

Codecov / codecov/patch/informational

catalog/app/containers/Bucket/File/AssistantContext.ts#L7

Added line #L7 was not covered by tests

interface VersionsContextProps {
data: $TSFixMe
Expand Down
Original file line number Diff line number Diff line change
@@ -1,6 +1,5 @@
import { basename } from 'path'

import * as dateFns from 'date-fns'
import * as R from 'ramda'
import * as React from 'react'
import { Link, useHistory, useLocation, useParams } from 'react-router-dom'
Expand All @@ -11,7 +10,6 @@
import * as FileEditor from 'components/FileEditor'
import Message from 'components/Message'
import * as Preview from 'components/Preview'
import Sparkline from 'components/Sparkline'
import cfg from 'constants/config'
import * as Bookmarks from 'containers/Bookmarks'
import * as Notifications from 'containers/Notifications'
Expand All @@ -21,23 +19,24 @@
import { useData } from 'utils/Data'
import MetaTitle from 'utils/MetaTitle'
import * as NamedRoutes from 'utils/NamedRoutes'
import * as SVG from 'utils/SVG'
import { linkStyle } from 'utils/StyledLink'
import copyToClipboard from 'utils/clipboard'
import * as Format from 'utils/format'
import parseSearch from 'utils/parseSearch'
import { up, decode, handleToHttpsUri } from 'utils/s3paths'
import { readableBytes, readableQuantity } from 'utils/string'

import AssistButton from './AssistButton'
import FileCodeSamples from './CodeSamples/File'
import * as AssistantContext from './FileAssistantContext'
import FileProperties from './FileProperties'
import * as FileView from './FileView'
import Section from './Section'
import renderPreview from './renderPreview'
import * as requests from './requests'
import { useViewModes, viewModeToSelectOption } from './viewModes'
import { readableBytes } from 'utils/string'

Check warning on line 27 in catalog/app/containers/Bucket/File/File.js

View check run for this annotation

Codecov / codecov/patch/informational

catalog/app/containers/Bucket/File/File.js#L27

Added line #L27 was not covered by tests

import AssistButton from '../AssistButton'
import FileCodeSamples from '../CodeSamples/File'
import FileProperties from '../FileProperties'
import * as FileView from '../FileView'
import Section from '../Section'
import renderPreview from '../renderPreview'
import * as requests from '../requests'
import { useViewModes, viewModeToSelectOption } from '../viewModes'

Check warning on line 36 in catalog/app/containers/Bucket/File/File.js

View check run for this annotation

Codecov / codecov/patch/informational

catalog/app/containers/Bucket/File/File.js#L29-L36

Added lines #L29 - L36 were not covered by tests

import Analytics from './Analytics'
import * as AssistantContext from './AssistantContext'

Check warning on line 39 in catalog/app/containers/Bucket/File/File.js

View check run for this annotation

Codecov / codecov/patch/informational

catalog/app/containers/Bucket/File/File.js#L38-L39

Added lines #L38 - L39 were not covered by tests

const useVersionInfoStyles = M.makeStyles(({ typography }) => ({
version: {
Expand Down Expand Up @@ -203,69 +202,6 @@
)
}

function Analytics({ bucket, path }) {
const [cursor, setCursor] = React.useState(null)
const s3 = AWS.S3.use()
const today = React.useMemo(() => new Date(), [])
const formatDate = (date) =>
dateFns.format(
date,
today.getFullYear() === date.getFullYear() ? 'd MMM' : 'd MMM yyyy',
)
const data = useData(requests.objectAccessCounts, { s3, bucket, path, today })

const defaultExpanded = data.case({
Ok: ({ total }) => !!total,
_: () => false,
})

return (
<Section icon="bar_charts" heading="Analytics" defaultExpanded={defaultExpanded}>
{data.case({
Ok: ({ counts, total }) =>
total ? (
<M.Box
display="flex"
width="100%"
justifyContent="space-between"
alignItems="center"
>
<M.Box>
<M.Typography variant="h5">Downloads</M.Typography>
<M.Typography variant="h4" component="div">
{readableQuantity(cursor === null ? total : counts[cursor].value)}
</M.Typography>
<M.Typography variant="overline" component="span">
{cursor === null
? `${counts.length} days`
: formatDate(counts[cursor].date)}
</M.Typography>
</M.Box>
<M.Box width="calc(100% - 7rem)">
<Sparkline
data={R.pluck('value', counts)}
onCursor={setCursor}
width={1000}
height={60}
stroke={SVG.Paint.Server(
<linearGradient x2="0" y2="100%" gradientUnits="userSpaceOnUse">
<stop offset="0" stopColor={M.colors.blueGrey[800]} />
<stop offset="100%" stopColor={M.colors.blueGrey[100]} />
</linearGradient>,
)}
/>
</M.Box>
</M.Box>
) : (
<M.Typography>No analytics available</M.Typography>
),
Err: () => <M.Typography>No analytics available</M.Typography>,
_: () => <M.CircularProgress />,
})}
</Section>
)
}

function CenteredProgress() {
return (
<M.Box textAlign="center" width="100%">
Expand Down
100 changes: 100 additions & 0 deletions catalog/app/containers/Bucket/File/gql/ObjectAccessCounts.generated.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,100 @@
/* eslint-disable @typescript-eslint/naming-convention */
import type { TypedDocumentNode as DocumentNode } from '@graphql-typed-document-node/core'
import * as Types from '../../../../model/graphql/types.generated'

export type containers_Bucket_File_gql_ObjectAccessCountsQueryVariables = Types.Exact<{
bucket: Types.Scalars['String']
key: Types.Scalars['String']
}>

export type containers_Bucket_File_gql_ObjectAccessCountsQuery = {
readonly __typename: 'Query'
} & {
readonly objectAccessCounts: Types.Maybe<
{ readonly __typename: 'AccessCounts' } & Pick<Types.AccessCounts, 'total'> & {
readonly counts: ReadonlyArray<
{ readonly __typename: 'AccessCountForDate' } & Pick<
Types.AccessCountForDate,
'date' | 'value'
>
>
}
>
}

export const containers_Bucket_File_gql_ObjectAccessCountsDocument = {

Check warning on line 25 in catalog/app/containers/Bucket/File/gql/ObjectAccessCounts.generated.ts

View check run for this annotation

Codecov / codecov/patch/informational

catalog/app/containers/Bucket/File/gql/ObjectAccessCounts.generated.ts#L25

Added line #L25 was not covered by tests
kind: 'Document',
definitions: [
{
kind: 'OperationDefinition',
operation: 'query',
name: { kind: 'Name', value: 'containers_Bucket_File_gql_ObjectAccessCounts' },
variableDefinitions: [
{
kind: 'VariableDefinition',
variable: { kind: 'Variable', name: { kind: 'Name', value: 'bucket' } },
type: {
kind: 'NonNullType',
type: { kind: 'NamedType', name: { kind: 'Name', value: 'String' } },
},
},
{
kind: 'VariableDefinition',
variable: { kind: 'Variable', name: { kind: 'Name', value: 'key' } },
type: {
kind: 'NonNullType',
type: { kind: 'NamedType', name: { kind: 'Name', value: 'String' } },
},
},
],
selectionSet: {
kind: 'SelectionSet',
selections: [
{
kind: 'Field',
name: { kind: 'Name', value: 'objectAccessCounts' },
arguments: [
{
kind: 'Argument',
name: { kind: 'Name', value: 'bucket' },
value: { kind: 'Variable', name: { kind: 'Name', value: 'bucket' } },
},
{
kind: 'Argument',
name: { kind: 'Name', value: 'key' },
value: { kind: 'Variable', name: { kind: 'Name', value: 'key' } },
},
{
kind: 'Argument',
name: { kind: 'Name', value: 'window' },
value: { kind: 'IntValue', value: '365' },
},
],
selectionSet: {
kind: 'SelectionSet',
selections: [
{ kind: 'Field', name: { kind: 'Name', value: 'total' } },
{
kind: 'Field',
name: { kind: 'Name', value: 'counts' },
selectionSet: {
kind: 'SelectionSet',
selections: [
{ kind: 'Field', name: { kind: 'Name', value: 'date' } },
{ kind: 'Field', name: { kind: 'Name', value: 'value' } },
],
},
},
],
},
},
],
},
},
],
} as unknown as DocumentNode<
containers_Bucket_File_gql_ObjectAccessCountsQuery,
containers_Bucket_File_gql_ObjectAccessCountsQueryVariables
>

export { containers_Bucket_File_gql_ObjectAccessCountsDocument as default }

Check warning on line 100 in catalog/app/containers/Bucket/File/gql/ObjectAccessCounts.generated.ts

View check run for this annotation

Codecov / codecov/patch/informational

catalog/app/containers/Bucket/File/gql/ObjectAccessCounts.generated.ts#L100

Added line #L100 was not covered by tests
Original file line number Diff line number Diff line change
@@ -0,0 +1,9 @@
query ($bucket: String!, $key: String!) {
objectAccessCounts(bucket: $bucket, key: $key, window: 365) {
total
counts {
date
value
}
}
}
1 change: 1 addition & 0 deletions catalog/app/containers/Bucket/File/index.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
export { default } from './File'

Check warning on line 1 in catalog/app/containers/Bucket/File/index.ts

View check run for this annotation

Codecov / codecov/patch/informational

catalog/app/containers/Bucket/File/index.ts#L1

Added line #L1 was not covered by tests
Loading