Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: implement storage management commands #1587

Merged
merged 16 commits into from
Oct 24, 2023
Merged

feat: implement storage management commands #1587

merged 16 commits into from
Oct 24, 2023

Conversation

sweatybridge
Copy link
Contributor

@sweatybridge sweatybridge commented Oct 17, 2023

What kind of change does this PR introduce?

feature
closes #1388

What is the new behavior?

Implements storage ls / cp / mv / rm as proposed.

Additional context

Not implemented in this PR:

  • support cp between buckets
  • support batch cp and mv
  • support self hosted
  • support cp file metadata
  • support file explorer for large buckets
  • support resumable TUS upload

@sweatybridge sweatybridge requested a review from a team as a code owner October 17, 2023 15:26
@sweatybridge sweatybridge changed the title Cmd storage feat: implement storage management commands Oct 17, 2023
@coveralls
Copy link

coveralls commented Oct 17, 2023

Pull Request Test Coverage Report for Build 6613802796

  • 369 of 407 (90.66%) changed or added relevant lines in 8 files are covered.
  • 5 unchanged lines in 1 file lost coverage.
  • Overall coverage increased (+1.7%) to 58.602%

Changes Missing Coverage Covered Lines Changed/Added Lines %
internal/storage/rm/rm.go 79 84 94.05%
internal/storage/cp/cp.go 96 105 91.43%
internal/storage/ls/ls.go 93 102 91.18%
cmd/storage.go 11 26 42.31%
Files with Coverage Reduction New Missed Lines %
internal/gen/keys/keys.go 5 12.31%
Totals Coverage Status
Change from base Build 6611627994: 1.7%
Covered Lines: 5661
Relevant Lines: 9660

💛 - Coveralls

@sweatybridge sweatybridge force-pushed the cmd-storage branch 2 times, most recently from c62e744 to d9f94cc Compare October 19, 2023 05:21
@sweatybridge sweatybridge force-pushed the cmd-storage branch 2 times, most recently from 1ce3edc to 41b1273 Compare October 19, 2023 13:19
@sweatybridge sweatybridge force-pushed the cmd-storage branch 2 times, most recently from 861309c to 490439d Compare October 20, 2023 07:08
cmd/storage.go Outdated Show resolved Hide resolved
cmd/storage.go Outdated Show resolved Hide resolved
}
)

func init() {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not sure if moving directory shouldn require a recursive call tbh, as this is not the behavior people are used to. Doing mv foo bar if foo is a directory will work in any unix environment. Only cp and rm require this flag.

Copy link
Contributor Author

@sweatybridge sweatybridge Oct 23, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agree on the difference. I was a bit reluctant to make mv recursive by default because it would mean at least 2 network calls to copy a file, ie. first call tries to move as a file and then second call to move as a directory.

I find adding a -r flag isn't too bad in this case because it's more consistent with other commands. Also we are only supporting moving within the same bucket (no local -> remote) so I hope the expectation doesn't need to be exactly like unix.

If it turns out to be a big inconvenience, reverting to unix behaviour would not be a breaking change for those who already use the -r flag. So this is probably ok for now.

}
fmt.Fprintln(os.Stderr, "Uploading:", filePath, "=>", dstPath)
err = client.UploadStorageObject(ctx, projectRef, dstPath, filePath, fsys)
if err != nil && strings.Contains(err.Error(), `"error":"Bucket not found"`) {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Tying ourselves to the exact error message is not great imo. Can we at least assert on http code instead? It's also available from what I see. Same for 2 different places.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

404 could either be thrown from http path not found or bucket doesn't exist. We can add the status code as an additional check but eventually we need a more specific error code to distinguish the two.

}
fmt.Fprintln(os.Stderr, "Uploading:", filePath, "=>", dstPath)
err = client.UploadStorageObject(ctx, projectRef, dstPath, filePath, fsys)
if err != nil && strings.Contains(err.Error(), `"error":"Bucket not found"`) {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not sure if we should always create a bucket without any additional flag, but from UX perspective it sounds ok.

cmd/storage.go Outdated Show resolved Hide resolved
Copy link
Member

@kamilogorek kamilogorek left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, although next time maybe let's try merge it command by command, as 2.5k LoC is difficult to fit in the head at once reviewing 😅 Especially that ls is the base, on which remaining 3 commands are built on.

@@ -49,6 +49,10 @@ func Run(ctx context.Context, paths []string, recursive bool, fsys afero.Fs) err
return err
}
for bucket, prefixes := range groups {
confirm := fmt.Sprintf("Confirm deleting files in bucket %v?", utils.Bold(bucket))
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I like that it handles non-TTY by default 😌

@sweatybridge sweatybridge enabled auto-merge (rebase) October 24, 2023 11:35
@sweatybridge sweatybridge disabled auto-merge October 24, 2023 11:38
@sweatybridge sweatybridge merged commit 691eff6 into main Oct 24, 2023
8 checks passed
@sweatybridge sweatybridge deleted the cmd-storage branch October 24, 2023 11:40
@homerjam
Copy link

This is just what I need (I think), are there any docs?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Downloading Storage Bucket Contents
4 participants