Skip to content

Commit

Permalink
CopyFile: support assets, archives, and recursive copying (#423)
Browse files Browse the repository at this point in the history
This PR allows the
[CopyFile](https://www.pulumi.com/registry/packages/command/api-docs/remote/copyfile/)
resource to recursively copy directories as well, similar to `scp -r`.

This PR enhances the CopyFile resource in a few ways.
- It can now copy directories as well, recursively with their contents.
- It now takes the standard Pulumi [asserts and
archives](https://www.pulumi.com/docs/concepts/assets-archives/) as
input, allowing for seamless interop with other resources.
- It now checks whether the specified file or directory have changed (in
content, not timestamp) and copies only if they did.

In light of these changes, I've renamed `CopyFile` to just `CopyToRemote` which
is a **breaking change**.

Resolves #23
Resolves #33
Resolves #42

## TODO

- [x] ~~The current implementation fails as soon as a file or directory
already exists on the remote, like the previous `CopyFile`.~~
- [x] Before GA, should we rename `CopyFile` to `Copy` or `RemoteCopy`
or _insert your proposal here_?
- [ ] The integration test copies a bunch of EC2 setup code with
`TestEc2RemoteTs`, which we should share instead. _What would be a good
way to do that, given the code is in the TS tests, not in the Go
driver?_

## Design considerations

The behavior of the copy operation was modeled after `cp` and `scp`.

```
source | dest - exists as dir | dest - does not exist | dest - exists as file
-------|----------------------|-----------------------|-----------------------
dir    | dest/dir             | dest/dir              | error
dir/   | dest/x for x in dir  | dest/dir              | error
file   | dest/file            | dest                  | dest (overwritten)
```

Specifically:
- When copying a directory, we overwrite existing files. (The current
CopyFile resource for single files does that, too.)
- When copying a directory, we _don't_ clear the remote directory first
so that no left-over files from previous copies will be around. We can
always add a flag for that if needed.

## Implementation notes

- I've looked around for open source Go packages implementing scp of
folders, but to my surprise, I couldn't find one with an acceptable
license that 1) wasn't ancient and 2) was cross-platform.
- The current implementation copies files sequentially and is therefore
probably slow for large trees. If we replaced the implementation, I
don't think the shape of the resource would change, so this shouldn't be
a GA blocker.
- The `size.ts` file in both steps of the integration test is useless
because it always has the same value. However, I found that without a TS
file present in the additional step, it would not be run. Haven't
debugged further.

---------

Co-authored-by: Ian Wahbe <[email protected]>
  • Loading branch information
thomas11 and iwahbe authored Jun 20, 2024
1 parent b424533 commit 40d4ab3
Show file tree
Hide file tree
Showing 66 changed files with 692,846 additions and 847 deletions.
1 change: 1 addition & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,7 @@ sdk/dotnet/version.txt
.DS_Store
/provider/cmd/pulumi-resource-command/schema-embed.json
dist/
.vscode

rsa*
password.txt
Expand Down
4 changes: 2 additions & 2 deletions examples/aws-py-ec2-command/__main__.py
Original file line number Diff line number Diff line change
Expand Up @@ -69,10 +69,10 @@ def decode_key(key):
)

# Copy a config file to our server.
cp_config = command.remote.CopyFile(
cp_config = command.remote.Copy(
'config',
connection=conn,
local_path='myapp.conf',
local_asset='myapp.conf',
remote_path='myapp.conf',
opts=pulumi.ResourceOptions(depends_on=[server])
)
Expand Down
7 changes: 7 additions & 0 deletions examples/dir_copy_ec2/Pulumi.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
name: copy-remote-ec2
runtime: nodejs
description: A command example copying a directory via sFTP
config:
pulumi:tags:
value:
pulumi:template: ""
Loading

0 comments on commit 40d4ab3

Please sign in to comment.