Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: add docker compose migration #40

Merged
merged 35 commits into from
Jun 26, 2024
Merged

feat: add docker compose migration #40

merged 35 commits into from
Jun 26, 2024

Conversation

colesnodgrass
Copy link
Member

@colesnodgrass colesnodgrass commented Jun 24, 2024

  • replace/build-upon feat: add persistent volumes #33
  • add --migrate flag to the local install command (see diagram below for workflow of how this works)
  • add --persisted flag to local uninstall to remove persisted data
  • cleanup how options are passed to install
    • replace too many parameters with a struct
  • change when k8s events are logged as warnings
    • will only show a warning if the event has occurred at least 5 times
    • previously this was set to any event was shown, which resulted in unnecessary noise
    • all events will still be logged if verbose mode -v is enabled

TODO

  • I need to take another pass at this code, I expect there is some strangeness in here, but want to get some other eyes on it before it goes on even longer

flowchart TD
    mStart[migration start] -->dv{does\n`airbyte_db`\nexist}
    dv -->|no| err(return error)
    dv -->|yes| conCopyStart(tmp container\nwith `airbyte_db` mounted)
    subgraph "                                              First Container"
        conCopyStart -->  conCopyAction(copy `airbyte_db` pgdata to\n`~/.airbyte/abctl/data/airbyte-volme-db`)
        conCopyAction --> conCopyEnd(tmp container removed) 
    end
    conCopyEnd --> conCopySuccess{success?}
    conCopySuccess -->|no| err    
    conCopySuccess -->|yes| conTransStart(tmp container with\n`~/.airbyte/abctl/data/airbyte-volume-db` mounted)
    subgraph "                                              Second Container"
        conTransStart --> conTransActionUser(add postgresql user)
        conTransActionUser --> conTransActionUserSuccess{success?}
        conTransActionUserSuccess -->|yes| conTransActionDB(rename database)
        conTransActionDB --> conTransActionDBSuccess{success?}
        conTransActionDBSuccess -->|yes| conTransActionDBEnd(tmp container removed)
    end
    conTransActionUserSuccess -->|no| err
    conTransActionDBSuccess -->|no| err
    conTransActionDBEnd --> mEnd[migration complete]
    mEnd --> install[existing installation continues...]
Loading

colesnodgrass and others added 30 commits May 8, 2024 09:05
@colesnodgrass colesnodgrass marked this pull request as ready for review June 24, 2024 18:44
internal/cmd/local/docker/docker.go Show resolved Hide resolved
internal/cmd/local/docker/docker.go Show resolved Hide resolved
internal/cmd/local/docker/docker.go Outdated Show resolved Hide resolved
internal/cmd/local/docker/docker.go Outdated Show resolved Hide resolved
defer func() { d.stopContainer(ctx, conTransform.ID) }()

// TODO figure out a better way to determine when the container has successfully started
time.Sleep(10 * time.Second)
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm concerned that with slower internet connections, this might not be enough time. Can we write a simple waitForStatus(id, status, timeout) helper that could be used here? and further down. It looks like below there is an implementation of just such a loop in exec()

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm going to handle this in a follow-up PR. Get this beast of a PR merged in first.

internal/cmd/local/docker/docker.go Outdated Show resolved Hide resolved
internal/cmd/local/k8s/client.go Outdated Show resolved Hide resolved
internal/cmd/local/local_install.go Show resolved Hide resolved
internal/cmd/local/local_install.go Outdated Show resolved Hide resolved
@perangel
Copy link
Collaborator

Few small comments, and a suggestion for the StopContainer method

@colesnodgrass colesnodgrass merged commit d88f184 into main Jun 26, 2024
2 checks passed
@colesnodgrass colesnodgrass deleted the cole/migrate branch June 26, 2024 21:27
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants