Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

FORNO-1704: SQL Import - Improve handling of compressed files #1533

Merged
merged 11 commits into from
Oct 24, 2023

Conversation

t-wright
Copy link
Contributor

Description

This PR aims to address some inconsistencies around how compressed files are handled during SQL imports.

  1. Currently only uncompressed and gz files can be successfully imported to a VIP environment using vip import sql, but we don't validate extensions in the CLI.
  2. Currently running vip import validate-sql on a compressed file will return normal validation errors like Error: SQL Error: DROP TABLE was not found. even if the uncompressed file is valid.
  3. Currently only uncompressed files can be imported using vip dev-env import sql; compressed files are attempted to be validated and encounter the same issue as described in 2 above.

This PR addresses these issues by:

  1. Adding validation to only permit .sql and .gz extensions for both vip import sql and vip dev-env import sql
  2. Adding support for .gz archives in vip dev-env import sql, extracting the archive to a temp directory before importing the resulting SQL file as normal
  3. Returning a useful error on vip import validate-sql to explain that compressed files can't be validated

Steps to Test

  1. Download the test files below which contain this SQL:
DROP TABLE IF EXISTS `wp_a8c_cron_control_jobs`;

CREATE TABLE `wp_a8c_cron_control_jobs` (
  `ID` bigint(20) UNSIGNED NOT NULL,
  `timestamp` bigint(20) UNSIGNED NOT NULL,
  `action` varchar(255) NOT NULL,
  `action_hashed` varchar(32) NOT NULL,
  `instance` varchar(32) NOT NULL,
  `args` longtext NOT NULL,
  `schedule` varchar(255) DEFAULT NULL,
  `interval` int(10) UNSIGNED DEFAULT 0,
  `status` varchar(32) NOT NULL DEFAULT 'pending',
  `created` datetime NOT NULL,
  `last_modified` datetime NOT NULL
) ENGINE=InnoDB DEFAULT CHARSET=utf8;

.sql: https://transfer.sh/5L7MiLwtQH/test.sql
.sql.gz: https://transfer.sh/1t712ht7JL/test.sql.gz
.zip: https://transfer.sh/KJOzR1paS9/test.zip

  1. Check out PR.
  2. Run npm run build

Validation flow:

  1. Run node ./dist/bin/vip-import-validate-sql.js test.zip
  2. Run node ./dist/bin/vip-import-validate-sql.js test.sql.gz
  3. Verify the command exits and an error is shown
  4. Run node ./dist/bin/vip-import-validate-sql.js test.sql
  5. Verify the sql file validates

Dev Env flow:

  1. Run vip dev-env start --slug <your slug> to start your local development environment
  2. Run node ./dist/bin/vip-dev-env-import-sql.js test.zip --slug <your slug>
  3. Verify the command fails with an error about the extension being unsupported.
  4. Run node ./dist/bin/vip-dev-env-import-sql.js test.sql.gz --slug <your slug>
  5. Run node ./dist/bin/vip-dev-env-import-sql.js test.sql --slug <your slug>
  6. Verify both commands succeed and the file is imported

Import flow:

  1. Run node ./dist/bin/vip-import-sql.js test.zip @<id>.production
  2. Verify the command fails with an error about the extension being unsupported.
  3. Run node ./dist/bin/vip-import-sql.js test.sql @<id>.production (don't confirm)
  4. Run node ./dist/bin/vip-import-sql.js test.sql.gz @<id>.production (don't confirm)

@t-wright t-wright force-pushed the forno-1704/add/validate-sql-compressed-file-error branch from 63310e4 to 4c3be57 Compare October 20, 2023 02:31
@t-wright t-wright requested a review from a team October 20, 2023 03:44
Copy link
Contributor

@saroshaga saroshaga left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code looks good! Haven't tested all flows yet

src/commands/dev-env-import-sql.js Show resolved Hide resolved
@t-wright t-wright added the [Status] Needs Docs The feature or update requires an update to our public VIP Documentation label Oct 23, 2023
Copy link
Contributor

@saroshaga saroshaga left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thanks for working on this!

@t-wright t-wright merged commit 5d74fa1 into trunk Oct 24, 2023
9 checks passed
@t-wright t-wright deleted the forno-1704/add/validate-sql-compressed-file-error branch October 24, 2023 22:52
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
[Status] Needs Docs The feature or update requires an update to our public VIP Documentation
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants