Support reading from gzipped files #17
Conversation
|
@Myles1 Thank you so much for your pull request and your patience. I think this definitely makes sense and is a value-add. The one thing that I would request to be changed is to avoid adding the Once updated we'll gladly accept this PR with much thanks! |
| @@ -10,7 +11,7 @@ import { | |||
| import { ValidationResult } from "hpt-validator/src/types" | |||
| import { InvalidArgumentError } from "commander" | |||
|
|
|||
| type FileFormat = "csv" | "json" | |||
| type FileFormat = "csv" | "json" | "json.gz" | "csv.gz" | |||
There was a problem hiding this comment.
Let's remove .gz extensions as acceptable format types.
| const fileExt = path.extname(filepath).toLowerCase().replace(".", "") | ||
| if (["csv", "json"].includes(fileExt)) { | ||
| return fileExt as FileFormat | ||
| const isGzipped = path.extname(filepath).toLowerCase() === ".gz" |
There was a problem hiding this comment.
Potentially hoist this check up to determine if one of the acceptable formats is gzipped.
| @@ -14,7 +14,9 @@ async function main() { | |||
| .addOption( | |||
| new Option("-f, --format <string>", "file format of file").choices([ | |||
| "csv", | |||
| "csv.gz", | |||
There was a problem hiding this comment.
Remove these new options as acceptable formats as the main branch of logic would handle this internally.
Thanks for publishing and maintaining this project. Super helpful stuff!
Feel free to just close this PR if you don't want it. No biggie either way :)
One line description of your change (less than 72 characters)
Support reading from
.csv.gzand.json.gzfiles.Problem
Explain the context and why you're making that change. What is the problem
you're trying to solve? In some cases there is not a problem and this can be
thought of being the motivation for your change.
I'm compressing the data files locally and need to unzip them before I can run the
hpt-validator-cli. This just lets us read gzipped files directly.Solution
Describe the modifications you've done.
Allow reading directly from gzipped data files.
Test Plan
(Write your test plan here. If you changed any code, please provide us with
clear instructions on how you verified your changes work.)
Create a gzipped data file
$gzip my_file.jsonYou should now have a file named
my_file.json.gzRun
hpt-validator-clion that new file.Ensure the results are the same as they were on the unzipped file.