-
Notifications
You must be signed in to change notification settings - Fork 0
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Merge pull request #57 from lyrasis/delete-transform-updates
Delete transform updates; add `Delete::EmptyFields` transform
- Loading branch information
Showing
26 changed files
with
1,675 additions
and
350 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,6 +1,7 @@ | ||
| /.bundle/ | ||
| /.yardoc | ||
| /_yardoc/ | ||
| /docs | ||
| /coverage/ | ||
| /pkg/ | ||
| /spec/reports/ | ||
|
|
||
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1 +1 @@ | ||
| 3.0.2 | ||
| 2.7.4 |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
111 changes: 111 additions & 0 deletions
111
lib/kiba/extend/transforms/delete/empty_field_values.rb
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,111 @@ | ||
| # frozen_string_literal: true | ||
|
|
||
| module Kiba | ||
| module Extend | ||
| module Transforms | ||
| module Delete | ||
|
|
||
| # @note Only useful for multi-valued fields | ||
| # | ||
| # Deletes any empty values from the field. Supports `usenull` = true to treat the value of | ||
| # `Kiba::Extend.nullvalue` as empty | ||
| # | ||
| # # Examples | ||
| # | ||
| # Assuming `Kiba::Extend.nullvalue` = `%NULLVALUE%`, and input table: | ||
| # | ||
| # ``` | ||
| # | data | | ||
| # |------------------| | ||
| # | abc;;;d e f | | ||
| # | ;;abc | | ||
| # | def;;;; | | ||
| # | ;;;;; | | ||
| # | ;;;%NULLVALUE%;; | | ||
| # | | | ||
| # | nil | | ||
| # ``` | ||
| # | ||
| # Used in pipeline as: | ||
| # | ||
| # ``` | ||
| # transform Delete::EmptyFieldValues, fields: [:data], sep: ';' | ||
| # ``` | ||
| # | ||
| # Results in: | ||
| # | ||
| # ``` | ||
| # | data | | ||
| # |-------------| | ||
| # | abc;d e f | | ||
| # | abc | | ||
| # | def | | ||
| # | | | ||
| # | %NULLVALUE% | | ||
| # | | | ||
| # | nil | | ||
| # ``` | ||
| # | ||
| # Used in pipeline as: | ||
| # | ||
| # ``` | ||
| # transform Delete::EmptyFieldValues, fields: [:data], sep: ';', usenull: true | ||
| # ``` | ||
| # | ||
| # Results in: | ||
| # | ||
| # ``` | ||
| # | data | | ||
| # |-----------| | ||
| # | abc;d e f | | ||
| # | abc | | ||
| # | def | | ||
| # | | | ||
| # | | | ||
| # | | | ||
| # | nil | | ||
| # ``` | ||
| # | ||
| class EmptyFieldValues | ||
| # @note `sep` will be removed in a future version. **DO NOT USE** | ||
| # @param fields [Array<Symbol>,Symbol] field(s) to delete from | ||
| # @param sep [String] **DEPRECATED; DO NOT USE** | ||
| # @param delim [String] on which to split multivalued fields. Defaults to `Kiba::Extend.delim` if not provided. | ||
| # @param usenull [Boolean] whether to treat `Kiba::Extend.nullvalue` string as an empty value | ||
| def initialize(fields:, sep: nil, delim: nil, usenull: false) | ||
| @fields = [fields].flatten | ||
| @usenull = usenull | ||
|
|
||
| if sep && delim | ||
| puts %Q[#{Kiba::Extend.warning_label}: Do not use both `sep` and `delim`. Prefer `delim`] | ||
| elsif sep | ||
| puts %Q[#{Kiba::Extend.warning_label}: The `sep` keyword is being deprecated in a future version. Change it to `delim` in your ETL code.] | ||
| @delim = sep | ||
| else | ||
| @delim = delim ? delim : Kiba::Extend.delim | ||
| end | ||
| end | ||
|
|
||
| # @private | ||
|
|
||
| def process(row) | ||
| fields.each do |field| | ||
| val = row.fetch(field) | ||
| next if val.nil? | ||
|
|
||
| row[field] = val.split(delim) | ||
| .compact | ||
| .reject{ |str| Helpers.empty?(str, usenull) } | ||
| .join(delim) | ||
| end | ||
| row | ||
| end | ||
|
|
||
| private | ||
|
|
||
| attr_reader :fields, :delim, :usenull | ||
| end | ||
| end | ||
| end | ||
| end | ||
| end |
Oops, something went wrong.