Skip to content

Commit

Permalink
rename config value
Browse files Browse the repository at this point in the history
  • Loading branch information
JamieDeMaria committed Mar 27, 2023
1 parent 22464ce commit 74458b4
Show file tree
Hide file tree
Showing 3 changed files with 9 additions and 9 deletions.
2 changes: 1 addition & 1 deletion MIGRATION.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@ When new releases include breaking changes or deprecations, this document descri
### Breaking Changes

#### Extension Libraries
[dagster-snowflake-pandas] Prior to `dagster-snowflake` version `0.19.0` the Snowflake I/O manager converted all timestamp data to strings before loading the data in Snowflake, and did the opposite conversion when fetching a DataFrame from Snowflake. The I/O manager now ensures timestamp data has a timezone attached and stores the data as TIMESTAMP_NTZ(9) type. If you used the Snowflake I/O manager prior to version `0.19.0` you can set the `time_data_to_string=True` configuration value for the Snowflake I/O manager to continue storing time data as strings while you do table migrations.
[dagster-snowflake-pandas] Prior to `dagster-snowflake` version `0.19.0` the Snowflake I/O manager converted all timestamp data to strings before loading the data in Snowflake, and did the opposite conversion when fetching a DataFrame from Snowflake. The I/O manager now ensures timestamp data has a timezone attached and stores the data as TIMESTAMP_NTZ(9) type. If you used the Snowflake I/O manager prior to version `0.19.0` you can set the `store_timestamps_as_strings=True` configuration value for the Snowflake I/O manager to continue storing time data as strings while you do table migrations.

To migrate a table created prior to `0.19.0` to one with a TIMESTAMP_NTZ(9) type, you can run the follow SQL queries in Snowflake. In the example, our table is located at `database.schema.table` and the column we want to migrate is called `time`:

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -35,7 +35,7 @@ def _convert_timestamp_to_string(s: pd.Series, column_types) -> pd.Series:
raise DagsterInvariantViolationError(
"Snowflake I/O manager configured to convert time data to strings, but the"
" corresponding column is not of type VARCHAR, it is of type"
f" {column_types[s.name]}. Please set time_data_to_string=False in the"
f" {column_types[s.name]}. Please set store_timestamps_as_strings=False in the"
" Snowflake I/O manager configuration to store time data as TIMESTAMP types."
)
return s.dt.strftime("%Y-%m-%d %H:%M:%S.%f %z")
Expand Down Expand Up @@ -97,7 +97,7 @@ def handle_output(
connector.paramstyle = "pyformat"
with_uppercase_cols = obj.rename(str.upper, copy=False, axis="columns")
column_types = _get_table_column_types(table_slice, connection)
if context.resource_config["time_data_to_string"]:
if context.resource_config and context.resource_config["store_timestamps_as_strings"]:
with_uppercase_cols = with_uppercase_cols.apply(
lambda x: _convert_timestamp_to_string(x, column_types),
axis="index",
Expand Down Expand Up @@ -134,7 +134,7 @@ def load_input(
result = pd.read_sql(
sql=SnowflakeDbClient.get_select_statement(table_slice), con=connection
)
if context.resource_config["time_data_to_string"]:
if context.resource_config and context.resource_config["store_timestamps_as_strings"]:
result = result.apply(_convert_string_to_timestamp, axis="index")
result.columns = map(str.lower, result.columns) # type: ignore # (bad stubs)
return result
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -129,7 +129,7 @@ def my_table_a(my_table: pd.DataFrame) -> pd.DataFrame:
),
is_required=False,
),
"time_data_to_string": Field(
"store_timestamps_as_strings": Field(
bool,
default_value=False,
description=(
Expand All @@ -143,13 +143,13 @@ def my_table_a(my_table: pd.DataFrame) -> pd.DataFrame:
}
)
def snowflake_io_manager(init_context):
if init_context.resource_config["time_data_to_string"]:
if init_context.resource_config["store_timestamps_as_strings"]:
deprecation_warning(
"Snowflake I/O manager config time_data_to_string",
"Snowflake I/O manager config store_timestamps_as_strings",
"2.0.0",
(
"Convert existing tables to use timestamps and remove time_data_to_string"
" configuration instead."
"Convert existing tables to use timestamps and remove"
" store_timestamps_as_strings configuration instead."
),
)
return DbIOManager(
Expand Down

0 comments on commit 74458b4

Please sign in to comment.