Skip to content

Commit

Permalink
Correct decumulation filter not dropping temp columns
Browse files Browse the repository at this point in the history
  • Loading branch information
gnrgomes committed Jun 26, 2024
1 parent 714edd6 commit 08acf2c
Show file tree
Hide file tree
Showing 9 changed files with 103,435 additions and 103,435 deletions.
6 changes: 3 additions & 3 deletions src/lisfloodutilities/gridding/lib/filters.py
Original file line number Diff line number Diff line change
Expand Up @@ -444,12 +444,12 @@ def filter(self, kiwis_files: List[Path], kiwis_timestamps: List[str], kiwis_dat
self.df_24h_with_neighbors = self.df_24h_with_neighbors.loc[self.df_24h_with_neighbors[self.COL_VALUE] >= 0.0]

# Clean the dataframes of internal columns before merging them
self.df_24h_without_neighbors = self.df_24h_without_neighbors.drop(columns=self.INTERNAL_COLUMNS, errors='ignore')
self.df_24h_without_neighbors = self.df_24h_without_neighbors.drop(columns=self.INTERNAL_COLUMNS, axis=1, errors='ignore')

# Insert the decumulated stations in the respective 6h slots
return_data_frames = [self.kiwis_24h_dataframe]
for df in filtered_data_frames:
df = df.drop(columns=self.INTERNAL_COLUMNS, errors='ignore')
df = df.drop(columns=self.INTERNAL_COLUMNS, axis=1, errors='ignore')
# Now we need to eliminate the stations that have neighbors on this 6h slot,
# which means the slot of 6h have already a 6h value in the radius and no
# decumulation is needed in this slot.
Expand All @@ -458,7 +458,7 @@ def filter(self, kiwis_files: List[Path], kiwis_timestamps: List[str], kiwis_dat
tree = cKDTree(df[[self.COL_LON, self.COL_LAT]])
df_decumulated_24h = self.update_column_if_provider_stations_are_in_radius(df=df_decumulated_24h, tree=tree)
df_decumulated_24h = df_decumulated_24h[df_decumulated_24h['has_neighbor_within_radius'] == False]
df_decumulated_24h.drop(columns=self.INTERNAL_COLUMNS, errors='ignore')
df_decumulated_24h = df_decumulated_24h.drop(columns=self.INTERNAL_COLUMNS, axis=1, errors='ignore')
# Append both decumulated 24h dataframes to the 6h slot
df_filtered = pd.concat([df, self.df_24h_without_neighbors, df_decumulated_24h])
return_data_frames.append(df_filtered)
Expand Down
25,898 changes: 12,949 additions & 12,949 deletions tests/data/decumulate/meteo_out/pr6/2005/12/25/pr6200512251200_all.kiwis

Large diffs are not rendered by default.

25,888 changes: 12,944 additions & 12,944 deletions tests/data/decumulate/meteo_out/pr6/2005/12/25/pr6200512251800_all.kiwis

Large diffs are not rendered by default.

25,838 changes: 12,919 additions & 12,919 deletions tests/data/decumulate/meteo_out/pr6/2005/12/26/pr6200512260000_all.kiwis

Large diffs are not rendered by default.

25,808 changes: 12,904 additions & 12,904 deletions tests/data/decumulate/meteo_out/pr6/2005/12/26/pr6200512260600_all.kiwis

Large diffs are not rendered by default.

25,898 changes: 12,949 additions & 12,949 deletions tests/data/decumulate/reference/pr6/2005/12/25/pr6200512251200_all.kiwis

Large diffs are not rendered by default.

25,888 changes: 12,944 additions & 12,944 deletions tests/data/decumulate/reference/pr6/2005/12/25/pr6200512251800_all.kiwis

Large diffs are not rendered by default.

25,838 changes: 12,919 additions & 12,919 deletions tests/data/decumulate/reference/pr6/2005/12/26/pr6200512260000_all.kiwis

Large diffs are not rendered by default.

25,808 changes: 12,904 additions & 12,904 deletions tests/data/decumulate/reference/pr6/2005/12/26/pr6200512260600_all.kiwis

Large diffs are not rendered by default.

0 comments on commit 08acf2c

Please sign in to comment.