Split.py does not filter all dataset

[Rolamjaya]

The criteria made inside create_data_filter is just applied to train dataset while validation and test dataset just left as it is.

Any idea what should we do to change it?

Anw, there is a slight issue in the criteria, it should be:

return (
(dataset["fare_amount"] > 0)
& (dataset["trip_distance"] < 400)
& (dataset["trip_distance"] > 0)
& (dataset["fare_amount"] < 1000)
) & (~dataset.isna().any(axis=1))