Model Fails to Perfectly Fit Data Despite Data Leakage

[asdfqwer2015]

Hi,

I hope you’re doing well. I am working on the iTransformer project and ran an experiment involving data leakage in a time series forecasting scenario. However, I encountered an issue where the model could not perfectly fit the data, even though I had fully provided future information. In theory, there should be no error since the model is given complete knowledge of future values.

For the experiment, I used my own 28-day dataset to predict the next 14 days, and the CSV file includes columns leak1, leak2, ..., leak14, representing the target values for days 1 to 14 in the future. To reduce the potential interference from regularization during the fitting process, I set the dropout rate to 0.

I will attach the model training logs for further context. Any insights or suggestions on what could be causing this issue would be greatly appreciated!

>>>>>>>start training : LeakExp_28_14_iTransformer_custom_MS_ft28_sl1_ll14_pl512_dm8_nh3_el1_dl512_df1_fctimeF_ebTrue_dt'Exp'_projection_0>>>>>>>>>>>>>>>>>>>>>>>>>>
train 1159
val 159
test 330
Epoch: 1 cost time: 5.087770938873291
Epoch: 1, Steps: 72 | Train Loss: 0.3431292 Vali Loss: 0.1607062 Test Loss: 0.2599300
Validation loss decreased (inf --> 0.160706).  Saving model ...
Updating learning rate to 0.0005
Epoch: 2 cost time: 2.9762840270996094
Epoch: 2, Steps: 72 | Train Loss: 0.2044745 Vali Loss: 0.2174404 Test Loss: 0.3305312
EarlyStopping counter: 1 out of 3
Updating learning rate to 0.00025
Epoch: 3 cost time: 2.8668558597564697
Epoch: 3, Steps: 72 | Train Loss: 0.1589616 Vali Loss: 0.0896386 Test Loss: 0.1736311
Validation loss decreased (0.160706 --> 0.089639).  Saving model ...
Updating learning rate to 0.000125
Epoch: 4 cost time: 2.8665754795074463
Epoch: 4, Steps: 72 | Train Loss: 0.0887348 Vali Loss: 0.0840708 Test Loss: 0.1486780
Validation loss decreased (0.089639 --> 0.084071).  Saving model ...
Updating learning rate to 6.25e-05
Epoch: 5 cost time: 2.8994083404541016
Epoch: 5, Steps: 72 | Train Loss: 0.0724858 Vali Loss: 0.0786882 Test Loss: 0.1377135
Validation loss decreased (0.084071 --> 0.078688).  Saving model ...
Updating learning rate to 3.125e-05
Epoch: 6 cost time: 2.881927251815796
Epoch: 6, Steps: 72 | Train Loss: 0.0661208 Vali Loss: 0.0753254 Test Loss: 0.1362566
Validation loss decreased (0.078688 --> 0.075325).  Saving model ...
Updating learning rate to 1.5625e-05
Epoch: 7 cost time: 2.8963589668273926
Epoch: 7, Steps: 72 | Train Loss: 0.0614291 Vali Loss: 0.0746888 Test Loss: 0.1371633
Validation loss decreased (0.075325 --> 0.074689).  Saving model ...
Updating learning rate to 7.8125e-06
Epoch: 8 cost time: 2.869476318359375
Epoch: 8, Steps: 72 | Train Loss: 0.0613874 Vali Loss: 0.0792014 Test Loss: 0.1365161
EarlyStopping counter: 1 out of 3
Updating learning rate to 3.90625e-06
Epoch: 9 cost time: 2.8526360988616943
Epoch: 9, Steps: 72 | Train Loss: 0.0605294 Vali Loss: 0.0713771 Test Loss: 0.1360513
Validation loss decreased (0.074689 --> 0.071377).  Saving model ...
Updating learning rate to 1.953125e-06
Epoch: 10 cost time: 2.84830379486084
Epoch: 10, Steps: 72 | Train Loss: 0.0601796 Vali Loss: 0.0744032 Test Loss: 0.1356441
EarlyStopping counter: 1 out of 3
Updating learning rate to 9.765625e-07
>>>>>>>testing : LeakExp_28_14_iTransformer_custom_MS_ft28_sl1_ll14_pl512_dm8_nh3_el1_dl512_df1_fctimeF_ebTrue_dt'Exp'_projection_0<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
test 330
test shape: (330, 1, 14, 1) (330, 1, 14, 1)
test shape: (330, 14, 1) (330, 14, 1)
mse:0.13605129718780518, mae:0.21121205389499664, mape:1.208109736442566