[datasets] fetching fail if corrupted file

[AlexandreAbraham]

If a file has not been downloaded properly, fetching fails instead of trying to download it again.

[AlexandreAbraham]

Should we put md5 hashtags in the code to ensure integrity ?

[AlexandreAbraham]

@Gael for your problem with NYU Test Retest dataset, my guess is that you have a corrupted zip file that was downloaded with a previous version of the downloader (now, if the download fails, it cleans everything).
Three solutions :

• you delete the file by hand
• you put the parameter "force_download=True"
• I can change the downloader to make it erase systematically any previous downloaded file if it cannot load the dataset.

[GaelVaroquaux]

Fourth solution: you change the downloader so that in such a situation it cleans up by it self. It should never fail and catch exceptions to fall back on its feets.

[GaelVaroquaux]

I cannot afford to wait any longer. I am on this issue and will fix it.

[AlexandreAbraham]

I have made a patch but it is not fully tested yet, I am testing it actually. I will push it.

[GaelVaroquaux]

I have made a patch but it is not fully tested yet, I am testing it actually. I will push it.

OK, so you are saying that I just lost an hour for nothing :$.

When do you expect to have the patch ready? I have 24 hours to get the
tutorial going. I am starting to be really nervous.

[AlexandreAbraham]

I have pushed it. For basic behavior (download from scratch, download with existing corrupted files) it works. I hope that it will work on your computer this time. I am testing it under windows right now...

[GaelVaroquaux]

I have pushed it. For basic behavior (download from scratch, download with existing corrupted files) it works. I hope that it will work on your computer this time. I am testing it under windows right now...

I have pushed a fix that works for me (I tested it manually) on a
nyu_trt_fix branch on my github. I am too tired to do anything more
tonight.

G

[AlexandreAbraham]

I think I'll make a phony dataset for testing purpose because I also find it hard to test with real dataset.

[GaelVaroquaux]

I have pushed it. For basic behavior (download from scratch, download with existing corrupted files) it works.

I don't understand: where is the branch in which the nyu_trt download is
supposed to work? From what I can see, you pushed the fix in your master,
in which the nyu_trt downloader does not exist (that's good, don't change
that). I tried creating a temporary branch to merge the changes that you
had done in your master into you nyu_trt branch, but I get non
functionning code.

Which code should I use to have a working set of examples with the NYU
dataset?

FYI, I believe that the problem with the nyu_trt is the URL used for
downloading: it tries the following:

In [2]: nyu = datasets.fetch_nyu_rest()
Downloading data from http://www.nitrc.org/frs/download.php/1071NYU_TRT_session1a.tar.gz ...

The URL above does not exist (try it). However, the following URL exists:
http://www.nitrc.org/frs/download.php/1071/NYU_TRT_session1a.tar.gz ...

I pushed a fix for this problem in my nyu_trt_fix branch, and it does
work. By the way, if you are going to use this fix, please do merge my
branch, so that we have somewhat meaningful history.

[AlexandreAbraham]

Ok, I did not understand that there also was a problem with the URL. I'll merge your branch.

[GaelVaroquaux]

AFAICT, I definitively fixed this issue in origin/master. I am closing it.

[GaelVaroquaux]

OK, I don't have permission to close it. @AlexandreAbraham you should do it.