InvalidDataException Unable to find deck with gatherling id '84479' -- recurrence of the gatherling-deck-changed-id problem

[vorpal-buildbot]

Reported on Discord by bakert#2193

[bakert]

This may not be a recurrence of that problem as we have this entry in our db:

| 68137 |      3976 |         2 | 84479      | Deck - Guildgate Control |   1582396200 |   1582410504 |           3139 | https://gatherling.com/deck.php?mode=view&id=84479 |          308 | NULL         | NULL          |  NULL | NULL          | NULL                |     11 | 4a5ae1c65f944e733b5b2ffbba6dcf5cac4003e3 |       0 |        1 |

Meanwhile this page loads just fine and shows a deck:

https://gatherling.com/deck.php?mode=view&id=84479

Whereas with the former bug I'd expect the URL in the db to lead us to a blank page with id not known.

[bakert]

I am not able to repro this bug on local. Local finds the deck and adds it fine and passes over it in subsequent runs fine too.

[bakert]

The problem seems to be that https://gatherling.com/deck.php?mode=view&id=68249 has no matches in the db. The competition is partially scraped.

Edit: that's a pauper deck I'm not sure what I meant here.

[bakert]

First occurrence of the error is Sat, Feb 22, 11:28 PM (2 days ago) GMT so the failure that caused the partial scrape is probably an hour before that. Note that the PD server is on Australian time.

[bakert]

That's 10:28am AEDT on Feb 23.

There is nothing suspicious in the log at that time. But does the scraper log to uwsgi journal?

[bakert]

scraper.log just logs the last run so nothing interesting there.

[bakert]

OK so the problem is that https://pennydreadfulmagic.com/decks/68137/ (Gatherling identifier 84479) has an id in the decksite db and matches in the decksite db. Then when we try to add_ids in the scraper to the match between Gatherling id 84489 and Gatherling id 84479 we fail because 84479 is not in our list because it already has id and matches.

MariaDB [decksite]> SELECT * FROM deck WHERE identifier = '84489';
Empty set (0.32 sec)

[bakert]

84489 was 84486 when we scraped the tournament. This is the root cause of the issue. Gatherling ids should be stable. See PennyDreadfulMTG/gatherling#273