-
Notifications
You must be signed in to change notification settings - Fork 45
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
util: update gen_data scripts to use xivapi library #28
Conversation
I don't have time to review the actual script changes today, but I can offer feedback on this. Are/were the cloud data centers marked as |
They are set to
Given your comments, I'll go ahead and push the commit for option 2. |
@valarnin Changes as discussed. All scripts re-run, and as expected, no data changes. Thanks for the feedback. The changeover from {} to [] in the xivapi library did break the "error-handling" (such as it was) in gen_effect_id. I have another PR queued up that I'm ready to submit after this lands that adds more robust logging/error handling, So I've quick-fixed it for now by removing the data lookup from |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This looks good now. We should make sure to merge the follow up PR before pushing an actual release though, just so it's included in the generated tag.
Rebase failed
) Util cleanup project, part 2 of 3 (ref: #26) This updates the remaining `util/gen_(data)` scripts to use the new xivapi library. I've deleted `coinach.ts`, since we are no longer reading SaintCoinach data directly and the file-writer function is now in the xivapi library. All 3 scripts were run and updated output files are included. The changes in `effect_id.ts` and `pet_names.ts` are routine; the underlying game data has updated since the last time these scripts were run. (These same changes occur if you run the unmodified scripts.) The `world_ids.ts` file has substantial changes. The original xivapi call in that script did not include a `limit=` param in the url; as a result, xivapi was only returning 100 records (and that's why we only have 100 zones in the file currently). This default limit is referenced [here](https://xivapi.com/docs/Game-Data). There are actually 861 records from the `World` [endpoint](https://xivapi.com/World), so we've never imported the bulk of them. Most are trash/dev/test zones, but we have been omitting a few valid public worlds. The new xivapi library picks up all records, but the SNR here is not ideal. Two possible options: 1. Import everything, que sera sera. But we should add the isPublic flag so we have a way to differentiate. (This is the PR currently) 2. Use `isPublic=true` to filter the output. This is much more manageable and adds the missing zones we want, but it does drop trash worlds that are in the current file. (With this approach, we don't need to set isPublic in the output, as it's implicit). I'm attaching the output for this option in case anyone wants to run a diff. [world_id_option_2.txt](https://github.com/OverlayPlugin/cactbot/files/13804000/world_id_option_2.txt) I lean toward option 2, since I think that covers our use cases. If that's the consensus, I'll do a quick fix before we land the PR. (cc: @valarnin - I think you last worked on `gen_world_id` for raidemulator. Any opinion?) 8c976bc
PR 3 of 3 (ref: #26 & #28) A handful of misc. improvements: - More robust console logging functionality to the util data scripts and xivapi library - Adds the 3 de-pythoned scripts to the `npm run util` CLI utility (along with a run-all option), and allows the user to specify a logging level to control console output - Better error capture throughout - Some minor code cleanup (mostly standardization around use of CSV/locale tables) - Corresponding docs updates to `util/README.md` and `docs/PatchUpdateChecklist.md` All scripts re-run, no data changes (as expected). This completes the work I had planned on this project, but in the course of pulling all this together, I wrote a super-lightweight CLI utility (integrated into `npm run util` ) to run custom queries against xivapi, along with some very basic filtering capability. It came in very handy for investigating data issues, and I could see it being useful if (when 👀 ) future patches create new data collisions or other issues. It needs some cleanup before it's ready for code review, but I'll likely add that as a future PR, time permitting.
PR 3 of 3 (ref: #26 & #28) A handful of misc. improvements: - More robust console logging functionality to the util data scripts and xivapi library - Adds the 3 de-pythoned scripts to the `npm run util` CLI utility (along with a run-all option), and allows the user to specify a logging level to control console output - Better error capture throughout - Some minor code cleanup (mostly standardization around use of CSV/locale tables) - Corresponding docs updates to `util/README.md` and `docs/PatchUpdateChecklist.md` All scripts re-run, no data changes (as expected). This completes the work I had planned on this project, but in the course of pulling all this together, I wrote a super-lightweight CLI utility (integrated into `npm run util` ) to run custom queries against xivapi, along with some very basic filtering capability. It came in very handy for investigating data issues, and I could see it being useful if (when 👀 ) future patches create new data collisions or other issues. It needs some cleanup before it's ready for code review, but I'll likely add that as a future PR, time permitting. 2c7b245
Util cleanup project, part 2 of 3 (ref: #26)
This updates the remaining
util/gen_(data)
scripts to use the new xivapi library. I've deletedcoinach.ts
, since we are no longer reading SaintCoinach data directly and the file-writer function is now in the xivapi library.All 3 scripts were run and updated output files are included. The changes in
effect_id.ts
andpet_names.ts
are routine; the underlying game data has updated since the last time these scripts were run. (These same changes occur if you run the unmodified scripts.)The
world_ids.ts
file has substantial changes. The original xivapi call in that script did not include alimit=
param in the url; as a result, xivapi was only returning 100 records (and that's why we only have 100 zones in the file currently). This default limit is referenced here. There are actually 861 records from theWorld
endpoint, so we've never imported the bulk of them. Most are trash/dev/test zones, but we have been omitting a few valid public worlds.The new xivapi library picks up all records, but the SNR here is not ideal. Two possible options:
isPublic=true
to filter the output. This is much more manageable and adds the missing zones we want, but it does drop trash worlds that are in the current file. (With this approach, we don't need to set isPublic in the output, as it's implicit). I'm attaching the output for this option in case anyone wants to run a diff.world_id_option_2.txt
I lean toward option 2, since I think that covers our use cases. If that's the consensus, I'll do a quick fix before we land the PR.
(cc: @valarnin - I think you last worked on
gen_world_id
for raidemulator. Any opinion?)