PyPoE - Wiki Fork

Collection of Python Tools for Path of Exile.

As of Aug 2021, the PoE Wiki project had forked from RePoE which itself was a fork of the discontinued OmegaK2/PyPoE repo and exists solely to keep the Community Wiki updated. The tooling in here serves to help with datamining efforts for Path of Exile and as a result any development and contributions are welcome and encouraged. Please chat to us on Discord #tools-dev channel or leave us a issue here on the repository.

How does this work?

Each new Path of Exile league has an updated game data file, which has to be parsed out and mined for information about the items, monsters, league mechanics and changes to the core and secondary/tertiary game mechanics which then need to make their way into the PoE Wiki.

These tools rely on specification files which are able to parse and read the *.dat files contained in the main game data file in order to determine what type of data holds what value at any given time and how to translate this into a format the Wiki can understand. These change patch to patch and have to be updated.

More detailed docs: http://omegak2.net/poe/PyPoE/

Overview

• Library toolkit for programmers (PyPoE/poe)
• UI based on Qt for browsing the game files (currently not working) -- marked for deprecation
• CLI interface for extracting/exporting data (for the wiki, more TBD)

Getting Started

1. Install:

   • Python 3.11 - https://www.python.org/downloads/release/python-3110/
   • Poetry - https://python-poetry.org/docs/#installation

2. Clone:

   git clone https://github.com/Project-Path-of-Exile-Wiki/PyPoE

3. Setup:

   • In the cloned folder run - poetry install to set up all dependencies and install project.
   • To run under the virtual environment that Poetry will make for you simply call poetry shell once for the lifetime of your terminal session. For the remainder of this documentation just assume you should always be in the active venv which Poetry will active for you using that command.
   • To exit the virtual environment Poetry activated simply type deactivate in the terminal window where you ran poetry shell.
   • to activate pre-commit hooks for your local repository, run pre-commit install while in the poetry shell.

4. Testing:

   pytest -s -v .

5. Updating the specification from the source-of-truth schema:

   • Update the generated schema:
     pypoe_schema_import -a stable

   • Use the generated schema by default:
     pypoe_exporter config set version GENERATED

   • Column aliases can be edited in PyPoE/poe/file/specification/generation/virtual_fields.py

6. Running:

   • Configure output directory:
     pypoe_exporter config set out_dir ../out/

   • Configure temp directory (images will be output here):
     pypoe_exporter config set temp_dir ../tmp/

   • Configure ggpk path:
     pypoe_exporter config set ggpk_path '.../Path of Exile/'

   • Perform dry run:
     ./export.bash --threads 30 -u <wiki-username> -p <wiki-password> --dry-run

   • Check exported data in the output directory, especially the ./diff/ subdirectory which will show all changes that would made to the wiki

   • Update the wiki (Caution! Updates the live site!):
     ./export.bash --threads 30 -u <wiki-username> -p <wiki-password> --export

7. Installing as a dependency

   • To install PyPoE as a dependency of a project that doesn't use poetry, run poetry build, then pip install ./dist/pypoe-*.*.*.tar.gz.

8. Running in docker

   • Build the docker image with:
     docker build . -t pypoe

   • Run a command within the docker image:
     docker run pypoe pypoe_exporter wiki items item name "Silver Flask" --print

   • Run the full export script:
     docker run pypoe ./export.bash --threads 30 -u <wiki-username> -p <wiki-password> --dry-run

   • Mount a volume within the docker (for instance you could mount a local directory to /pypoe/out to save exporter output, or mount your local code to the PyPoE module location to test local changes without rebuilding the docker image):
     docker run -v "$PWD"/out:/pypoe/out -v "$PWD"/PyPoE:$(docker run pypoe python -c "import PyPoE; print(PyPoE.__path__[0])") pypoe ./export.bash --write

   • The image reads game content from patchcdn.pathofexile.com by default. To mount your local ggpk dir you will need to open a shell and configure pypoe before running your commands:

   docker run -v /path/to/ggpk:/ggpk_path -it pypoe bash
   pypoe_exporter config set ggpk_path /ggpk_path
   pypoe_exporter setup perform
   # ...
   

Setting up on VSCode

VSCode has some great integrations with all this tooling. In order for you to benefit from them, please ensure you adjust your settings to the following..

VSCode Extensions

• Python
• isort
• Pylance

VSCode user-settings.json

..Activated by CTRL(CMD) + SHIFT + P and by typing > Open user settings (JSON). Ensure you have the below in your settings JSON.

    ...
    "[python]": {
        "editor.defaultFormatter": "ms-python.black-formatter",
        "editor.formatOnSave": true,
        "editor.codeActionsOnSave": {
            "source.organizeImports": true,
        },
    },
    "flake8.args": [
        "--config",
        ".flake8"
    ],
    "editor.formatOnSave": true,
    ...

Common problems & advisory

---

• GUI code: UI will be reworked for bundle support and is not functional at the moment

• Running on Windows: On Windows 10 machines there seems to a be bug in the Python installation that prevents arguments being passed to the command line interface; you can identify this issue if you get a "help" listing if you supplied more then 1 argument. See this on stack overflow for possible solutions

• Merging older branches: merging branches created before the repository was formatted will result in a lot of merge conflicts. The following process can help to reduce conflicts:

# merge the last commit before the repository was formatted
git merge 0f61251e463b3a43780a0e019e008c9ab0cdd35a

# follow the steps above to setup to setup poetry,
# or if poetry has already been set up in this directory
# just run:
poetry shell

# run the formatter on the merged code and create a new commit:
pre-commit run --all-files
git commit -am 'format all'

# merge with the formatted commit
git merge 1319a525c2f165c6b0f9f389717e8bce35e00083

# merge with the target branch
git merge dev

Further Reading

• Exporting data for the wiki
• Contribution guide

Credits - People

---

• Grinding Gear Games - they created many of the file formats and Path of Exile obviously, so do not reuse their files anywhere without their permission and support them if you are able to :)
• OmegaK2 - Original developer of PyPoE
• brather1ng - For the updated fork
• Chriskang and the original VisualGGPK2
• chuanhsing (poedb) for helping with meaning of certain specification values and retrieving monster stats

Credits - Libraries

---

• pyside2 (pypi)
• configobj (pypi)
• colorama (pypi)
• sphinx (pypi)
• pytest (pypi)
• PyOpenGL (pypi)
• tqdm (pypi)
• graphviz (pypi)
• mwclient (pypi)
• mwclientparserfromhell (pypi)
• rapidfuzz (pypi)
• pydds (pypi)
• pyooz (pypi)