Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

POI Dispute service #12

Open
wants to merge 11 commits into
base: main
Choose a base branch
from
Open

POI Dispute service #12

wants to merge 11 commits into from

Conversation

goaaron
Copy link

@goaaron goaaron commented Aug 10, 2021 •
edited
Loading

Code here is an MVP for automating the process of collecting indexer data,generating divergent blocks and matching events, and using those to determine a root cause for POI mismatch. This is one part of the tool, the other part is a subsequent PR against graph-node.

Workflow goes like this (potential add-ons noted):

  1. A dispute is created in the contract (on chain).

    Can the tenderley notification here be used to pipe to the next step automatically?

  2. Dispute and any indexers that can contribute to it are created in the dispute service database.

    Currently the API has you provide a list of indexer IDs. Can be modified to make a call to the network subgraph for a given dispute ID and automatically populate those IDs.

  3. Indexers supply their POI through graphman CLI. Data is sent in a compressed format.

    Currently the only check here is that the indexer ID provided by the graphman CLI is contained in the dispute record that the data is being uploaded for. People could spoof their indexer id. One idea here is to use the provided POI data to validate the authenticity of this indexer. Can we get a signature from the indexer's wallet as a better guarantee?

  4. When enough indexers are accounted for (TBD how to formalize this), a team mate progresses the dispute process and has the service calculate divergent blocks and save subgraph event data at those blocks.

    Progression means changing the state of the dispute in the database and making an API request for the next stage (only possible when state changes in the DB). It may be possible to determine a progression criteria to make this automated, but that ultimately sounds brittle. Would be nice to have a dispute portal frontend that allows someone to interact with these disputes and summarize them without having to connect to the database or make api calls directly.

  5. Indexers dump their entities and call cache at the divergent blocks by making a request to the service, getting the block numbers, and subsequently uploading their filtered table data.

    All data is stored as csvs. Need to also verify that these are not tampered with (same concern as pt. 3).

  6. All of the data is in one spot and exploratory analysis can be run. Some automated analysis can also be run (finding all mismatched entities and events)

API DOCS and Stubs

SwaggerUI available at http://localhost:8000/docs

Insomnia API request collections also in its own directory

Immediate things to do

  1. GCloud secrets are required to interact with the cloud storage. What is the best way of accessing those?
  2. Containerize the application
  3. Make all constants environment variable configurations (.env.dev,.env.prod)
  4. Get a database instance to connect to in GCS

@goaaron goaaron requested a review from fordN August 10, 2021 16:31
@goaaron goaaron changed the title POI Dispute service[WIP] POI Dispute service Sep 23, 2021
@goaaron
apis and data models for data wrangling prior to calculating divergen…
cf4bf77 
…t blocks...coming up

automate the creation of divergent blocks

broken

fix conflict with sqlalchemy version

make the dispute stage an enum

all apis v0 and insomnia requests

cleanup and documentation on how to run the application

remove debugger

modify version of ipfshttpclient, change module name from web3 to w3

addendum to readme requirements

container file and Makescripts

Update README.md

multi stage docker build. use poetry inside of docker

Build dockerfile

added cicd and k8s files

run on updated k8s .github paths

fix cicd context

fix repo name

moving some fields to enums. cleanup some async code

base frontend for dispute portal

foundation redux actions

bump webpack, setup base dispute page

expose get all disputes

list disputes

simplify state with zustand

merge state updates

swap out the stages of the dispute

allows frontend to toggle the stages of the dispute

swap state in the frontend prior to revalidation. revalidation still hits later

sort, filter, visualize uploaded data

dispute portal list events

Fix rainbow

Fix setup when the config file is not yet created

Define files to use when publishing package

Add bin and cleanup gitignore

filter disputes by status (#13)

* filter by status

* Update README.md

Simplify cli (#14)

* remove dispute command and show help by default

* update README.md

* fix usage

Add Economics section and fetch network settings (#15)

Sort disputes by date (#16)

v0.2.0

Add defaultDisplayName to indexer and fisherman (#17)

Check thawing period before resolving dispute (#19)

Support for multiple ids on show and improved styling (#20)

* Support for multiple ids on show and improved styling

* Replace treeify with object-treeify

Add closing epoch startBlock and poi info (#21)

v0.3.0

[#24] Fix show command (#25)

v0.3.1

remove conf file

testing deployment
@goaaron goaaron force-pushed the goaaron/poi_dispute_service branch from 37f67e2 to cf4bf77 Compare September 23, 2021 19:28
@goaaron
install readme
a6303cd
@goaaron
Merge branch 'main' of https://github.com/edgeandnode/graph-disputes …
e81af8d 
…into goaaron/poi_dispute_service
@goaaron
cleanup and try redeploy
fc1f162
@goaaron
cleanup
84cfc87 
remove debugging

print out databse connection

point to the correct container repostiory

don't fully declar ghcr.io for container. update on k8s update

move to entrypoint

change to shell

forego the entrypoint script
@goaaron goaaron force-pushed the goaaron/poi_dispute_service branch from 089ce9a to 84cfc87 Compare September 23, 2021 23:35
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@fordN fordN Awaiting requested review from fordN

At least 1 approving review is required to merge this pull request.

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
1 participant
@goaaron