Feat/actions crawler #1
Open
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Signed-off-by: meows <[email protected]>
Signed-off-by: meows <[email protected]>
Signed-off-by: meows <[email protected]>
meowsbits
commented
Mar 4, 2020
Comment on lines
+65
to
+66
| git remote set-url origin https://${GITHUB_USER}:${GITHUB_PAT}@github.com/${GITHUB_REPOSITORY}.git | ||
| git push origin master |
There was a problem hiding this comment.
Theoretically actions/checkout@v2 should make it as easy as git push, but I just couldn't get that to work.
meowsbits
commented
Mar 4, 2020
| [[ ! -d ${p}.${network}.${ETH_DNS_DISCV4_PARENT_DOMAIN} ]] || [[ ! -f ${p}.${network}.${ETH_DNS_DISCV4_PARENT_DOMAIN}/nodes.json ]] && { echo " | DNE, skipping"; continue; } | ||
|
|
||
| echo | ||
| devp2p dns to-cloudflare --zoneid "$ETH_DNS_CLOUDFLARE_ZONEID" "${p}.${network}.${ETH_DNS_DISCV4_PARENT_DOMAIN}" |
There was a problem hiding this comment.
This will need to be changed if you're not using Cloudflare. Also, I was hitting a quota exceeded error when I had more than 1000 records, so some kind of error handling here might be needed.
|
Hi, this looks nice, but we prefer running the crawler on our own infrastructure because that makes it easier to monitor it. Really appreciate the effort though. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Hey, We're running a downstream iteration of this at https://github.com/etclabscore/discv4-dns-lists, and have the crawler running on the Github Actions CI there. This frees up a devops task and makes the provenance of the lists transparent.
Here's a start at adding the same functionality here if you're interested.
The config currently uses Actions'
schedule: cronconfig with what I thought is a sufficient time span. There are other ways to handle this, but this seemed the simplest to me.TODO
Repo Secrets
The Action config depends on a few Secrets being set at the Github repo level.
CLOUDFLARE_API_TOKENETH_DNS_CLOUDFLARE_ZONEID(This isn't necessary if you can get get thedevp2p to-xxxto work without it.... I couldn't.)ETH_DNS_DISCV4_KEY: The full content of the signing key file, eg.cat key.json | pbcopyor whatever.ETH_DNS_DISCV4_KEYPASS: The key's password.GITHUB_USER_PERSONAL_ACCESS_TOKEN(PAT). If you can get thegit pushto work without explicit user authentication (see comment below) then you won't need this.