Feature Request: Webhook Integration for Custom Miners and Validators

[muhmad-f-k]

Feature Request: Webhook Integration for Custom Scraper for Miners and Validators

Description

I've encountered challenges when trying to optimize and integrate custom miners into the current system. To make it easier for miners and validators, I propose adding webhook functionality. This would allow miners to connect their private scrapers without needing to modify or rework any of the subnet's code.

Proposal

Implement a webhook option for miners, allowing them to run external scrapers while still connecting to the subnet without running a built-in miner. Validators and miners could then start their node with the following example command:

pm2 start python -- ./neurons/miner.py --wallet.name your-wallet --wallet.hotkey your-hotkey --webhook 0.0.0.0 --webhookport 12345

This command would start the webhook without running any local miner, allowing external miners to send data through the webhook and receive requests to validate miners' data.

Scraper Details Validators

Reddit

Push to custom miner through webhook: A url with requestId and source id = 1 (Reddit) is provided.

Miner responds with:

• requestId
• id
• url
• username
• communityName
• body
• createdAt
• dataType
• title
• parentId

Twitter

Push to custom miner through webhook: A url with requestId and source id = 2 (Twitter) is provided.

Miner responds with:

• requestId
• username
• text
• url
• timestamp
• tweet_hashtags

Scraper Details Miners

Miner Data Insertion

• Reddit: Miners can push data in batches through the webhook (e.g., batch of 5000), which will be added to the database.

  • Fields: id, url, username, communityName, body, createdAt, dataType, title, parentId, source

• Twitter: Miners can push data in batches through the webhook (e.g., batch of 5000), which will be added to the database.

  • Fields: username, text, url, timestamp, tweet_hashtags, source