Skip to content

Latest commit

 

History

History
78 lines (68 loc) · 2.92 KB

VALIDATOR.md

File metadata and controls

78 lines (68 loc) · 2.92 KB

LogicNet: Validator documentation

Overview

The Validator is responsible for generating challenges for the Miner to solve. The Validator will receive the solutions from the Miner, evaluate them, and reward the Miner based on the quality of the solutions. The Validator will also be responsible for calculating the rewards based on the correctness and quality of the solutions.

Protocol: LogicSynapse.

  • Validator prepare:
    • raw_logic_question: The math problem generated from MathGenerator.
    • logic_question: The challenge generated by the Validator. It's rewrited by LLM from raw_logic_question with personalization noise.
  • Miner will be provided:
    • logic_question: The challenge generated by the Validator.
  • Miner have to fill following content in the synapse to submit the solution:
    • logic_reasoning: The step by step reasoning to solve the challenge.
    • logic_answer: The final answer of the challenge as a short sentence.

Reward Structure:

  • correctness (bool): Validator ask LLM to check the matching between logic_answer and the ground truth.
  • similarity (float): Validator compute cosine similarity between logic_reasoning and validator's reasoning.
  • time_penalty (float): Penalty for late response. It's value of process_time / timeout * MAX_PENALTY.

Minimum Compute Requirements

  • 1x GPU 24GB VRAM (RTX 4090, A100, A6000, etc)
  • Storage: 100GB
  • Python 3.10

Setup for Validator

  1. Git clone the repository
git clone https://github.com/LogicNet-Subnet/LogicNet logicnet
cd logicnet
  1. Install the requirements
python -m venv main
. main/bin/activate

bash install.sh

or manually install the requirements

pip install -e .
pip uninstall uvloop -y
pip install git+https://github.com/lukew3/mathgenerator.git
  1. Create env for vLLM
python -m venv vllm
. vllm/bin/activate
pip install vllm

For ease of use, you can run the scripts as well with PM2. To install PM2:

sudo apt update && sudo apt install jq && sudo apt install npm && sudo npm install pm2 -g && pm2 update
  1. Setup LLM Configuration
  • Self host a vLLM server
. vllm/bin/activate
pm2 start "vllm serve Qwen/Qwen2-7B-Instruct --port 8000 --host 0.0.0.0" --name "sn35-vllm" # change port and host to your preference
  1. Run the following command to start the validator
. main/bin/activate
pm2 start python --name "sn35-validator" -- neurons/validator/validator.py \
--netuid 35 --wallet.name "wallet-name" --wallet.hotkey "wallet-hotkey" \
--subtensor.network finney \
--llm_client.base_url http://localhost:8000/v1 \ # vLLM server base url
--llm_client.model Qwen/Qwen2-7B-Instruct \ # vLLM model name
--logging.debug \ # Optional: Enable debug logging
  1. (Optional) Enable public access to the validator. Add this to the above step along with your publicly exposed port. This will enable a validator proxy.
--axon.port "your-public-open-port"