Validators Using Local Block Building Without Receiving MEV Rewards

[xiaowangba2dai]

1. Problem Description

We has detected that validators are not receiving MEV rewards and are instead using locally built blocks.

2. Logs Analysis

Lighthouse Logs:

0:38:12.403 INFO New block received                      root: 0x294131c1e4320dbe6dea0c63175b81cc84c8417b171b16f8199a51e89f13c6c3, slot: 10636989
Dec 17 20:38:13.234 INFO Prepared beacon proposer                parent_root: ..., validator: 1001548, prepare_slot: 10636990, service: beacon
Dec 17 20:38:23.230 INFO Requesting blinded header from connected builder, parent_hash: ..., pubkey: ..., slot: Slot(10636990), service: exec
Dec 17 20:38:24.232 INFO Requested blinded execution payload ..., relay_response_ms: 1001, relay_fee_recipient: request failed, service: exec
Dec 17 20:38:24.233 WARN Builder error when requesting payload ..., relay_error: HttpClient(url: http://127.0.0.1:18550/, kind: timeout, detail: operation timed out), info: falling back to local execution client, service: exec
...

Key Points:

• Timeout Errors: Lighthouse attempts to request blinded execution payload from the local MEV-Boost service (http://127.0.0.1:18550/) but encounters a timeout (operation timed out).
• Fallback to Local Execution: Due to the timeout, Lighthouse falls back to the local execution client, bypassing MEV-Boost and thus not receiving MEV rewards.

MEV-Boost Logs:

time="2024-12-18T04:38:15.587+08:00" level=info msg="http: GET / 200" duration=0.000033 method=GET path=/ status=200 version=1.8
time="2024-12-18T04:38:23.317+08:00" level=debug msg=getHeader method=getHeader parentHash=0x086f8fd8d7682acf989b896fabeb08c747ccdf311211feb18712069be5393e94 pubkey=0xa68cb645afd25506580e9bd579eb6d0c4cbc43d3178f79e3ee19e9c60443ce29298c3e6244f0ed2c597bbaf68d488121 slot=10636990 ua=Lighthouse/v5.3.0-d6ba8c3 version=1.8
time="2024-12-18T04:38:23.364+08:00" level=info msg="getHeader request start - 364 milliseconds into slot 10636990" genesisTime=1606824023 method=getHeader msIntoSlot=364 parentHash=0x086f8fd8d7682acf989b896fabeb08c747ccdf311211feb18712069be5393e94 pubkey=0xa68cb645afd25506580e9bd579eb6d0c4cbc43d3178f79e3ee19e9c60443ce29298c3e6244f0ed2c597bbaf68d488121 slot=10636990 slotTimeSec=12 slotUID=4dcbd001-5c3a-4461-b868-0f52d56aed2d ua=Lighthouse/v5.3.0-d6ba8c3 version=1.8
time="2024-12-18T04:38:23.664+08:00" level=warning msg="error making request to relay" error="HTTP error response: 429 / {\"code\":429,\"message\":\"known validator ip for current proposer, but top bid already delivered. if you believe this is mistaken, please contact us at https://t.me/ultrasoundrelay\"}" method=getHeader parentHash=0x086f8fd8d7682acf989b896fabeb08c747ccdf311211feb18712069be5393e94 pubkey=0xa68cb645afd25506580e9bd579eb6d0c4cbc43d3178f79e3ee19e9c60443ce29298c3e6244f0ed2c597bbaf68d488121 slot=10636990 slotUID=4dcbd001-5c3a-4461-b868-0f52d56aed2d ua=Lighthouse/v5.3.0-d6ba8c3 url="https://***97bbaf68d488121" version=1.8
time="2024-12-18T04:38:23.757+08:00" level=debug msg="bid received" blockHash=0x6e46cba20406d1198b2782891d04b87cf71fe9b917d30da3118545ebaf54ee08 blockNumber=21424696 method=getHeader parentHash=0x086f8fd8d7682acf989b896fabeb08c747ccdf311211feb18712069be5393e94 pubkey=0xa68cb645afd25506580e9bd579eb6d0c4cbc43d3178f79e3ee19e9c60443ce29298c3e6244f0ed2c597bbaf68d488121 slot=10636990 slotUID=4dcbd001-5c3a-4461-b868-0f52d56aed2d txRoot=0x42fb783c7d63fdec2fc0c783638805b3159de270fdf59dc9e78079e214d02daf ua=Lighthouse/v5.3.0-d6ba8c3 url="https://***d488121" value=0.037993231374191542 version=1.8
time="2024-12-18T04:38:23.757+08:00" level=debug msg="new best bid" blockHash=0x6e46cba20406d1198b2782891d04b87cf71fe9b917d30da3118545ebaf54ee08 blockNumber=21424696 method=getHeader parentHash=0x086f8fd8d7682acf989b896fabeb08c747ccdf311211feb18712069be5393e94 pubkey=0xa68cb645afd25506580e9bd579eb6d0c4cbc43d3178f79e3ee19e9c60443ce29298c3e6244f0ed2c597bbaf68d488121 slot=10636990 slotUID=4dcbd001-5c3a-4461-b868-0f52d56aed2d txRoot=0x42fb783c7d63fdec2fc0c783638805b3159de270fdf59dc9e78079e214d02daf ua=Lighthouse/v5.3.0-d6ba8c3 url="https://***baf68d488121" value=0.037993231374191542 version=1.8
time="2024-12-18T04:38:23.871+08:00" level=debug msg="bid received" blockHash=0x1cad7c58bac12caf06d71b51a74bcca6819055d2ef9bc3d9aff527e62cf1bdeb blockNumber=21424696 method=getHeader parentHash=0x086f8fd8d7682acf989b896fabeb08c747ccdf311211feb18712069be5393e94 pubkey=0xa68cb645afd25506580e9bd579eb6d0c4cbc43d3178f79e3ee19e9c60443ce29298c3e6244f0ed2c597bbaf68d488121 slot=10636990 slotUID=4dcbd001-5c3a-4461-b868-0f52d56aed2d txRoot=0xbc10e5c1c6cbdabf3c5c6b00478b56b9ccf0ce5f74c95aee374e5845ec44c005 ua=Lighthouse/v5.3.0-d6ba8c3 url="https://***af68d488121" value=0.041609602665196384 version=1.8
time="2024-12-18T04:38:23.871+08:00" level=debug msg="new best bid" blockHash=0x1cad7c58bac12caf06d71b51a74bcca6819055d2ef9bc3d9aff527e62cf1bdeb blockNumber=21424696 method=getHeader parentHash=0x086f8fd8d7682acf989b896fabeb08c747ccdf311211feb18712069be5393e94 pubkey=0xa68cb645afd25506580e9bd579eb6d0c4cbc43d3178f79e3ee19e9c60443ce29298c3e6244f0ed2c597bbaf68d488121 slot=10636990 slotUID=4dcbd001-5c3a-4461-b868-0f52d56aed2d txRoot=0xbc10e5c1c6cbdabf3c5c6b00478b56b9ccf0ce5f74c95aee374e5845ec44c005 ua=Lighthouse/v5.3.0-d6ba8c3 url="https://***bbaf68d488121" value=0.041609602665196384 version=1.8
time="2024-12-18T04:38:23.917+08:00" level=debug msg="bid received" blockHash=0x388f443eab7975ad66e548e86c26937e378a17499c7d831862222f2926f2f647 blockNumber=21424696 method=getHeader parentHash=0x086f8fd8d7682acf989b896fabeb08c747ccdf311211feb18712069be5393e94 pubkey=0xa68cb645afd25506580e9bd579eb6d0c4cbc43d3178f79e3ee19e9c60443ce29298c3e6244f0ed2c597bbaf68d488121 slot=10636990 slotUID=4dcbd001-5c3a-4461-b868-0f52d56aed2d txRoot=0xa9f2b3096d11601df93c36a9b254156539b480b24333ce87e30dd7a8477ac454 ua=Lighthouse/v5.3.0-d6ba8c3 url="https://***af68d488121" value=0.041764803016680768 version=1.8
time="2024-12-18T04:38:23.917+08:00" level=debug msg="new best bid" blockHash=0x388f443eab7975ad66e548e86c26937e378a17499c7d831862222f2926f2f647 blockNumber=21424696 method=getHeader parentHash=0x086f8fd8d7682acf989b896fabeb08c747ccdf311211feb18712069be5393e94 pubkey=0xa68cb645afd25506580e9bd579eb6d0c4cbc43d3178f79e3ee19e9c60443ce29298c3e6244f0ed2c597bbaf68d488121 slot=10636990 slotUID=4dcbd001-5c3a-4461-b868-0f52d56aed2d txRoot=0xa9f2b3096d11601df93c36a9b254156539b480b24333ce87e30dd7a8477ac454 ua=Lighthouse/v5.3.0-d6ba8c3 url="https://***68d488121" value=0.041764803016680768 version=1.8
time="2024-12-18T04:38:23.938+08:00" level=debug msg="bid received" blockHash=0x5a5b576e282525c47dc8a539529024dc8670a9978c0a0e34a811b2905bfcee21 blockNumber=21424696 method=getHeader parentHash=0x086f8fd8d7682acf989b896fabeb08c747ccdf311211feb18712069be5393e94 pubkey=0xa68cb645afd25506580e9bd579eb6d0c4cbc43d3178f79e3ee19e9c60443ce29298c3e6244f0ed2c597bbaf68d488121 slot=10636990 slotUID=4dcbd001-5c3a-4461-b868-0f52d56aed2d txRoot=0x8029e6b0cf3a7f9088d6a7411441e07d6a0abad9c13c3a27d1b47d626e6877b6 ua=Lighthouse/v5.3.0-d6ba8c3 url="https://***44f0ed2c597bbaf68d488121" value=0.041746156248760765 version=1.8
time="2024-12-18T04:38:24.181+08:00" level=debug msg="bid received" blockHash=0x7a49475a198b8b9333034859a1fe2e748d1d8b3ac4bca9797989e85aa51ebb9d blockNumber=21424696 method=getHeader parentHash=0x086f8fd8d7682acf989b896fabeb08c747ccdf311211feb18712069be5393e94 pubkey=0xa68cb645afd25506580e9bd579eb6d0c4cbc43d3178f79e3ee19e9c60443ce29298c3e6244f0ed2c597bbaf68d488121 slot=10636990 slotUID=4dcbd001-5c3a-4461-b868-0f52d56aed2d txRoot=0xba660ef29f3885e6250a365f1b88cfdc8fd2f9118c0382658deda04b18c53dd9 ua=Lighthouse/v5.3.0-d6ba8c3 url="https://***597bbaf68d488121" value=0.038354287159174889 version=1.8
time="2024-12-18T04:38:24.185+08:00" level=debug msg="bid received" blockHash=0x7a49475a198b8b9333034859a1fe2e748d1d8b3ac4bca9797989e85aa51ebb9d blockNumber=21424696 method=getHeader parentHash=0x086f8fd8d7682acf989b896fabeb08c747ccdf311211feb18712069be5393e94 pubkey=0xa68cb645afd25506580e9bd579eb6d0c4cbc43d3178f79e3ee19e9c60443ce29298c3e6244f0ed2c597bbaf68d488121 slot=10636990 slotUID=4dcbd001-5c3a-4461-b868-0f52d56aed2d txRoot=0xba660ef29f3885e6250a365f1b88cfdc8fd2f9118c0382658deda04b18c53dd9 ua=Lighthouse/v5.3.0-d6ba8c3 url="https://***bbaf68d488121" value=0.038354287159174889 version=1.8
time="2024-12-18T04:38:24.219+08:00" level=debug msg="bid received" blockHash=0x6e46cba20406d1198b2782891d04b87cf71fe9b917d30da3118545ebaf54ee08 blockNumber=21424696 method=getHeader parentHash=0x086f8fd8d7682acf989b896fabeb08c747ccdf311211feb18712069be5393e94 pubkey=0xa68cb645afd25506580e9bd579eb6d0c4cbc43d3178f79e3ee19e9c60443ce29298c3e6244f0ed2c597bbaf68d488121 slot=10636990 slotUID=4dcbd001-5c3a-4461-b868-0f52d56aed2d txRoot=0x42fb783c7d63fdec2fc0c783638805b3159de270fdf59dc9e78079e214d02daf ua=Lighthouse/v5.3.0-d6ba8c3 url="https://***af68d488121" value=0.037993231374191542 version=1.8
time="2024-12-18T04:38:24.318+08:00" level=warning msg="error making request to relay" error="Get \"https://deadline exceeded (Client.Timeout exceeded while awaiting headers)" method=getHeader parentHash=0x086f8fd8d7682acf989b896fabeb08c747ccdf311211feb18712069be5393e94 pubkey=0xa68cb645afd25506580e9bd579eb6d0c4cbc43d3178f79e3ee1***baf68d488121\": context 9e9c60443ce29298c3e6244f0ed2c597bbaf68d488121 slot=10636990 slotUID=4dcbd001-5c3a-4461-b868-0f52d56aed2d ua=Lighthouse/v5.3.0-d6ba8c3 url="https://m***f68d488121" version=1.8
time="2024-12-18T04:38:24.32+08:00" level=info msg="best bid" blockHash=0x388f443eab7975ad66e548e86c26937e378a17499c7d831862222f2926f2f647 blockNumber=21424696 method=getHeader parentHash=0x086f8fd8d7682acf989b896fabeb08c747ccdf311211feb18712069be5393e94 pubkey=0xa68cb645afd25506580e9bd579eb6d0c4cbc43d3178f79e3ee19e9c60443ce29298c3e6244f0ed2c597bbaf68d488121 relays="https://***n.com" slot=10636990 slotUID=4dcbd001-5c3a-4461-b868-0f52d56aed2d txRoot=0xa9f2b3096d11601df93c36a9b254156539b480b24333ce87e30dd7a8477ac454 ua=Lighthouse/v5.3.0-d6ba8c3 value=0.041764803016680768 version=1.8
time="2024-12-18T04:38:24.324+08:00" level=info msg="http: GET /eth/v1/builder/header/10636990/0x086f8fd8d7682acf989b896fabeb08c747ccdf311211feb18712069be5393e94/0xa68cb645afd25506580e9bd579eb6d0c4cbc43d3178f79e3ee19e9c60443ce29298c3e6244f0ed2c597bbaf68d488121 200" duration=1.093066 method=GET path=/eth/v1/builder/header/10636990/0x086f8fd8d7682acf989b896fabeb08c747ccdf311211feb18712069be5393e94/0xa68cb645afd25506580e9bd579eb6d0c4cbc43d3178f79e3ee19e9c60443ce29298c3e6244f0ed2c597bbaf68d488121 status=200 version=1.8

3. Request for Community Assistance

We seek assistance from the MEV-Boost community to:

1. Beacon to MEV-Boost Request Time
   There is a discrepancy in request times between Beacon and MEV-Boost, potentially indicating underlying latency issues.
   20:38:23.230 - (Lighthouse) Requesting blinded header from connected builder
   20:38:23.317 - (Mev-boost) method=getHeader

2. MEV-Boost Log Gaps
   Short intervals between MEV-Boost logs suggest rapid processing, but overall latency remains an issue..
   20:38:23.317 - (Mev-boost) method=getHeader
   20:38:23.364 - (Mev-boost) getHeader request start

Thank you for your support and assistance in resolving this matter.