Panic after enabling txindex

[infertux]

Version: compiled at commit 71569b6

Steps to reproduce:

1. Spin up a new bchd node with the default config file
2. Let it fully sync
3. Shut it down
4. Add txindex=1 in config file
5. Restart bchd

Crash after the index creation is complete:

2021-03-31 18:17:40.993 [INF] INDX: Indexed 116 blocks in the last 10.06s (280186 transactions, height 681119/681136 (100.00%), 2021-03-31 11:06:54 +0000 UTC)
2021-03-31 18:17:42.455 [INF] INDX: Indexes caught up to height 681136
[...]
2021-03-31 18:17:50.307 [INF] CMGR: 40 addresses found from DNS seed seed.bch.loping.net                                                                                                                          
panic: runtime error: invalid memory address or nil pointer dereference [recovered]                                                                          
        panic: runtime error: invalid memory address or nil pointer dereference                                                                              
[signal SIGSEGV: segmentation violation code=0x1 addr=0x38 pc=0x93cb0c]                                                                                                                                           
                                                                                                                                                                                                                  
goroutine 51 [running]:                                                                                                                                                                                           
github.com/gcash/bchd/database/ffldb.rollbackOnPanic(0xc00028da40)                                                                                                                                                
        /root/go/src/github.com/gcash/bchd/database/ffldb/db.go:1903 +0x77                                                                                                                                        
panic(0xd0fa80, 0x15519d0)                                                                                                                                                                                        
        /usr/lib/go-1.15/src/runtime/panic.go:969 +0x1b9                                                                                                                                                          
github.com/gcash/bchd/database/ffldb.(*blockStore).writeData(0xc000140aa0, 0xc024f22b90, 0x4, 0x4, 0xe341bd, 0x11, 0xc000102780, 0xc01587b8f0)               
        /root/go/src/github.com/gcash/bchd/database/ffldb/blockio.go:387 +0x4c                                                                                                                                    
github.com/gcash/bchd/database/ffldb.(*blockStore).writeBlock(0xc000140aa0, 0xc02faf6000, 0x67ec66, 0x67ec66, 0xa64b1, 0x0, 0x0, 0x0, 0x0)                                                                        
        /root/go/src/github.com/gcash/bchd/database/ffldb/blockio.go:433 +0x10a                                                                              
github.com/gcash/bchd/database/ffldb.(*transaction).writePendingAndCommit(0xc00028da40, 0xc00028da40, 0xc01aae46e0)                                                                                               
        /root/go/src/github.com/gcash/bchd/database/ffldb/db.go:1685 +0x439                                                                                                                                       
github.com/gcash/bchd/database/ffldb.(*transaction).Commit(0xc00028da40, 0x0, 0x0)                                                                                                                                
        /root/go/src/github.com/gcash/bchd/database/ffldb/db.go:1758 +0x9e                                                                                   
github.com/gcash/bchd/database/ffldb.(*db).Update(0xc00014e000, 0xc01a887000, 0x0, 0x0)                                                                      
        /root/go/src/github.com/gcash/bchd/database/ffldb/db.go:1970 +0x10a                                                                                  
github.com/gcash/bchd/blockchain.(*BlockChain).maybeAcceptBlock(0xc0002fbdc0, 0xc015e60140, 0xc00007d404, 0xc00da95650, 0x4, 0x0, 0x0, 0x9dc1384701ec0532)  
        /root/go/src/github.com/gcash/bchd/blockchain/accept.go:55 +0x122                                                                                                                                         
github.com/gcash/bchd/blockchain.(*BlockChain).ProcessBlock(0xc0002fbdc0, 0xc015e60140, 0xc000000004, 0xc015520000, 0x0, 0x0)                                                                                     
        /root/go/src/github.com/gcash/bchd/blockchain/process.go:218 +0x285                                                                                                                                       
github.com/gcash/bchd/netsync.(*SyncManager).handleBlockMsg(0xc018d2e000, 0xc00e00f040)                                                                                                                           
        /root/go/src/github.com/gcash/bchd/netsync/manager.go:843 +0x245                                                                                     
github.com/gcash/bchd/netsync.(*SyncManager).blockHandler(0xc018d2e000)                                                                                                                                           
        /root/go/src/github.com/gcash/bchd/netsync/manager.go:1492 +0x67b                                                                                   
created by github.com/gcash/bchd/netsync.(*SyncManager).Start                                                                                                                                                     
        /root/go/src/github.com/gcash/bchd/netsync/manager.go:1752 +0xd7

Now it keeps crashing every time I start it. I tried to start it with --reindexchainstate but it's crashing as well.

[infertux]

The error I'm getting with --reindexchainstate is different though. It gets killed by the OS because it wants to use more than my 8 GB of RAM:

./bchd --reindexchainstate                                                                                                                                                                            
2021-04-01 03:02:53.656 [INF] BCHD: Version 0.17.1
2021-04-01 03:02:53.656 [INF] BCHD: Loading block database from '/var/lib/bchd/.bchd/data/mainnet/blocks_ffldb'                                                                                                   
2021-04-01 03:02:53.834 [INF] BCHD: Block database loaded
2021-04-01 03:02:53.839 [INF] INDX: Transaction index is enabled
2021-04-01 03:02:53.839 [INF] INDX: Committed filter index is enabled
2021-04-01 03:02:53.849 [INF] CHAN: Loading block index...
2021-04-01 03:02:57.344 [INF] INDX: Catching up indexes from height 680578 to 681136
2021-04-01 03:03:07.449 [INF] INDX: Indexed 132 blocks in the last 10.1s (330377 transactions, height 680710/681136 (99.94%), 2021-03-28 12:08:42 +0000 UTC)                                                      
2021-04-01 03:03:17.739 [INF] INDX: Indexed 127 blocks in the last 10.29s (321919 transactions, height 680837/681136 (99.96%), 2021-03-29 09:40:41 +0000 UTC)                                                     
2021-04-01 03:03:27.869 [INF] INDX: Indexed 105 blocks in the last 10.12s (305718 transactions, height 680942/681136 (99.97%), 2021-03-30 06:14:01 +0000 UTC)                                                     
2021-04-01 03:03:37.954 [INF] INDX: Indexed 114 blocks in the last 10.08s (299685 transactions, height 681056/681136 (99.99%), 2021-03-31 02:06:16 +0000 UTC)                                                     
2021-04-01 03:03:45.163 [INF] INDX: Indexes caught up to height 681136
2021-04-01 03:03:51.311 [WRN] CHAN: Unknown block versions are being mined, so new rules might be in effect.  Are you running the latest version of the software?                                                 
2021-04-01 03:03:51.312 [INF] CHAN: Re-indexing UTXO set from disk. This will take a while...
2021-04-01 03:03:51.312 [INF] CHAN: Deleting UTXO database bucket...
Killed

This seems like another bug - why would it need so much RAM to delete something on the disk?

[infertux]

Duplicate of #401?

[zquestz]

Brutal. We will debug this. Should be pretty easy to replicate.

[jcramer]

@infertux has this issue happened for you more than once? I haven't been able to reproduce on a testnet, and haven't had time to try it on mainnet.

[zquestz]

I am dropping indexes on my node, and recreating to see if I can replicate the issue.

[infertux]

has this issue happened for you more than once? I haven't been able to reproduce on a testnet, and haven't had time to try it on mainnet.

@jcramer It happened on a brand new node. My other bchd nodes are not experiencing this issue but they are running a stable release (as opposed to commit 71569b6).

I am dropping indexes on my node, and recreating to see if I can replicate the issue.

@zquestz I'm doing this as well right now and will report back...

[infertux]

@zquestz So bchd --droptxindex has completed successfully but restarting bchd (without txindex) raises the same error (panic: runtime error: invalid memory address or nil pointer dereference [recovered]).

[infertux]

To recap:

1. the initial block download went fine
2. adding txindex corrupted the DB
3. dropping indexes doesn't fix it

Looks like I need to resync the DB from scratch...

[zquestz]

I had no problems dropping the indexes, then rebuilding them from scratch. My node is happily running right now. Sorry, I couldn't reproduce. =\

[infertux]

@zquestz Ok thanks for looking into it. Too bad we cannot reproduce it consistently but it looks like the same bug as #401 looking at the error message so I'm gonna close this issue as duplicate.