Using indexes with prune/fastsync mode

[cpacia]

We have some kludgy code that stops us from using indexes with a fastsynced or pruned node. Because of this I have outright disabled indexes. See here (note this isn't the correct way or place to disable indexes either):

https://github.com/gcash/bchd/blob/master/blockchain/chain.go#L2321

Now you might ask why allow any indexes with a pruned/fastsynced node? After all why would you want an incomplete index?

Well, one of our indexes is the compact filter index which is used by neutrino nodes and in theory neutrino can still use these nodes to sync from, just not from genesis or a deep checkpoint. Currently this functionality is disabled but we would like to enable it at some point. See #143.

Also, there is talk about a address -> utxo index for use with fastsynced nodes. Which would require indexes turned on.

So why is it turned off?

Basically it's an issue with the migration code in the CFIndex. At each startup it checks to see if it should run a migration on the index. The migration basically just drops the existing index and exits. Then the remainder of the Init() code checks to see if any indexes are behind the tip of the chain. If so, it loads the blocks and indexes them to the tip. Since the migration just deleted the index it is obviously behind the tip (at genesis specifically) so this causes a reindex.

However you can see the problem... pruned and fastsynced nodes don't have the blocks going back to genesis and so when the Init() code attempts to load them, it errors and causes the node not to start. Hence why we have it disabled.

I think the solution may be to have an entry in the db where we track the hash or height of the earliest block in the chain. This likely means updating this entry every block connect for fastsynced and pruned nodes.

Then the migration can delete the index, but set the index tip (which is tracked by the index) to the earliest saved block instead of to genesis. I'm not positive, but I think that would allow for the migration to proceed without error and we can remove the code disabling indexes for pruned/fastsynced nodes.

A challenge with this approach is that we need to set the indexer tip to the block before the first block we have so that it indexes our first block. If we only track the first block then the Migrate() function will not know the previous block to set it as the indexer tip. The Migrate() function does not have any access to the chain data to calculate the prior block.