tapdb: Add Universe indices, optimize SQL queries, RWMutex for cache

[jbrill]

Universe Query Performance Optimization and Lock Safety

This PR introduces two key improvements:

1. Strategic composite indices to optimize tapdb universe queries
2. Enhanced lock safety for concurrent stats access

Lock Safety Improvements

// Added finer-grained locking for stats access
statsCacheLogger    *cacheLogger
statsRefresh        *time.Timer
eventsMtx          sync.RWMutex    // Upgraded from sync.Mutex
assetEventsCache   assetEventsCache
eventsCacheLogger  *cacheLogger
syncStatsMtx      sync.RWMutex    // Upgraded from sync.Mutex
syncStatsCache    *atomicSyncStatsCache
syncStatsRefresh  *time.Timer

Key changes:

• Upgraded mutexes to RWMutex for better concurrency
• Separated read/write lock paths in QueryAssetStatsPerDay and QuerySyncStats
• Proper lock ordering to prevent deadlocks
• Added atomic cache for sync stats

Performance Optimizations: Strategic Indexing

Key Changes

-- Optimizes universe root lookups and filtering
CREATE INDEX idx_universe_roots_composite
ON universe_roots(namespace_root, proof_type, asset_id);

-- Enhances universe leaf queries and joins
CREATE INDEX idx_universe_leaves_composite
ON universe_leaves(leaf_node_namespace, universe_root_id, leaf_node_key);

-- Improves MSSMT node lookups and aggregations 
CREATE INDEX idx_mssmt_nodes_composite 
ON mssmt_nodes(namespace, key, hash_key, sum);

Performance Impact

PostgreSQL

• Asset Stats Query: 4.58x improvement (205ms → 44.8ms)

  • Query cost reduced from 49.17 to 23.58
  • Eliminated sequence scans
  • Optimized sort operations and hash joins

• Universe Leaves Query: 1.24x improvement (20.9ms → 16.8ms)

  • Query cost reduced from 59.91 to 20.99
  • Replaced sequence scans with efficient index usage

• Root Query: 1.05x improvement (17.7ms → 16.9ms)

  • Maintained efficient lookups with better join patterns

SQLite

• Asset Stats Query: 2.02x improvement (55.9ms → 27.7ms)

  • Replaced BLOOM FILTER with covering index
  • More efficient join strategy

• Universe Leaves and Root Queries: Maintain baseline performance

  • Slight overhead (~5%) from index maintenance
  • Already optimized by SQLite's automatic indexing

Write Performance

• Single inserts: <5% overhead
• Batch inserts (100+): 10-15% overhead
• Storage impact: ~25% additional space across all indices

Technical Details

Query Cost Analysis

PostgreSQL shows significant cost reductions:

• Complex aggregations: 52% reduction in total cost
• Join operations: 65% reduction in materialization overhead
• Sort operations: ~50% reduction in cost

The improvements come from:

• Better join order selection through index availability
• Reduced materialization steps
• Elimination of table scans
• More efficient sort operations through index coverage

Implementation Considerations

• Indices designed to balance read/write performance
• Column ordering optimized for common query patterns
• Careful consideration of storage overhead vs performance gains
• Support for both PostgreSQL and SQLite query planners

Testing

• Comprehensive benchmarks across both databases
• Test dataset: 100 assets, 300 leaves per tree
• 100 iterations per query type
• Full query plan analysis for optimization verification

Next Steps

1. Monitor index usage patterns in production
2. Gather metrics on maintenance overhead
3. Consider database-specific optimizations
4. Evaluate impact on write-heavy workflows
5. Monitor lock contention in production

The PR delivers two key improvements: significantly better query performance through strategic indexing and enhanced concurrency safety through improved locking patterns. Both changes maintain reasonable overhead while providing substantial benefits for production deployments.

[coveralls]

Pull Request Test Coverage Report for Build 11937864141

Warning: This coverage report may be inaccurate.

This pull request's base commit is no longer the HEAD commit of its target branch. This means it includes changes from outside the original pull request, including, potentially, unrelated coverage changes.

• For more information on this, see Tracking coverage changes with pull request builds.
• To avoid this issue with future PRs, see these Recommended CI Configurations.
• For a quick fix, rebase this PR at GitHub. Your next report should be accurate.

Details

• 5 of 5 (100.0%) changed or added relevant lines in 1 file are covered.
• 19 unchanged lines in 6 files lost coverage.
• Overall coverage increased (+0.04%) to 41.275%

---

Files with Coverage Reduction	New Missed Lines	%
tappsbt/create.go	2	53.22%
tapdb/addrs.go	2	79.14%
commitment/tap.go	2	84.17%
asset/asset.go	2	80.96%
tapchannel/aux_leaf_signer.go	3	36.33%
universe/interface.go	8	53.68%

Totals
Change from base Build 11936061745:	0.04%
Covered Lines:	25478
Relevant Lines:	61728

---

💛 - Coveralls

[jbrill]

note: incorporating postgres functionality now for the test

[guggero]

Awesome work on analyzing and optimizing the universe tables! Did a first pass just to get some early feedback in.

[guggero]

Looks good, thanks for the updates!
We're mostly down to code style, so getting closer.

[jbrill]

@guggero thanks so much for the feedback. I've made those changes, which include some sweeping changes to both the indices and the performance test.

I opted to use the EXPLAIN ANALYZE statement to get more insightful details of our indices and how they are altering our main queries.

Here is the output for postgres and sqlite.

[guggero]

Can you please squash the two middle commits so it's a bit easier to review? Going to take another look next week.

[jbrill]

Can you please squash the two middle commits so it's a bit easier to review? Going to take another look next week.

Sounds good. I've also updated the PR description to reflect the lock changes and analysis from the query plans.

[guggero]

Very nice! Extremely close, going to assign another reviewer as soon as the build tags are working (see commit I pushed).

[jbrill]

@guggero thanks for the round of feedback! I've patched that commit onto the performance test commit in this PR. I've also updated the code style based on the formatting rules doc. Thanks a bunch for this reference.

[guggero]

Nice, thanks a lot! LGTM 🎉

[GeorgeTsagk]

why not compile these queries with sqlc?

[GeorgeTsagk]

(nit, non blocking)

[guggero]

They would then be part of the "production" code, which we don't necessarily want. So I think in tests it's fine to have pure SQL, as long as it's compatible with both SQLite and Postgres.

[GeorgeTsagk]

kicking itests, probably flake