Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Panic on leveldb range query fabric peer #5048

Open
sam-at-luther opened this issue Nov 1, 2024 · 0 comments
Open

Panic on leveldb range query fabric peer #5048

sam-at-luther opened this issue Nov 1, 2024 · 0 comments
Labels

Comments

@sam-at-luther
Copy link

sam-at-luther commented Nov 1, 2024

Description

Version: 2.5.5, with external chaincode, levelDB, using custom Go chaincode.

We're seeing this fairly frequently in a busy network.

It seems to only happen when a transaction has exceeded the execution timeout duration, but is in the midst of a range query on the fabric peer.

From what I can tell, the corresponding lines in levelDB is:

func (i *indexedIterator) dataErr() bool {
	if err := i.data.Error(); err != nil {
		if i.errf != nil {
			i.errf(err)
		}
		if i.strict || !errors.IsCorrupted(err) {
			i.err = err
			return true
		}
	}
	return false
}

where

if err := i.data.Error(); err != nil {

is triggering the panic.

I suspect the peer has a read lock on level db, but in the middle of ranging, the peer hits an execution timeout, and the interruption logic it’s not correctly waiting for the iterator to close before releasing the lock. In other words, Close() is called on the iterator before it’s finished the range query.

Possibly relates to: #5001

Stack traces:

2024-10-02 10:05:27.738 UTC c30993 ERRO [endorser] simulateProposal -> failed to invoke chaincode com_luthersystems_chaincode_substrate01, error: timeout expired while executing transaction
github.com/hyperledger/fabric/core/chaincode.(*Handler).Execute
	/core/chaincode/handler.go:1234
github.com/hyperledger/fabric/core/chaincode.(*ChaincodeSupport).execute
	/core/chaincode/chaincode_support.go:278
github.com/hyperledger/fabric/core/chaincode.(*ChaincodeSupport).Invoke
	/core/chaincode/chaincode_support.go:208
github.com/hyperledger/fabric/core/chaincode.(*ChaincodeSupport).Execute
	/core/chaincode/chaincode_support.go:161
github.com/hyperledger/fabric/core/endorser.(*SupportImpl).Execute
	/core/endorser/support.go:126
github.com/hyperledger/fabric/core/endorser.(*Endorser).callChaincode
	/core/endorser/endorser.go:120
github.com/hyperledger/fabric/core/endorser.(*Endorser).simulateProposal
	/core/endorser/endorser.go:187
github.com/hyperledger/fabric/core/endorser.(*Endorser).ProcessProposalSuccessfullyOrError
	/core/endorser/endorser.go:409
github.com/hyperledger/fabric/core/endorser.(*Endorser).ProcessProposal
	/core/endorser/endorser.go:350
github.com/hyperledger/fabric/core/handlers/auth/filter.(*expirationCheckFilter).ProcessProposal
	/core/handlers/auth/filter/expiration.go:61
github.com/hyperledger/fabric/core/handlers/auth/filter.(*filter).ProcessProposal
	/core/handlers/auth/filter/filter.go:32
github.com/hyperledger/fabric-protos-go/peer._Endorser_ProcessProposal_Handler.func1
	/vendor/github.com/hyperledger/fabric-protos-go/peer/peer.pb.go:107
github.com/hyperledger/fabric/internal/peer/node.serve.unaryGrpcLimiter.func13
	/internal/peer/node/grpc_limiters.go:56
github.com/hyperledger/fabric/internal/pkg/comm.NewGRPCServerFromListener.ChainUnaryServer.func9.1.1
	/vendor/github.com/grpc-ecosystem/go-grpc-middleware/chain.go:25
github.com/hyperledger/fabric/common/grpclogging.UnaryServerInterceptor.func1
	/common/grpclogging/server.go:92
github.com/hyperledger/fabric/internal/pkg/comm.NewGRPCServerFromListener.ChainUnaryServer.func9.1.1
	/vendor/github.com/grpc-ecosystem/go-grpc-middleware/chain.go:25
github.com/hyperledger/fabric/internal/peer/node.serve.UnaryServerInterceptor.func11
	/common/grpcmetrics/interceptor.go:31
github.com/hyperledger/fabric/internal/pkg/comm.NewGRPCServerFromListener.ChainUnaryServer.func9.1.1
	/vendor/github.com/grpc-ecosystem/go-grpc-middleware/chain.go:25
github.com/hyperledger/fabric/internal/pkg/comm.NewGRPCServerFromListener.ChainUnaryServer.func9
	/vendor/github.com/grpc-ecosystem/go-grpc-middleware/chain.go:34
github.com/hyperledger/fabric-protos-go/peer._Endorser_ProcessProposal_Handler
	/vendor/github.com/hyperledger/fabric-protos-go/peer/peer.pb.go:109
google.golang.org/grpc.(*Server).processUnaryRPC
	/vendor/google.golang.org/grpc/server.go:1336
google.golang.org/grpc.(*Server).handleStream
	/vendor/google.golang.org/grpc/server.go:1704
google.golang.org/grpc.(*Server).serveStreams.func1.2
	/vendor/google.golang.org/grpc/server.go:965
runtime.goexit
	/usr/local/go/src/runtime/asm_amd64.s:1650
error sending
failed to execute transaction c078f5529e0ec5a22649101fc02db922c36e5c6b20d71e109a3aa639afaf30de
github.com/hyperledger/fabric/core/chaincode.processChaincodeExecutionResult
	/core/chaincode/chaincode_support.go:167
github.com/hyperledger/fabric/core/chaincode.(*ChaincodeSupport).Execute
	/core/chaincode/chaincode_support.go:162
github.com/hyperledger/fabric/core/endorser.(*SupportImpl).Execute
	/core/endorser/support.go:126
github.com/hyperledger/fabric/core/endorser.(*Endorser).callChaincode
	/core/endorser/endorser.go:120
github.com/hyperledger/fabric/core/endorser.(*Endorser).simulateProposal
	/core/endorser/endorser.go:187
github.com/hyperledger/fabric/core/endorser.(*Endorser).ProcessProposalSuccessfullyOrError
	/core/endorser/endorser.go:409
github.com/hyperledger/fabric/core/endorser.(*Endorser).ProcessProposal
	/core/endorser/endorser.go:350
github.com/hyperledger/fabric/core/handlers/auth/filter.(*expirationCheckFilter).ProcessProposal
	/core/handlers/auth/filter/expiration.go:61
github.com/hyperledger/fabric/core/handlers/auth/filter.(*filter).ProcessProposal
	/core/handlers/auth/filter/filter.go:32
github.com/hyperledger/fabric-protos-go/peer._Endorser_ProcessProposal_Handler.func1
	/vendor/github.com/hyperledger/fabric-protos-go/peer/peer.pb.go:107
github.com/hyperledger/fabric/internal/peer/node.serve.unaryGrpcLimiter.func13
	/internal/peer/node/grpc_limiters.go:56
github.com/hyperledger/fabric/internal/pkg/comm.NewGRPCServerFromListener.ChainUnaryServer.func9.1.1
	/vendor/github.com/grpc-ecosystem/go-grpc-middleware/chain.go:25
github.com/hyperledger/fabric/common/grpclogging.UnaryServerInterceptor.func1
	/common/grpclogging/server.go:92
github.com/hyperledger/fabric/internal/pkg/comm.NewGRPCServerFromListener.ChainUnaryServer.func9.1.1
	/vendor/github.com/grpc-ecosystem/go-grpc-middleware/chain.go:25
github.com/hyperledger/fabric/internal/peer/node.serve.UnaryServerInterceptor.func11
	/common/grpcmetrics/interceptor.go:31
github.com/hyperledger/fabric/internal/pkg/comm.NewGRPCServerFromListener.ChainUnaryServer.func9.1.1
	/vendor/github.com/grpc-ecosystem/go-grpc-middleware/chain.go:25
github.com/hyperledger/fabric/internal/pkg/comm.NewGRPCServerFromListener.ChainUnaryServer.func9
	/vendor/github.com/grpc-ecosystem/go-grpc-middleware/chain.go:34
github.com/hyperledger/fabric-protos-go/peer._Endorser_ProcessProposal_Handler
	/vendor/github.com/hyperledger/fabric-protos-go/peer/peer.pb.go:109
google.golang.org/grpc.(*Server).processUnaryRPC
	/vendor/google.golang.org/grpc/server.go:1336
google.golang.org/grpc.(*Server).handleStream
	/vendor/google.golang.org/grpc/server.go:1704
google.golang.org/grpc.(*Server).serveStreams.func1.2
	/vendor/google.golang.org/grpc/server.go:965
runtime.goexit
	/usr/local/go/src/runtime/asm_amd64.s:1650 channel=luther txID=c078f552
2024-10-02 10:05:27.738 UTC c30994 WARN [endorser] ProcessProposal -> Failed to invoke chaincode channel=luther chaincode=com_luthersystems_chaincode_substrate01 error="error in simulation: failed to execute transaction c078f5529e0ec5a22649101fc02db922c36e5c6b20d71e109a3aa639afaf30de: error sending: timeout expired while executing transaction"
2024-10-02 10:05:27.738 UTC c30995 INFO [comm.grpc.server] 1 -> unary call completed grpc.service=protos.Endorser grpc.method=ProcessProposal grpc.request_deadline=2024-10-02T10:05:29.979Z grpc.peer_address=10.0.6.175:1306 grpc.peer_subject="[email protected],ST=London,C=UK" grpc.code=OK grpc.call_duration=8.75856771s
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x18 pc=0x12b5c9f]

goroutine 1046998611 [running]:
github.com/syndtr/goleveldb/leveldb/iterator.(*indexedIterator).dataErr(0xc00da8f380)
	/vendor/github.com/syndtr/goleveldb/leveldb/iterator/indexed_iter.go:59 +0x1f
github.com/syndtr/goleveldb/leveldb/iterator.(*indexedIterator).Next(0xc00da8f380)
	/vendor/github.com/syndtr/goleveldb/leveldb/iterator/indexed_iter.go:150 +0xa5
github.com/syndtr/goleveldb/leveldb/iterator.(*indexedIterator).First(0xc00da8f380)
	/vendor/github.com/syndtr/goleveldb/leveldb/iterator/indexed_iter.go:89 +0xe5
github.com/syndtr/goleveldb/leveldb/iterator.(*mergedIterator).First(0xc00a6b7a00)
	/vendor/github.com/syndtr/goleveldb/leveldb/iterator/merged_iter.go:72 +0xb6
github.com/syndtr/goleveldb/leveldb/iterator.(*mergedIterator).Next(0xc00c86dbc0?)
	/vendor/github.com/syndtr/goleveldb/leveldb/iterator/merged_iter.go:157 +0x265
github.com/syndtr/goleveldb/leveldb.(*dbIter).Next(0xc0056ac480)
	/vendor/github.com/syndtr/goleveldb/leveldb/db_iter.go:249 +0x7e
github.com/hyperledger/fabric/core/ledger/kvledger/txmgmt/statedb/stateleveldb.(*kvScanner).Next(0xc00c86db60)
	/core/ledger/kvledger/txmgmt/statedb/stateleveldb/stateleveldb.go:313 +0x42
github.com/hyperledger/fabric/core/ledger/kvledger/txmgmt/txmgr.(*resultsItr).Next(0xc0031456d0)
	/core/ledger/kvledger/txmgmt/txmgr/query_executor.go:439 +0x28
github.com/hyperledger/fabric/core/chaincode.(*QueryResponseGenerator).BuildQueryResponse(0xc00561e068, 0xc005b01380?, {0x1d24820, 0xc0031456d0}, {0xc005b01380, 0x24}, 0x0, 0x186a0)
	/core/chaincode/query_response_generator.go:31 +0xc4
github.com/hyperledger/fabric/core/chaincode.(*Handler).HandleGetStateByRange(0xc005630360, 0xc005726d80, 0xc008cee320)
	/core/chaincode/handler.go:766 +0x40d
github.com/hyperledger/fabric/core/chaincode.(*Handler).HandleTransaction(0xc005630360, 0xc005726d80, 0xc0061e0950)
	/core/chaincode/handler.go:253 +0x373
created by github.com/hyperledger/fabric/core/chaincode.(*Handler).handleMessageReadyState in goroutine 187
	/core/chaincode/handler.go:195 +0x3e9

2024-10-08 13:41:47.064 UTC 2a47c7 ERRO [endorser] simulateProposal -> failed to invoke chaincode com_luthersystems_chaincode_substrate01, error: timeout expired while executing transaction
github.com/hyperledger/fabric/core/chaincode.(*Handler).Execute
	/core/chaincode/handler.go:1234
github.com/hyperledger/fabric/core/chaincode.(*ChaincodeSupport).execute
	/core/chaincode/chaincode_support.go:278
github.com/hyperledger/fabric/core/chaincode.(*ChaincodeSupport).Invoke
	/core/chaincode/chaincode_support.go:208
github.com/hyperledger/fabric/core/chaincode.(*ChaincodeSupport).Execute
	/core/chaincode/chaincode_support.go:161
github.com/hyperledger/fabric/core/endorser.(*SupportImpl).Execute
	/core/endorser/support.go:126
github.com/hyperledger/fabric/core/endorser.(*Endorser).callChaincode
	/core/endorser/endorser.go:120
github.com/hyperledger/fabric/core/endorser.(*Endorser).simulateProposal
	/core/endorser/endorser.go:187
github.com/hyperledger/fabric/core/endorser.(*Endorser).ProcessProposalSuccessfullyOrError
	/core/endorser/endorser.go:409
github.com/hyperledger/fabric/core/endorser.(*Endorser).ProcessProposal
	/core/endorser/endorser.go:350
github.com/hyperledger/fabric/core/handlers/auth/filter.(*expirationCheckFilter).ProcessProposal
	/core/handlers/auth/filter/expiration.go:61
github.com/hyperledger/fabric/core/handlers/auth/filter.(*filter).ProcessProposal
	/core/handlers/auth/filter/filter.go:32
github.com/hyperledger/fabric-protos-go/peer._Endorser_ProcessProposal_Handler.func1
	/vendor/github.com/hyperledger/fabric-protos-go/peer/peer.pb.go:107
github.com/hyperledger/fabric/internal/peer/node.serve.unaryGrpcLimiter.func13
	/internal/peer/node/grpc_limiters.go:56
github.com/hyperledger/fabric/internal/pkg/comm.NewGRPCServerFromListener.ChainUnaryServer.func9.1.1
	/vendor/github.com/grpc-ecosystem/go-grpc-middleware/chain.go:25
github.com/hyperledger/fabric/common/grpclogging.UnaryServerInterceptor.func1
	/common/grpclogging/server.go:92
github.com/hyperledger/fabric/internal/pkg/comm.NewGRPCServerFromListener.ChainUnaryServer.func9.1.1
	/vendor/github.com/grpc-ecosystem/go-grpc-middleware/chain.go:25
github.com/hyperledger/fabric/internal/peer/node.serve.UnaryServerInterceptor.func11
	/common/grpcmetrics/interceptor.go:31
github.com/hyperledger/fabric/internal/pkg/comm.NewGRPCServerFromListener.ChainUnaryServer.func9.1.1
	/vendor/github.com/grpc-ecosystem/go-grpc-middleware/chain.go:25
github.com/hyperledger/fabric/internal/pkg/comm.NewGRPCServerFromListener.ChainUnaryServer.func9
	/vendor/github.com/grpc-ecosystem/go-grpc-middleware/chain.go:34
github.com/hyperledger/fabric-protos-go/peer._Endorser_ProcessProposal_Handler
	/vendor/github.com/hyperledger/fabric-protos-go/peer/peer.pb.go:109
google.golang.org/grpc.(*Server).processUnaryRPC
	/vendor/google.golang.org/grpc/server.go:1336
google.golang.org/grpc.(*Server).handleStream
	/vendor/google.golang.org/grpc/server.go:1704
google.golang.org/grpc.(*Server).serveStreams.func1.2
	/vendor/google.golang.org/grpc/server.go:965
runtime.goexit
	/usr/local/go/src/runtime/asm_amd64.s:1650
error sending
failed to execute transaction f8df1d185389530537d58407709edfa8342dd8e8feed19cdea23f8e4df2256a7
github.com/hyperledger/fabric/core/chaincode.processChaincodeExecutionResult
	/core/chaincode/chaincode_support.go:167
github.com/hyperledger/fabric/core/chaincode.(*ChaincodeSupport).Execute
	/core/chaincode/chaincode_support.go:162
github.com/hyperledger/fabric/core/endorser.(*SupportImpl).Execute
	/core/endorser/support.go:126
github.com/hyperledger/fabric/core/endorser.(*Endorser).callChaincode
	/core/endorser/endorser.go:120
github.com/hyperledger/fabric/core/endorser.(*Endorser).simulateProposal
	/core/endorser/endorser.go:187
github.com/hyperledger/fabric/core/endorser.(*Endorser).ProcessProposalSuccessfullyOrError
	/core/endorser/endorser.go:409
github.com/hyperledger/fabric/core/endorser.(*Endorser).ProcessProposal
	/core/endorser/endorser.go:350
github.com/hyperledger/fabric/core/handlers/auth/filter.(*expirationCheckFilter).ProcessProposal
	/core/handlers/auth/filter/expiration.go:61
github.com/hyperledger/fabric/core/handlers/auth/filter.(*filter).ProcessProposal
	/core/handlers/auth/filter/filter.go:32
github.com/hyperledger/fabric-protos-go/peer._Endorser_ProcessProposal_Handler.func1
	/vendor/github.com/hyperledger/fabric-protos-go/peer/peer.pb.go:107
github.com/hyperledger/fabric/internal/peer/node.serve.unaryGrpcLimiter.func13
	/internal/peer/node/grpc_limiters.go:56
github.com/hyperledger/fabric/internal/pkg/comm.NewGRPCServerFromListener.ChainUnaryServer.func9.1.1
	/vendor/github.com/grpc-ecosystem/go-grpc-middleware/chain.go:25
github.com/hyperledger/fabric/common/grpclogging.UnaryServerInterceptor.func1
	/common/grpclogging/server.go:92
github.com/hyperledger/fabric/internal/pkg/comm.NewGRPCServerFromListener.ChainUnaryServer.func9.1.1
	/vendor/github.com/grpc-ecosystem/go-grpc-middleware/chain.go:25
github.com/hyperledger/fabric/internal/peer/node.serve.UnaryServerInterceptor.func11
	/common/grpcmetrics/interceptor.go:31
github.com/hyperledger/fabric/internal/pkg/comm.NewGRPCServerFromListener.ChainUnaryServer.func9.1.1
	/vendor/github.com/grpc-ecosystem/go-grpc-middleware/chain.go:25
github.com/hyperledger/fabric/internal/pkg/comm.NewGRPCServerFromListener.ChainUnaryServer.func9
	/vendor/github.com/grpc-ecosystem/go-grpc-middleware/chain.go:34
github.com/hyperledger/fabric-protos-go/peer._Endorser_ProcessProposal_Handler
	/vendor/github.com/hyperledger/fabric-protos-go/peer/peer.pb.go:109
google.golang.org/grpc.(*Server).processUnaryRPC
	/vendor/google.golang.org/grpc/server.go:1336
google.golang.org/grpc.(*Server).handleStream
	/vendor/google.golang.org/grpc/server.go:1704
google.golang.org/grpc.(*Server).serveStreams.func1.2
	/vendor/google.golang.org/grpc/server.go:965
runtime.goexit
	/usr/local/go/src/runtime/asm_amd64.s:1650 channel=luther txID=f8df1d18
2024-10-08 13:41:47.064 UTC 2a47c8 WARN [endorser] ProcessProposal -> Failed to invoke chaincode channel=luther chaincode=com_luthersystems_chaincode_substrate01 error="error in simulation: failed to execute transaction f8df1d185389530537d58407709edfa8342dd8e8feed19cdea23f8e4df2256a7: error sending: timeout expired while executing transaction"
2024-10-08 13:41:47.064 UTC 2a47c9 INFO [comm.grpc.server] 1 -> unary call completed grpc.service=protos.Endorser grpc.method=ProcessProposal grpc.request_deadline=2024-10-08T13:41:49.637Z grpc.peer_address=10.0.5.138:39365 grpc.peer_subject="[email protected],ST=London,C=UK" grpc.code=OK grpc.call_duration=8.426256026s
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x18 pc=0x12b5c9f]

goroutine 256351988 [running]:
github.com/syndtr/goleveldb/leveldb/iterator.(*indexedIterator).dataErr(0xc00c18e8a0)
	/vendor/github.com/syndtr/goleveldb/leveldb/iterator/indexed_iter.go:59 +0x1f
github.com/syndtr/goleveldb/leveldb/iterator.(*indexedIterator).Next(0xc00c18e8a0)
	/vendor/github.com/syndtr/goleveldb/leveldb/iterator/indexed_iter.go:150 +0xa5
github.com/syndtr/goleveldb/leveldb/iterator.(*indexedIterator).First(0xc00c18e8a0)
	/vendor/github.com/syndtr/goleveldb/leveldb/iterator/indexed_iter.go:89 +0xe5
github.com/syndtr/goleveldb/leveldb/iterator.(*mergedIterator).First(0xc003783480)
	/vendor/github.com/syndtr/goleveldb/leveldb/iterator/merged_iter.go:72 +0xb6
github.com/syndtr/goleveldb/leveldb/iterator.(*mergedIterator).Next(0xc00a0b23c0?)
	/vendor/github.com/syndtr/goleveldb/leveldb/iterator/merged_iter.go:157 +0x265
github.com/syndtr/goleveldb/leveldb.(*dbIter).Next(0xc007a7b170)
	/vendor/github.com/syndtr/goleveldb/leveldb/db_iter.go:249 +0x7e
github.com/hyperledger/fabric/core/ledger/kvledger/txmgmt/statedb/stateleveldb.(*kvScanner).Next(0xc00a0b2360)
	/core/ledger/kvledger/txmgmt/statedb/stateleveldb/stateleveldb.go:313 +0x42
github.com/hyperledger/fabric/core/ledger/kvledger/txmgmt/txmgr.(*resultsItr).Next(0xc00f5012c0)
	/core/ledger/kvledger/txmgmt/txmgr/query_executor.go:439 +0x28
github.com/hyperledger/fabric/core/chaincode.(*QueryResponseGenerator).BuildQueryResponse(0xc0116f0778, 0xc0079cb4a0?, {0x1d24820, 0xc00f5012c0}, {0xc0079cb4a0, 0x24}, 0x0, 0x186a0)
	/core/chaincode/query_response_generator.go:31 +0xc4
github.com/hyperledger/fabric/core/chaincode.(*Handler).HandleGetStateByRange(0xc000248a20, 0xc003783080, 0xc003eb3b80)
	/core/chaincode/handler.go:766 +0x40d
github.com/hyperledger/fabric/core/chaincode.(*Handler).HandleTransaction(0xc000248a20, 0xc003783080, 0xc007aadf50)
	/core/chaincode/handler.go:253 +0x373
created by github.com/hyperledger/fabric/core/chaincode.(*Handler).handleMessageReadyState in goroutine 71935
	/core/chaincode/handler.go:195 +0x3e9

Steps to reproduce

No response

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

1 participant