Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

*: add invocations to applicationlog #3569

Open
wants to merge 5 commits into
base: master
Choose a base branch
from
Open

*: add invocations to applicationlog #3569

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
d8a0d13
Store invocations in application log
ixje Jun 28, 2024
04c71f5
rename params to arguments
ixje Sep 2, 2024
dfd5c9b
make invocation saving configurable, safely handle args
ixje Oct 31, 2024
cd7caf5
re-use serialized arguments for storing
ixje Oct 31, 2024
e7063a2
feedback
ixje Nov 20, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions pkg/config/ledger_config.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -19,6 +19,8 @@ type Ledger struct {
// SkipBlockVerification allows to disable verification of received
// blocks (including cryptographic checks).
SkipBlockVerification bool `yaml:"SkipBlockVerification"`
// SaveInvocations enables contract smart contract invocation data saving.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

s/contract smart contract/smart contract

SaveInvocations bool `yaml:"SaveInvocations"`
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please, adjust docs/node-configuration.md. Write some details on how this feature works and recommendations on situations when it should be enabled/disabled.

}

// Blockchain is a set of settings for core.Blockchain to use, it includes protocol
Expand Down
1 change: 1 addition & 0 deletions pkg/core/blockchain.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1717,6 +1717,7 @@ func (bc *Blockchain) storeBlock(block *block.Block, txpool *mempool.Pool) error
Stack: v.Estack().ToArray(),
Events: systemInterop.Notifications,
FaultException: faultException,
Invocations: systemInterop.InvocationCalls,
},
}
appExecResults = append(appExecResults, aer)
Expand Down
21 changes: 19 additions & 2 deletions pkg/core/dao/dao_test.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -180,8 +180,25 @@ func TestStoreAsTransaction(t *testing.T) {
Container: hash,
Execution: state.Execution{
Trigger: trigger.Application,
Events: []state.NotificationEvent{},
Stack: []stackitem.Item{},
Events: []state.NotificationEvent{
{
ScriptHash: util.Uint160{},
Name: "fakeTransferEvent",
Item: stackitem.NewArray([]stackitem.Item{
stackitem.NewBool(false),
}),
},
},
Stack: []stackitem.Item{},
Invocations: []state.ContractInvocation{{
Hash: util.Uint160{},
Method: "fakeMethodCall",
Arguments: stackitem.NewArray([]stackitem.Item{
stackitem.NewBool(false),
}),
ArgumentsCount: 1,
Truncated: false,
}},
},
}
err := dao.StoreAsTransaction(tx, 0, aer)
Expand Down
31 changes: 17 additions & 14 deletions pkg/core/interop/context.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -63,13 +63,15 @@ type Context struct {
VM *vm.VM
Functions []Function
Invocations map[util.Uint160]int
InvocationCalls []state.ContractInvocation
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

s/InvocationCalls/InvocationTree

It was unclear to me based on #3569 (comment) if I should have made it a tree or keep it flat.

Regarding this: if it's not hard, then let's make it a proper tree. It may be beneficial for some situations. AFAIU, Roman agrees with it.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Discussed with Roman, the final solution is to keep it flat. It's more simple way and right now there's no real demand in the tree structure. But it's important to properly document this behaviour in the ./docs/rpc.md

@ixje, sorry for misleading comment.

cancelFuncs []context.CancelFunc
getContract func(*dao.Simple, util.Uint160) (*state.Contract, error)
baseExecFee int64
baseStorageFee int64
loadToken func(ic *Context, id int32) error
GetRandomCounter uint32
signers []transaction.Signer
SaveInvocations bool
}

// NewContext returns new interop context.
Expand All @@ -80,20 +82,21 @@ func NewContext(trigger trigger.Type, bc Ledger, d *dao.Simple, baseExecFee, bas
dao := d.GetPrivate()
cfg := bc.GetConfig().ProtocolConfiguration
return &Context{
Chain: bc,
Network: uint32(cfg.Magic),
Hardforks: cfg.Hardforks,
Natives: natives,
Trigger: trigger,
Block: block,
Tx: tx,
DAO: dao,
Log: log,
Invocations: make(map[util.Uint160]int),
getContract: getContract,
baseExecFee: baseExecFee,
baseStorageFee: baseStorageFee,
loadToken: loadTokenFunc,
Chain: bc,
Network: uint32(cfg.Magic),
Hardforks: cfg.Hardforks,
Natives: natives,
Trigger: trigger,
Block: block,
Tx: tx,
DAO: dao,
Log: log,
Invocations: make(map[util.Uint160]int),
getContract: getContract,
baseExecFee: baseExecFee,
baseStorageFee: baseStorageFee,
loadToken: loadTokenFunc,
SaveInvocations: bc.GetConfig().SaveInvocations,
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Move a call to bc.GetConfig to the upper level (before cfg := bc.GetConfig().ProtocolConfiguration), reuse it for cfg retrieval and for SaveInvocations retrieval.

}
}

Expand Down
19 changes: 19 additions & 0 deletions pkg/core/interop/contract/call.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -69,6 +69,25 @@ func Call(ic *interop.Context) error {
return fmt.Errorf("method not found: %s/%d", method, len(args))
}
hasReturn := md.ReturnType != smartcontract.VoidType

if ic.SaveInvocations {
var (
arrCount = len(args)
truncated = false
argBytes []byte
)
if argBytes, err = ic.DAO.GetItemCtx().Serialize(stackitem.NewArray(args), false); err != nil {
truncated = true
}

ic.InvocationCalls = append(ic.InvocationCalls, state.ContractInvocation{
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We're missing the code responsible for reverted execution. We should either revert invocations along with context unloading if exception has been thrown (exactly the same way as it's done for ic.Notifications) or at least we need to set a proper status (ref. #3569 (comment)).

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Discussed with Roman, let's now keep more simple version that does not revert invocation records. So just keep every record even if transaction is FAULTed or if execution is reverted due to exception. This behaviour diverges from what we have for Notifications, but it's OK for the initial prototype. This behaviour also needs to be properly documented at the ./docs/rpc.md.

Hash: u,
Method: method,
ArgumentsBytes: argBytes,
ArgumentsCount: uint32(arrCount),
Truncated: truncated,
})
}
return callInternal(ic, cs, method, fs, hasReturn, args, true)
}

Expand Down
104 changes: 98 additions & 6 deletions pkg/core/state/notification_event.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -20,6 +20,79 @@ type NotificationEvent struct {
Item *stackitem.Array `json:"state"`
}

// ContractInvocation contains method call information.
// The Arguments field will be nil if serialization of the arguments exceeds a predefined limit
// (for security reasons). In that case Truncated will be set to true.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please, preserve the overall common line width, see for example documentation of other methods.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's a documentation, we need presize number for the predefined limit to be specified. Use a link to the constant.

type ContractInvocation struct {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Move it to a separate go file. notification_event.go is for NotificationEvent structure.

Hash util.Uint160 `json:"contracthash"`
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

s/contracthash/contract

We need Executions to be unified with Notifications.

Method string `json:"method"`
Arguments *stackitem.Array `json:"arguments"`
ArgumentsBytes []byte
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's a dangerous pattern. Sometimes only ArgumentsBytes is set, and sometimes Arguments is set. Whereas in fact at the interop context level you only use ArgumentsBytes.It's the RPC server and RPC client who uses Arguments. Ideal solution would be to split core-side and RPC-side Invocation structure into two, but it will lead to the fact that the whole RPC-side result.ApplicationLog has to be rewritten. We can't do that, because earlier we intentionally made this structure reuse as much from the state package as possible. Hence, I suggest:

  1. Make ArgumentsBytes private, document Arguments/argumentsBytes differences properly.
  2. Create a constructor for ContractInvocation that accepts arguments bytes.
  3. Move arguments decoding to the JSON serialization/deserialization level. Don't decode arguments at the binary encoding level, DB is OK with encoded argumentsBytes representation.

ArgumentsCount uint32 `json:"argumentscount"`
Truncated bool `json:"truncated"`
}

func (ci *ContractInvocation) DecodeBinary(r *io.BinReader) {
ci.Hash.DecodeBinary(r)
ci.Method = r.ReadString()
ci.ArgumentsBytes = r.ReadVarBytes()
si, err := stackitem.Deserialize(ci.ArgumentsBytes)
if err != nil {
return
}
ci.Arguments = si.(*stackitem.Array)
ci.ArgumentsCount = r.ReadU32LE()
ci.Truncated = r.ReadBool()
}

func (ci *ContractInvocation) EncodeBinaryWithContext(w *io.BinWriter, sc *stackitem.SerializationContext) {
ci.Hash.EncodeBinary(w)
w.WriteString(ci.Method)
w.WriteVarBytes(ci.ArgumentsBytes)
w.WriteU32LE(ci.ArgumentsCount)
w.WriteBool(ci.Truncated)
}

// MarshalJSON implements the json.Marshaler interface.
func (ci ContractInvocation) MarshalJSON() ([]byte, error) {
item, err := stackitem.ToJSONWithTypes(ci.Arguments)
if err != nil {
item = []byte(fmt.Sprintf(`"error: %v"`, err))
}
return json.Marshal(ContractInvocationAux{
Hash: ci.Hash,
Method: ci.Method,
Arguments: item,
ArgumentsCount: ci.ArgumentsCount,
Truncated: ci.Truncated,
})
}

// UnmarshalJSON implements the json.Unmarshaler interface.
func (ci *ContractInvocation) UnmarshalJSON(data []byte) error {
aux := new(ContractInvocationAux)
if err := json.Unmarshal(data, aux); err != nil {
return err
}
params, err := stackitem.FromJSONWithTypes(aux.Arguments)
if err != nil {
return err
}
if t := params.Type(); t != stackitem.ArrayT {
return fmt.Errorf("failed to convert invocation state of type %s to array", t.String())
}
ci.Arguments = params.(*stackitem.Array)
ci.Method = aux.Method
ci.Hash = aux.Hash
ci.ArgumentsCount = aux.ArgumentsCount
ci.Truncated = aux.Truncated
return nil
}

func (ci *ContractInvocation) EncodeBinary(w *io.BinWriter) {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Exported method needs comment.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please, check it everywhere.

ci.EncodeBinaryWithContext(w, stackitem.NewSerializationContext())
}

// AppExecResult represents the result of the script execution, gathering together
// all resulting notifications, state, stack and other metadata.
type AppExecResult struct {
Expand Down Expand Up @@ -95,6 +168,10 @@ func (aer *AppExecResult) EncodeBinaryWithContext(w *io.BinWriter, sc *stackitem
aer.Events[i].EncodeBinaryWithContext(w, sc)
}
w.WriteVarBytes([]byte(aer.FaultException))
w.WriteVarUint(uint64(len(aer.Invocations)))
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You need to check if it's nil and encode only if it's not nil. It will save a single byte for cases when the node is running with SaveInvocations off.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Regarding DB compatibility: I'm thinking about adding SaveInvocations to dao.Version because: if the node was running with SaveInvocatoins: true and then it was switched to SaveInvocations: false, then currently there's no way from the user side to distinguish missing invocatoins from empty invocations, and it seems like a broken DB for me. I propose to explicitly return an error from the (bc *Blockchain) init() if SaveInvocations was changed without DB resync.

@roman-khimov, ACK?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Makes sense, follows https://github.com/nspcc-dev/neo-go/blob/master/docs/node-configuration.md#db-compatibility (don't forget to update it).

for i := range aer.Invocations {
aer.Invocations[i].EncodeBinaryWithContext(w, sc)
}
}

// DecodeBinary implements the Serializable interface.
Expand All @@ -120,6 +197,9 @@ func (aer *AppExecResult) DecodeBinary(r *io.BinReader) {
aer.Stack = arr
r.ReadArray(&aer.Events)
aer.FaultException = r.ReadString()
if r.Len() > 0 {
r.ReadArray(&aer.Invocations)
}
}

// notificationEventAux is an auxiliary struct for NotificationEvent JSON marshalling.
Expand Down Expand Up @@ -209,16 +289,26 @@ type Execution struct {
Stack []stackitem.Item
Events []NotificationEvent
FaultException string
Invocations []ContractInvocation
}

type ContractInvocationAux struct {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Exported structure needs a comment.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

And BTW, all auxiliary structures should be unexported.

Hash util.Uint160 `json:"contracthash"`
Method string `json:"method"`
Arguments json.RawMessage `json:"arguments"`
ArgumentsCount uint32 `json:"argumentscount"`
Truncated bool `json:"truncated"`
}

// executionAux represents an auxiliary struct for Execution JSON marshalling.
type executionAux struct {
Trigger string `json:"trigger"`
VMState string `json:"vmstate"`
GasConsumed int64 `json:"gasconsumed,string"`
Stack json.RawMessage `json:"stack"`
Events []NotificationEvent `json:"notifications"`
FaultException *string `json:"exception"`
Trigger string `json:"trigger"`
VMState string `json:"vmstate"`
GasConsumed int64 `json:"gasconsumed,string"`
Stack json.RawMessage `json:"stack"`
Events []NotificationEvent `json:"notifications"`
FaultException *string `json:"exception"`
Invocations []ContractInvocation `json:"invocations"`
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We need to document this node extension at least as an RPC protocol extension until it's implemented on the C# side. Please, adjust docs/rpc.md, section ## Extensions.

}

// MarshalJSON implements the json.Marshaler interface.
Expand Down Expand Up @@ -246,6 +336,7 @@ func (e Execution) MarshalJSON() ([]byte, error) {
Stack: st,
Events: e.Events,
FaultException: exception,
Invocations: e.Invocations,
})
}

Expand Down Expand Up @@ -287,6 +378,7 @@ func (e *Execution) UnmarshalJSON(data []byte) error {
if aux.FaultException != nil {
e.FaultException = *aux.FaultException
}
e.Invocations = aux.Invocations
return nil
}

Expand Down
Loading