katzenmint: prune documents if it's not corrected #183
Conversation
katzenmint/state.go
Outdated
| @@ -383,15 +384,31 @@ func (state *KatzenmintState) GetDocument(epoch uint64, height int64) ([]byte, * | |||
| * Document * | |||
| *****************************************/ | |||
|
|
|||
| func (s *KatzenmintState) pruneDocument() { | |||
| begin := storageKey(descriptorsBucket, []byte{}, s.currentEpoch) | |||
| end := storageKey(descriptorsBucket, []byte{}, s.currentEpoch+1) | |||
There was a problem hiding this comment.
Are you sure this is the correct end calculation? Considering that we have epochs in minutes e.g. 10 mins, if you add +1 then it makes the end at 11 minutes and not at the next epoch tecnically.
There was a problem hiding this comment.
Yes, remove this function for now. Probably need to discuss how many old documents we should keep (katzenpost kept 3).
katzenmint/state.go
Outdated
| escaped := s.tree.IterateRange(begin, end, true, func(key, value []byte) (ret bool) { | ||
| desc, err := s11n.ParseDescriptorWithoutVerify(value) | ||
| if err != nil { | ||
| // TODO: check different error |
There was a problem hiding this comment.
Can you be more precise here with what needs to be the actual error and if it's something more involved and complex, you should open an issue
There was a problem hiding this comment.
The error might be corrupted data or invalid certificate (outdated/incorrect encoding). Because the network was stuck or offline (still need to find the root cause), the descriptors might be expired in local database (saved before generate document). How about we remove descriptors and wait them submit again?
There was a problem hiding this comment.
@Mikerah is this possible to generate different block hash for old blocks with expired certificate?
| } | ||
|
|
||
| adjFetchDelay() | ||
| c.log.Debugf("[Deprecated] Dequeued GetConsesus for send.") |
There was a problem hiding this comment.
If it's deprecated, why not just remove it?
| // might happened when network stuck or the data was corrupted | ||
| // should remove the outdated or corrupted data | ||
| s.tree.Remove(key) | ||
| // s.tree.Remove(key) |
There was a problem hiding this comment.
Shouldn't you do more processing here if the tree data is corrupted?
| @@ -107,6 +110,8 @@ func newTendermint(app abci.Application, config *cfg.Config, logger log.Logger) | |||
| func init() { | |||
| rootCmd.PersistentFlags().StringVar(&configFile, "config", "katzenmint.toml", "Path to katzenmint.toml") | |||
| runCmd.Flags().IntVar(&dbCacheSize, "dbcachesize", 100, "Cache size for katzenmint db") | |||
| runCmd.Flags().StringVar(&cpuProfilePath, "cpuprofilepath", "", "Path to the pprof cpu profile") | |||
There was a problem hiding this comment.
Should add that these are for debugging
Description
see #181
prune documents if it's not corrected
remove rawdoc from cache and server
TODO:
Type of change
Please delete options that are not relevant.
How Has This Been Tested?
Please describe the tests that you ran to verify your changes. Provide instructions so we can reproduce. Please also list any relevant details for your test configuration
Checklist: