add: refactor backtesting to a different fork #281
Conversation
This commit updates backtesting to be executed in a different fork, it allows to start up several different executions independently, either by different tabs or even different pairs at the same time. In the future, this process could be persisted in database with the intention of running long backtestings of years of information on several dozen pairs. The exclusive locking had to be disabled so that each fork also has access to transact with the database. It would be interesting to move it to a Postgres, including a potential docker file
| pairs.forEach(pair => { | ||
| backtestPendingPairs[key].push(pair); | ||
|
|
||
| const forked = fork('src/command/backtest.js'); |
There was a problem hiding this comment.
just call the code directly instead of forking.
"LOCKING_MODE = EXCLUSIVE" is needed for performance reason.
There was a problem hiding this comment.
If you call the code directly it doesn't benefit from parallel execution for various pairs at the same time, right? Also, the UI was struggling when a backtest was configured for various thousand hours
There was a problem hiding this comment.
Then it should have some sort of lock or limiter, if this is not handled carefully it may induce too much load onto the server and cause a heap overflow.
There was a problem hiding this comment.
Good point, I'm completely unaware of that, I'll dig a bit into it to see if I can figure out how to implement it neatly, thanks!
There was a problem hiding this comment.
You can setup a queue that consists out of an array holding each job.
That would be easiest, to just pop each job after the prior got done.
Main benefit of the array would be that the jobs just die away once you kill the server
This commit updates backtesting to be executed in a different fork, it allows to start up several different executions independently, either by different tabs or even different pairs at the same time.
In the future, this process could be persisted in database with the intention of running long backtestings of years of information on several dozen pairs.
The exclusive locking had to be disabled so that each fork also has access to transact with the database. It would be interesting to move it to a Postgres, including a potential docker file