Towards independent modules #510
Closed
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Run the analysis/reporting modules from qmstrctl analyze / qmstrctl report commands. Modules now can run outside from the master server.
Add function to ping the server that the current phase has finished.
This channel will be used by the analyzers. The analyzers will wait for the signal to be closed to start their analysis.
Return the whole config declaration for the specific analyzer
Close analysis phase once all the analyzers have finished with their processes.
Shutdown analyzers gracefully after they finish with their processes or in the case of an error. Also they notify the master server.
* Create signal channel to start the reporters * Count running reporters and notify master server once they have finished
Modules get connected to the running master server.
With the use of sync.WaitGroup we wait for the goroutines to finish before exiting the code.
…ndependantModules2
This was
linked to
issues
Aug 4, 2020
fullsushidev
reviewed
Aug 6, 2020
| // wait until all modules have finished | ||
| ModulesAreDone = make(chan struct{}) | ||
| log.Printf("Waiting for modules to finish.. \n") | ||
| <-ModulesAreDone // <-- THIS MAY NOT WORK!!! Select{} |
There was a problem hiding this comment.
need help to understand
| flag.Int32Var(&anaID, "rid", -1, "unique reporter id") | ||
| var rprID int32 | ||
| CountReporters++ | ||
| flag.StringVar(&serviceAddress, "rserv", "", "connect to reporting service address") |
There was a problem hiding this comment.
add a TODO flag for the DB address
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
closes #498 #485
Description
This PR changes the way that modules interact with qmstr.
Before, the master server was starting the modules when we were switching to the specific phase. A prerequisite was the modules to be present in the master container in order to start them. Since we want to change this monolithic approach, we needed to change the way that modules are interacting.
With this PR modules can be initialized independently and they will be waiting for a signal channel to start their processes.
For instance, scancode-module dependencies are in a dedicated image which is called and started in our k8s job.
The module will be waiting until the master server has changed to the analysis phase. Once we have switched to the analysis phase, a signal channel will be sent to all the analyzer modules. Once all the modules are done the master will get notified and continue with the next phase.
Tasks to make this PR work:
Rebase over wip/christoph/k8s-mvn
Move module's dependencies in dedicated docker images
Add the modules in the k8s job. Start with scancode-module. Pull the scancode-module image, and run the module.
Run the job and debug.
Notes
dependes on #477 #503 #499