Are vulnerabilities information at the package level applied to their related callables in the stitched call graph?

[michelescarlato]

@tmortagne Hi Thomas,

I am reaching out because we are trying to understand the main logic that Maven plugin is using (to be consistent in fasten-pypi-plugin) while augmenting the metadata field at the callable level in the stitched call graph.

In a dialogue between @gdrosos Giorgos and @amphioxis Martin, I understood that they were trying to understand Maven plugin code (particularly these two parts of the code: 1 and 2).

Martin's main question is:
"Are everywhere in the metadata field of the stitched call graph added vulnerabilities information retrieved at the package level?"

Giorgos supposition is:
"Yes"

My understanding is that:
"The Maven plugin retrieves vulnerabilities information at the package level (through a FASTEN API endpoint). Then, it applies this information all over the stitched call graph (reasonably, to all the callables related to that specific package, being a stitched call graph the product of more call graphs, each of them belonging to a package)."

Can you confirm this first part of the reasoning?

More questions will probably follow, but at least we should be sure to answer this doubt to fix this first important step for the Pypi plugin development.

[tmortagne]

Vulnerability handling is a bit special: in the FASTEN Knowledge Base the vulnerabilities of each callable are aggregated in the package metadata, so the plugin is only getting package metadata for those (much faster than asking callable metadata vulnerabilities) and then the security analyzer just checks if it can find a callables in the optimized call graph for each vulnerability. So contrary to others info like the quality metrics, the vulnerabilities are never associated with callables in the generated graph itself (only to dependencies).

[michelescarlato]

It sounds interesting.
Is it possible to look at an example of a stitched call graph enriched with information by the Maven plugin ?

I think for @amphioxis it would be the best way to go, and to ask! :)

Thanks for the fast and precise clarification.

[tmortagne]

I'm not really sure what you mean exactly by "example". There is no serialization of the enriched call graph, but you can see the code which deal with that in

fasten-maven-plugin/src/main/java/eu/fasten/maven/CheckMojo.java

Line 386 in 4ed9b7c

private Set<MavenExtendedRevisionJavaCallGraph> enrichStitchedCallables()

.

[michelescarlato]

Well, we thought that Maven plugin was enriching the final stitched call graph with metadata information (vulnerabilities, quality metrics, licensing).

So we really thought the Maven plugin was generating a JSON.

We are in fact designing the PyPI plugin with a logic like this one:

1. The PyPI plugin, by means of pycg and pycg-stitch, creates locally a stitched call graph taking as input the package dependencies chosen by the developer that will use the tool on its local machine.
2. Then it enriches the stitched call-graph performing queries towards the FASTEN REST APIs, writing this result in a JSON which reflects the original stitch call-graphs (stitched by pycg-stitch), with information added (which is the added value, for this step, of the PyPI plugin).
3. It will provide a summarized output to the final user, parsing the JSON generated in step 2.

Based on this design and considerations, we would like to understand what is the output of the Maven plugin.
Could you please provide us with a concrete example of the Maven plugin execution?

Thanks in advance,
Michele

[gdrosos]

Just for clarification, for Python, the output of the stitcher is a stiched call graph along with a metadata field for each corresponding Python URI. We produced an Output of the graph in a JSON representation in order to make our tool shareable. But the main logic is that we have a graph data structure which could be used to perform callable level analysies. For example in Java as far as I understand the stitched call graph structure is of type MavenGraph.

If we want to enrich the sitched python graph with vulnerability information in the package level, @amphioxis proposed adding on the stitched call graph some package level fields. But this should not be performed by pycg-sticher since the stitched graph presents information on the callable level.

Instead, an idea could be for the pypi-plugin to create a data structure similar to the MavenExtendedRevisionJavaCallGraph and add a dictionary corellating the revision name with the metadata information, similar to Maven. The you can parse the Uris of the stiched call graph and look whether their corresponding revision has vulnerability information in the dictionary, and then proceed with the analysies.

[tmortagne]

Well, we thought that Maven plugin was enriching the final stitched call graph with metadata information (vulnerabilities, quality metrics, licensing).

You are mixing very different things. The Maven plugin does enrich a stitched call graph, but that's a Java representation of that call graph and it does not have any reason to serialize it as JSON.

[michelescarlato]

@tmortagne thanks for the clarification.

Shall we see the maven plugin functioning concretely?

[tmortagne]

Shall we see the maven plugin functioning concretely?

Sorry, not sure I understand the question. Are you asking how to use the plugin in a Maven project ?

[michelescarlato]

Shall we see the maven plugin functioning concretely?

Sorry, not sure I understand the question. Are you asking how to use the plugin in a Maven project ?

It could be a good starting point to understand what we are missing to replicate it for Python.

[tmortagne]

@michelescarlato I just sent you a message on Gitter as it's starting to look like a chat :)