Extend HTTPClient usage for downloads to ArtifactorySearch

[aikebah]

          Confirmed... only NexusSearch and ArtifactorySearch are pending. As both are typically solutions within the enterprise datacenter not requiring proxy access I think it would be fine to finish up this part and leave the remainder phase-out of URLConnectionFactory for a later feature-release in the 11.x series.

Originally posted by @aikebah in #6949 (comment)

Follow-up issue to #6949 to use HTTPClient for ArtifactorySearch in order to honor the standard proxy configuration also for the (expected rare) cases in which an Artifactory repository is only reachable via a proxy

[aikebah]

@irrandon, @jthurne Are you still using Artifactory with authentication?
If so, would you be able to test a snapshot version of DependencyCheck?

I'm looking for someone who can verify the correct working of ODC snapshot against an artifactory server requiring authentication by the time I've finished coding the required changes for this enhancement in order to not break its connectivity in a future ODC release.

[aikebah]

@marcelstoer Spotted in other tickets that you have some items stored in Artifactory. Are you using Artifactory Pro version and would you be able to help validate correct working of authentication towards it (preferably both basic and bearer) by the ArtifactoryAnalyzer using a snapshot-build of a feature-branch in this project?

[marcelstoer]

@aikebah I'd be happy to help. I am authenticating through <username> and <password> in settings.xml. I know we have a commercial Artifactory license but I have no idea what version we use.

[aikebah]

@marcelstoer any version would do, as long as it's the Artifactory Pro. Open source setup responds on the search with HTTP 400:

This REST API is available only in Artifactory Pro (see: jfrog.com/artifactory/features). If you are already running Artifactory Pro please make sure your server is activated with a valid license key.

I hope to work on polishing up the ArtifactorySearch transformation to httpclient5 coming Wednesday and will ping back here when a testable branch has been pushed to this repo

[aikebah]

@marcelstoer Implementation is complete. I would appreciate a test using a snapshot built from the feat/artifactory branch.

[marcelstoer]

All good, the check passed w/o problem. Would I need to enable the Artifactory analyzer for what you need me to test?

[aikebah]

@marcelstoer Yes, artifactoryAnalyzer would have to be enabled. And would be best if at least one of your project's dependencies would be findable by sha1 on your Artifactory instance's search API - that/those dependencies should then have artifactory evidences included in the evidences section of the dependency-check report.

[marcelstoer]

I had no idea what the Artifactory analyzer is or does and the best documentation I could find appears to be #60. Hence, I didn't really know what I was doing here 😄

Anyway, I hope the output I collected is of any use for you.

Command

mvn --debug org.owasp:dependency-check-maven:11.1.2-SNAPSHOT:aggregate \
-DartifactoryAnalyzerEnabled -DartifactoryAnalyzerUrl=https://my-bin/artifactory -DartifactoryAnalyzerServerId=my-bin \
-DnvdDatafeedUrl="https://dependency-check.github.io/DependencyCheck_Builder/nvd_cache/nvdcve-{0}.json.gz" \
-DfailBuildOnCVSS=7 -DcveStartYear=2020 -DskipProvidedScope=true -DnodeAuditSkipDevDependencies=true -DnodePackageSkipDevDependencies=true > odc.log

Debug snippet

...
[DEBUG] Initializing Artifactory Analyzer
[DEBUG] Initializing Artifactory analyzer
[DEBUG] Artifactory analyzer enabled: true
[DEBUG] Artifactory Search URL https://my-bin/artifactory
[DEBUG] Using default non-legacy proxy configuration
[DEBUG] Starting Artifactory Analyzer
[DEBUG] Parallel processing with up to 16 threads: Artifactory Analyzer.
[DEBUG] Begin Analysis of '/Users/marcelstoer/.m2/repository/some-jar.jar' (Artifactory Analyzer)
[DEBUG] Begin Analysis of '/Users/marcelstoer/.m2/repository/some-other-jar.jar' (Artifactory Analyzer)
...

ODC report excerpt

[aikebah]

@marcelstoer at least no errors encountered, otherwise you'd shown those I guess.
For that aws sdk library, in the evidences section, are there also rows with artifactory in the source-column?

And if not: is
https://my-bin/artifactory/api/search/checksum?sha1=1e02ef5a66694ed83bab70b5bdc3f73fe6ec0998 indeed also not returning hits?

[marcelstoer]

Yes, no errors.

For that aws sdk library, in the evidences section, are there also rows with artifactory in the source-column?

No, maybe because the dependencies are all present in the local repo?

is https://my-bin/artifactory/api/search/checksum?sha1=1e02ef5a66694ed83bab70b5bdc3f73fe6ec0998 indeed also not returning hits?

It's returning the AWS SDK

{
  "results" : [ {
    "uri" : "https://my-bin/artifactory/api/storage/maven-central-cache/com/amazonaws/aws-java-sdk-core/1.11.277/aws-java-sdk-core-1.11.277.jar"
  } ]
}

[aikebah]

Ah... I'd missed the detail of...

        for (Evidence e : dependency.getEvidence(EvidenceType.VENDOR)) {
            if ("pom".equals(e.getSource())) {
                return;
            }
        }

in the artifactoryAnalyzer so indeed... it would not use the Artifactory search as the jar will no doubt already have the pom associated with it already and therefor pom-based evidences are already present.

I guess best way to try and get it to really hit Artifactory with a request would be to put the jar in some folder and use the CLI from within that folder to scan the current folder (with artifactory analyzer enabled and centralAnalyzer disabled)

path/to/odc/bin/dependency-check.sh \
-s . \
--disableCentral \
--enableArtifactory \
--artifactoryUrl https://my-bin/artifactory \
--artifactoryUsername $ARTIFACTORY_USER \
--artifactoryApiToken $ARTIFCATORY_TOKEN \
-d ~/.m2/repository/org/owasp/dependency-check-data/11.0 \
--noupdate \
-l odc.log

should allow you to run it and simply re-use the already cached cveDB in your maven local repo, assuming that your register your login credentials for artifactory in the referenced env variables
(after build of the snapshot you would typically be able to run dependency-check.sh directly from the project target folder at <path/to/DependencyCheck>/cli/target/release/bin/dependency-check.sh)

[aikebah]

@marcelstoer Would you be able to run a trial with the CLI against your artifactory? When 'seen working' I would mark the PR ready and have it merged for the next release. I'd rather not merge and then find out that it 'breaks when used' if we can avoid that by testing the snapshot.