Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add an Embedded dependency property #17

Draft
wants to merge 2 commits into
base: main
Choose a base branch
from
Draft

Add an Embedded dependency property #17

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
5d57af5
Add a IsBundled dependency property
May 25, 2022
0c9c28c
Add cmd flag to choose which type of dependencies to print
ret2libc Jul 13, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
51 changes: 49 additions & 2 deletions cmd/deplist/deplist.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,18 +3,56 @@ package main
import (
"flag"
"fmt"
"strings"

"github.com/RedHatProductSecurity/deplist"
purl "github.com/mcoops/packageurl-go"
log "github.com/sirupsen/logrus"
)

type searchModeFlag []string

func (i *searchModeFlag) String() string {
return strings.Join(*i, ", ")
}

func (i *searchModeFlag) Set(value string) error {
values := strings.Split(value, ",")
for _, value := range values {
switch value {
case "deps":
*i = append(*i, "deps")
case "bundled":
*i = append(*i, "bundled")
}
}
return nil
}

func (i *searchModeFlag) ShouldHandleDep(dep deplist.Dependency) bool {
res := false
for _, searchMode := range *i {
switch searchMode {
case "deps":
res = res || !dep.IsBundled
case "bundled":
res = res || dep.IsBundled
}
}
return res
}

func main() {
deptypePtr := flag.Int("deptype", -1, "golang, nodejs, python etc")
debugPtr := flag.Bool("debug", false, "debug logging (default false)")
var searchModes searchModeFlag
flag.Var(&searchModes, "modes", "search mode (bundled, deps)")

flag.Parse()

if len(searchModes) == 0 {
searchModes = []string{"deps", "bundled"}
}
if *debugPtr == true {
log.SetLevel(log.DebugLevel)
}
Expand All @@ -33,16 +71,25 @@ func main() {

if *deptypePtr == -1 {
for _, dep := range deps {
if !searchModes.ShouldHandleDep(dep) {
continue
}
version := dep.Version

inst, _ := purl.FromString(fmt.Sprintf("pkg:%s/%s@%s", deplist.GetLanguageStr(dep.DepType), dep.Path, version))
fmt.Println(inst)
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Perhaps instead of changing the output format for embedded dependencies we should add a command flag which indicates if we want embedded dependencies in the output of not? I think it would avoid the need to have special code in clients which for interrupting the output.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@jasinner I could add a flag -modes which accepts deps, bundled, or deps,bundled and prints dependencies based on the mode. However, for the case where you print both "regular dependencies" and bundled code you do need anyway a way to differentiate them, don't you? Opinions @jasinner ? @sfowl was suggesting to have a JSON output instead.

Do we have clients that use the output mode of deplist?

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We are planning on using deplist in component-registry. Although the integration is not done yet.

I think for component-registry it would be nice to have json output and split deps and bundled into 2 groups.

fmt.Print(inst)
fmt.Println()
}
} else {
deptype := deplist.Bitmask(*deptypePtr)
for _, dep := range deps {
if !searchModes.ShouldHandleDep(dep) {
continue
}

if (dep.DepType & deptype) == deptype {
fmt.Printf("%s@%s\n", dep.Path, dep.Version)
fmt.Printf("%s@%s", dep.Path, dep.Version)
fmt.Println()
}
}
}
Expand Down
9 changes: 5 additions & 4 deletions dependencies.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,10 +5,11 @@ type Bitmask uint32

// Dependency per dependency info
type Dependency struct {
DepType Bitmask // golang, nodejs, python etc
Path string // the module path, github.com/teris-io/shortid
Version string // v0.0.0-20171029131806-771a37caa5cf
Files []string // if available, list of all files for a package
DepType Bitmask // golang, nodejs, python etc
Path string // the module path, github.com/teris-io/shortid
Version string // v0.0.0-20171029131806-771a37caa5cf
Files []string // if available, list of all files for a package
IsBundled bool // whether the dependency is bundled somehow in the dependency (e.g. vendored code, copy-pasted code, embedded code, etc.)
// /usr/lib/go-1.13/src/regexp/syntax/compile.go
// /usr/lib/go-1.13/src/regexp/syntax/doc.go
}
Expand Down
52 changes: 36 additions & 16 deletions deplist.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -80,16 +80,32 @@ func getDeps(fullPath string) ([]Dependency, Bitmask, error) {
}

if info.IsDir() {
// prevent walking down the vendors, docs, etc
// prevent walking down the docs, .git, tests, etc.
if utils.BelongsToIgnoreList(info.Name()) {
return filepath.SkipDir
}
} else {
// Two checks, one for filenames and the second switch for full
// paths. Useful if we're looking for top of repo

switch filename := info.Name(); filename {
// for now only go for yarn and npm
case "package.json":
pkg, err := scan.GetNodeJSPackage(path)
if err != nil {
log.Debugf("failed to scan for nodejs package: %s", path)
return nil
}

foundTypes.DepFoundAddFlag(LangNodeJS)

deps = append(deps,
Dependency{
DepType: LangNodeJS,
Path: pkg.Name,
Version: pkg.Version,
Files: []string{},
IsBundled: true,
})
case "package-lock.json":
// if theres not a yarn.lock fall thru
if _, err := os.Stat(
Expand Down Expand Up @@ -160,10 +176,11 @@ func getDeps(fullPath string) ([]Dependency, Bitmask, error) {
if !strings.HasSuffix(version, "-javadoc") && !strings.HasSuffix(version, "-sources") {
deps = append(deps,
Dependency{
DepType: LangJava,
Path: name,
Version: version,
Files: []string{},
DepType: LangJava,
Path: name,
Version: version,
Files: []string{},
IsBundled: true,
})
}
}
Expand All @@ -184,10 +201,11 @@ func getDeps(fullPath string) ([]Dependency, Bitmask, error) {

for path, goPkg := range pkgs {
d := Dependency{
DepType: LangGolang,
Path: path,
Files: goPkg.Gofiles,
Version: goPkg.Version,
DepType: LangGolang,
Path: path,
Files: goPkg.Gofiles,
Version: goPkg.Version,
IsBundled: true,
}
deps = append(deps, d)
}
Expand All @@ -202,9 +220,10 @@ func getDeps(fullPath string) ([]Dependency, Bitmask, error) {
}
for _, goPkg := range pkgs {
d := Dependency{
DepType: LangGolang,
Path: goPkg.Name,
Version: goPkg.Version,
DepType: LangGolang,
Path: goPkg.Name,
Version: goPkg.Version,
IsBundled: true,
}
deps = append(deps, d)
}
Expand All @@ -219,9 +238,10 @@ func getDeps(fullPath string) ([]Dependency, Bitmask, error) {
}
for _, goPkg := range pkgs {
d := Dependency{
DepType: LangGolang,
Path: goPkg.Name,
Version: goPkg.Version,
DepType: LangGolang,
Path: goPkg.Name,
Version: goPkg.Version,
IsBundled: true,
}
deps = append(deps, d)
}
Expand Down
24 changes: 24 additions & 0 deletions internal/scan/nodejs.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,7 @@ package scan
import (
"encoding/json"
"fmt"
"os"
"os/exec"
"path/filepath"
"strings"
Expand All @@ -25,6 +26,10 @@ type yarnOutput struct {
}
}

type packageJsonFormat struct {
Name string `json:"name"`
Version string `json:"version"`
}
type npmDependency struct {
Version string `json:"version"`
Dependencies map[string]npmDependency `json:"dependencies"`
Expand Down Expand Up @@ -121,6 +126,25 @@ func GetNodeJSDeps(path string) (map[string]NodeJSGather, error) {
return nil, fmt.Errorf("unknown NodeJS dependency file %q", path)
}

func GetNodeJSPackage(path string) (NodeJSGather, error) {
log.Debugf("GetNodeJSPackage %s", path)

data, err := os.ReadFile(path)
if err != nil {
return NodeJSGather{}, err
}

var packageJson packageJsonFormat
err = json.Unmarshal(data, &packageJson)
if err != nil {
return NodeJSGather{}, err
}
if packageJson.Name == "" {
return NodeJSGather{}, fmt.Errorf("Empty package")
}
return NodeJSGather{Name: packageJson.Name, Version: packageJson.Version}, nil
}

func getYarnDeps(path string) (map[string]NodeJSGather, error) {
var yarnOutput yarnOutput
gatheredNode = make(map[string]NodeJSGather)
Expand Down
2 changes: 0 additions & 2 deletions internal/utils/utils.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,8 +9,6 @@ import (
func BelongsToIgnoreList(needle string) bool {
switch needle {
case
"node_modules",
"vendor",
Copy link
Collaborator

@sfowl sfowl Jul 13, 2022 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hmm, this could be dangerous, go list traverses the go pkg graph to identify what packages are used by the code in the provided directory. I think we filtered out vendor/ because it's possible for code to be stored there that is not actually used by code in the rest of the repo, i.e. it does not make it into the built binaries. Two causes I think that can lead to this are an out-of-sync vendor dir or that vendor dir is populated by the go module graph, not the go pkg graph, which can be different.

EDIT: By default go list also does not report test deps, however they will appear in vendor/

"scripts",
"docs",
"test",
Expand Down