Re write modules #12

aaronkaplan · 2021-05-20T22:22:03Z

Description

See the CHANGELOG file

Related issue

#11 , #7 , #10 #9 , #2 , #5 .

Check list

Related issue / work item is attached
Unit-tests are written (if applicable)
Documentation is updated (if applicable)
Changes are tested, tests pass, no code linting errors and no high/critical vulnerabilities identified in codebase.

Testing

Did you write new unit tests for this change?
Did you write new integration tests for this change?
Include the test commands you ran locally to test this change
e.g.:

python -m pytest -vv --cov-branch --cov=./ --cov-report=term --color=yes tests/

Monitoring

n/a/

Rollout

n/a

Signed-off-by: aaronkaplan <[email protected]> Signed-off-by: aaronkaplan <[email protected]>

data format

Missing stuff: write the results to the DB at the end.

…the GH actions would work out. Signed-off-by: aaronkaplan <[email protected]> Signed-off-by: aaronkaplan <[email protected]>

Signed-off-by: aaronkaplan <[email protected]> Signed-off-by: aaronkaplan <[email protected]>

…racefully. Signed-off-by: aaronkaplan <[email protected]> Signed-off-by: aaronkaplan <[email protected]>

Signed-off-by: aaronkaplan <[email protected]> Signed-off-by: aaronkaplan <[email protected]>

- [X] delimiter gets automatically detected in spycloud - [X] make sure that if a row can't be enriched properly, that we set needs_human_intervention = True - [X] the enrichment pipeline works now Signed-off-by: aaronkaplan <[email protected]> Signed-off-by: aaronkaplan <[email protected]>

* abuse_contact.py: AbuseContact.lookup() returns a List[str] now. Fixes #45. * [X] cleanup code in main.py: imports * fix a testcase for the spycloud collector Signed-off-by: aaronkaplan <[email protected]> Signed-off-by: aaronkaplan <[email protected]>

abuse_contact enricher

less debug print()

…credentialLeakDB into re-write-modules

needs_human_intervention: bool. Fixes #51

…modules

Signed-off-by: aaronkaplan <[email protected]> Signed-off-by: aaronkaplan <[email protected]>

clean up main.py merge models/outdf.py and api/models.py. The latter becoming obsolete. Signed-off-by: aaronkaplan <[email protected]> Signed-off-by: aaronkaplan <[email protected]>

…ot need the notify or needs_human_attention fields every time. So make them optional Signed-off-by: aaronkaplan <[email protected]> Signed-off-by: aaronkaplan <[email protected]>

…alLeakDB into re-write-modules

Signed-off-by: aaronkaplan <[email protected]> Signed-off-by: aaronkaplan <[email protected]>

…ternalDataFormat, be ready to serve an empty dict but with an error message. Signed-off-by: aaronkaplan <[email protected]> Signed-off-by: aaronkaplan <[email protected]>

…alLeakDB into re-write-modules � Conflicts: � modules/parsers/spycloud.py

implemented the store() function Signed-off-by: aaronkaplan <[email protected]>

Signed-off-by: aaronkaplan <[email protected]>

Signed-off-by: aaronkaplan <[email protected]> Signed-off-by: aaronkaplan <[email protected]>

Signed-off-by: aaronkaplan <[email protected]>

… functions Signed-off-by: aaronkaplan <[email protected]>

… And finally, anonymize passwords in log files! damn it. Signed-off-by: aaronkaplan <[email protected]>

Merge branch 'main' of github.com:EC-DIGIT-CSIRC/credentialLeakDB into re-write-modules

aaronkaplan and others added 30 commits May 11, 2021 17:32

human_intervention becomes a sub-dict with a reason field

403920f

Signed-off-by: aaronkaplan <[email protected]> Signed-off-by: aaronkaplan <[email protected]>

bump requriremnets - security bug in pydantic

b2365d1

Add abstract classes for collectors, enrichers, and parsers

62aba6b

make sure local data does not end up in github

dae4998

add model definitons: input data format, internal data format, output

32a773c

data format

Added filters and enrichers + test cases

2318a99

add anonymized data in the correct spycloud format

f340d9f

add unit tests for the new collectors, parsers and enrichers

af5a4e3

Complete new rewrite / refactoring of nearly everything.

e191ce3

Missing stuff: write the results to the DB at the end.

okay, don't intentionally create a syntax error - we want to see how …

0a8b7b5

…the GH actions would work out. Signed-off-by: aaronkaplan <[email protected]> Signed-off-by: aaronkaplan <[email protected]>

minor fixes which pylint found

25fc8f5

Signed-off-by: aaronkaplan <[email protected]> Signed-off-by: aaronkaplan <[email protected]>

minor bug, parse a df not a Path object

6b8afae

Signed-off-by: aaronkaplan <[email protected]> Signed-off-by: aaronkaplan <[email protected]>

don't upload the VIPs test file thank you very much :)

8b95461

fixing bug: what happens when no LDAP connection is here? Handle it g…

b60070c

…racefully. Signed-off-by: aaronkaplan <[email protected]> Signed-off-by: aaronkaplan <[email protected]>

Add helper to figure out the delimiter of a CSV file. Refactored code...

79310e9

Signed-off-by: aaronkaplan <[email protected]> Signed-off-by: aaronkaplan <[email protected]>

minor bug, parse a df not a Path object

faf8ac9

Signed-off-by: aaronkaplan <[email protected]> Signed-off-by: aaronkaplan <[email protected]>

Fix:

893f4c5

* abuse_contact.py: AbuseContact.lookup() returns a List[str] now. Fixes #45. * [X] cleanup code in main.py: imports * fix a testcase for the spycloud collector Signed-off-by: aaronkaplan <[email protected]> Signed-off-by: aaronkaplan <[email protected]>

cover the other directories with pylint, fix a syntax error in the

e354d51

abuse_contact enricher

fix abuse_contact() lookup test case

d93bccd

rebasing

cd63a23

merge

d3717a8

email_domain instead of domain

7dff2d9

less debug print()

fix a http 500

2a2c4be

Merge branch 're-write-modules' of https://github.com/EC-DIGIT-CSIRC/…

9315617

…credentialLeakDB into re-write-modules

fix unit tests - adapt to the new format with notify: bool,

9e10797

needs_human_intervention: bool. Fixes #51

Merge remote-tracking branch 'origin/re-write-modules' into re-write-…

56ddbc4

…modules

Expose is_vip function via RESTAPI + write unit tests for them.

72fbf68

Signed-off-by: aaronkaplan <[email protected]> Signed-off-by: aaronkaplan <[email protected]>

fix unittests. Also re-enable them for develoment branches

4a3c9e2

map email_domain -> domain

d9f045a

clean up main.py merge models/outdf.py and api/models.py. The latter becoming obsolete. Signed-off-by: aaronkaplan <[email protected]> Signed-off-by: aaronkaplan <[email protected]>

aaronkaplan added 29 commits May 19, 2021 16:07

bug: output validation did not work... but the internal format does n…

52bcd3b

…ot need the notify or needs_human_attention fields every time. So make them optional Signed-off-by: aaronkaplan <[email protected]> Signed-off-by: aaronkaplan <[email protected]>

Merge branch 're-write-modules' of github.com:EC-DIGIT-CSIRC/credenti…

6c2ea5a

…alLeakDB into re-write-modules

again... sigh

e48d657

Signed-off-by: aaronkaplan <[email protected]> Signed-off-by: aaronkaplan <[email protected]>

again... sigh

53d280a

Signed-off-by: aaronkaplan <[email protected]> Signed-off-by: aaronkaplan <[email protected]>

in case something goes wrong with the validation + conversion into In…

538b9fb

…ternalDataFormat, be ready to serve an empty dict but with an error message. Signed-off-by: aaronkaplan <[email protected]> Signed-off-by: aaronkaplan <[email protected]>

in case something goes wrong with the validation + conversion into In…

acc0e5f

…ternalDataFormat, be ready to serve an empty dict but with an error message. Signed-off-by: aaronkaplan <[email protected]> Signed-off-by: aaronkaplan <[email protected]>

Merge branch 're-write-modules' of github.com:EC-DIGIT-CSIRC/credenti…

cb8d02b

…alLeakDB into re-write-modules � Conflicts: � modules/parsers/spycloud.py

clean up. Make PEP8 happy

abc248d

implement output data format conversion - trivial at the moment

fffdc68

Don't notify external users.

93068c9

added the PostgresqlOutput module.

c577fa3

implemented the store() function Signed-off-by: aaronkaplan <[email protected]>

added the PostgresqlOutput module.

45d467a

implemented the store() function Signed-off-by: aaronkaplan <[email protected]>

remove outdated & unused code.

702f8dc

Signed-off-by: aaronkaplan <[email protected]>

Make pylint happier, make the editor happier, make flake8 happier.

46f2de8

Signed-off-by: aaronkaplan <[email protected]> Signed-off-by: aaronkaplan <[email protected]>

make flake8 happier

efc0951

more making the editor checks happy.

769cd93

Signed-off-by: aaronkaplan <[email protected]>

make pytest faster, no point in coverage for an venv/* directory

ba488d8

also ignore venv/* in sonarcloud settings

b99aca4

add unit test for basecollector. Fix a bug on the way :)

dd80ce6

add more testcases

dc169fb

add getlogger() - our generic logger which also works for classes and…

ce08dca

… functions Signed-off-by: aaronkaplan <[email protected]>

move all prints() to logger.debug(). use the logger from lib.helpers.…

26f6bac

… And finally, anonymize passwords in log files! damn it. Signed-off-by: aaronkaplan <[email protected]>

add logger

04d069a

write unittest for enrichers/external_eail

c5eebd9

add a unit test for import spycloud. Coverage goes up a lot!

876edf5

switch to custom logger (1)

8f836d5

more unit tests... for enricher restful endpoint

74f412c

get up to date for a merge

7bf67e7

Merge branch 'main' of github.com:EC-DIGIT-CSIRC/credentialLeakDB into re-write-modules

update CHANGELOG

77037e9

aaronkaplan merged commit 2b72913 into main May 20, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Re write modules #12

Re write modules #12

aaronkaplan commented May 20, 2021

Re write modules #12

Re write modules #12

Conversation

aaronkaplan commented May 20, 2021

Description

Related issue

Check list

Testing

Monitoring

Rollout