feat: Portfolios #61
Merged
feat: Portfolios #61
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
* add pydocstyle devreq * add pydocstyle pc hook * add pydocstyle to lint checks * add lint and type envs to gh-actions * add pydocstyle config * add a pydocstyle ignore * add test docstring * add pydocstyle config to tox
* delete doctest * first interface sketch * add result dep * remove capital letters * split complete into subclass * move cv-outcome to dataclass * add placeholder docstring * rename verifiers -> verifier * fix import * WIP dummy test * silence warnings * remove init imports
* setup cli command * add __version__ module variable, and add it to bumpver patterns * add placeholder cli args
…face changes (#27) Adds an interface for creating verifier configuration spaces Refactors parts of the verifier interface Use attrs.define instead of dataclass (more flexibility)
* add mnist and acasxu vnncomp benchmarks * standardize benchmark layout * reorder configs * only lint src and tests * add toy dataset (test props from vnncomp2021)
* refactor verifier interface * turn verifier config space into dataclass again * add __init__ file to folder * remove unused import * pass -> NotImplementedError
* setup dir structure * rename folder * simplify dummy class attributes, add imports to inits * util + test organization * WIP cli setup * add xdg_base_dirs dependency * WIP paths setup * WIP small changes * WIP setup error and success messages, make some functions private * Installing and directory setups * clean up old tests * add install_dir param to install functions * move directory creation to function * put dir names as constants * version test * Install tests
* setup dir structure * rename folder * simplify dummy class attributes, add imports to inits * util + test organization * WIP cli setup * add xdg_base_dirs dependency * WIP paths setup * WIP small changes * WIP setup error and success messages, make some functions private * Installing and directory setups * clean up old tests * add install_dir param to install functions * move directory creation to function * put dir names as constants * version test * Install tests * fix coverage omit
* WIP nnenum setup * venv -> env, simplify base dirs creation * lowercase name * git util class * WIP nnenum installer * WIP rename to tool dir * move dir creation to installers * minor edits * subprocess error handling * WIP conda env dataclass * put installers into folders and add env yaml file * copy env.yaml file to dest * copy env util func * add nnenum conda env * conda utils * clean all script * docstrings * docstr indent
* uninstall and conda functions * add manifest.in for yaml files * remove script
* WIP * activate conda env function * get py path function * update docstring * working proof of concept + tests * nnenum tests * move function to util file * WIP tests * mark installs as slow * fix import * add --runslow flag to tox pytest * rename pytest marker slow to install * env contextmanager * remove print * move conda util up * format docstring * conda info and path util functions * modularize verifier functions * move env util up * add err message * disable failing installation tests, fix later * disable nnenum tests
* mark verifier tests as optional for now * nnenum counterexample parsing * update todos
* add dockerignore * setup first dockerfile version * change dockerfile image, simplify it * add missing nnenum req * remove install test -- will be handled by docker failing * simplify dockerfile * fix permission err * wip tox-docker * remove tox-docker req * use commands instead of tox-docker plugin * add tests to dockerfile * add tox integeration test to gh-actions
* add subparser group, fix tests * fix args * layout
* nnenum cfgspace rename * abcrown verifier files skeleton * abcrown setup steps * move clone+checkout procedure to util function * fix logging, add docstring placeholders * push for ssh access * rename env yml files [skip ci] * fix names [skip ci] * move try-except [skip ci] * update dockerfile [skip ci] * abcrown cfg maker [skip ci] * remove chdir * move yml gen to own file [skip ci] * name read [skip ci] * [skip ci] * abcrown runner attempt [skip ci] * [skip ci] * [skip ci] * [skip ci] * [skip ci] * abcrown result parsing, fix tests [skip ci] * pyyaml types in pre-commit mypy hook * fix mypy errors * add type stubs for pyyaml * seperate gpu and cpu tests * debug outputs * fix tox tests * abcrown counter-example saving * remove todo * fix dockerfile
* initial setup [skip ci] * cli setup [skip ci] * verbose install help * add todo [skip ci] * refactor tests again * update docstring * add lazy_fixture req * update docker test arg * comment * more setup [skip ci] * hacky mnbab install [skip ci] * fix docstring * onnx utility [skip ci] * mn-bab installer * fix test flags * gh-actions duplicate PR running fix * only run workflows on main/develop * nevermind
* gh actions tests * ovalbab setup + fix tox tests * ovalbab install + env * add ovalbab to tests [skip ci] * working ovalbab runner * parse from result file [skip ci] * enable tests
* start [skip ci] * cfgspace [skip ci] * fix cfg dict [skip ci] * sampling [skip ci] * rename keys, abcrown dict conversion [skip ci] * organize config conversion * add docstrings * additional file check [skip ci] * fix counter-example saving [skip ci] * rm print [skip ci] * change layout [skip ci] * unzip benchmarks * refactor configspaces [skip ci] * fix pre-commit hooks * consistent function param orders * rename folder * cleanup
* some setup [skip ci] * target func factory * docstring [skip ci] * benchmark util * test util * rename func * fix Err return type * convert str to int * no integration tests until we fix them * add smac output * vnncomp as smac instances * move to av util * smac running * running and writing instances * pandas data implementation * add docstrings * fix tests * add pandas req * remove unused import * use param in csv write * write intermediate results to csv * embed hydrasmac into av * hacky fix docstrings * seq runner setup * fix type and lints * temp readme instructions * csv header writing * type guards * rename files [skip ci] * add the file [skip ci] * todos [skip ci] * selecting verifier script * fix types and lints * remove debug prints
* reading benchmark tests * docstring + exp setups * ovalbab cfg doc * remove file [skip ci] * more util [skip ci] * mnbab installing [skip ci] * mnbab setup [skip ci] * setup [skip ci] * mnbab configspace [skip ci] * add temp config [skip ci] * more params [skip ci]
* refactor * setup baselines * ovalbab mnist [skip ci] * ovalbab experiments [skip ci] * wip hydra update [skip ci] * wip [skip ci] * exp setup [skip ci] * exp setup [skip ci] * fix instances [skip ci] * move func [skip ci] * wt lim [skip ci] * exps [skip ci] * better err reporting [skip ci] * setup exp [skip ci] * tighten abcrown configspace [skip ci] * eval scripts [skip ci] * warmup param [skip ci] * ready [skip ci] * mnist cfg [skip ci] * wip refactor [skip ci] * delete bench [skip ci] * wip refactor [skip ci] * add onnx req [skip ci] * fix no filter arg [skip ci] * verinet fixes [skip ci] * hacky verinet params [skip ci] * start on tests, minor changes [skip ci] * more tests, lib path fix [skip ci] * update verinet commit hash [skip ci] * nnenum cfg, more util tests [skip ci] * update nnenum [skip ci] * add more tests [skip ci] * more tests [skip ci] * eval scripts [skip ci] * tuning and tempfile cleaning [skip ci] * remove comment [skip ci] * abcrown yaml override [skip ci] * update verinet [skip ci] * eval fix [skip ci] * add and fix some tests [skip ci] * update nnenum hash [skip ci] * experiment setup [skip ci] * create csv if it doesnt exist yet [skip ci] * fix paths [skip ci] * fix nnenum experiment params [skip ci] * fix typo [skip ci] * lib paths, smac util [skip ci] * fix cli args [skip ci] * remove print [skip ci] * update nnenum hash [skip ci] * bump smac, add more tests [skip ci] * update eval script [skip ci] * vnn-compat eval arg [skip ci] * compat eval arg [skip ci] * fix typo [skip ci] * hydra setup * mark gpu tests * number of cores arg [skip ci] * update allocation init [skip ci] * hacky gpu alloc [skip ci] * enable nnenum inf params again [skip ci] * hydra picking * mark more gpu test * enable integration tests * update marks * shorten imports [skip ci] * pf exp ready * fix tests * fix setting resource in hydra portfolio * wip [skip ci] * vbs runner wip [skip ci] * fix timing out [skip ci] * more accurate timeout * remove dead code * wip pf runner [skip ci] * exit PF runner gracefully [skip ci] * add csv logging back [skip ci] * ecdf plot, update parsers [skip ci] * types, lints, tests * remove print * check version cmd [skip ci] * check if a gpu avail [skip ci] * instance file arg name [skip ci] * use simplified networks and verifier specific kwargs [skip ci] * fix types and lints * remove prints [skip ci] * realloc if shorter len [skip ci] * update verinet, arg passing in pf runner [skip ci] * actually allocate in pf runner [skip ci] * abcrown configspace rework [skip ci] * fix using simple instance [skip ci] * abcrown configspace more params [skip ci] * update verinet commit hash [skip ci] * uniq network func [skip ci] * use simple net in eval [skip ci] * update verinet hash [skip ci] * cfgspace test [skip ci] * remove print * more tests [skip ci] * docs start [skip ci] * update abcrown cfgspace [skip ci] * sync [skip ci] * remove tempfiles [skip ci] * fix type and lint [skip ci] * update docs [skip ci] * wip docstrings [skip ci] * more docstrings [skip ci] * mockable cpu/gpu counts for tests * Update docstrings, remove debug prints * fix mark name * actually push tests
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
No description provided.