Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[21pt] Add Alpha Testing to CI / CD (Step Functions) #1368

Open
RobHanna-NOAA opened this issue Dec 10, 2024 · 0 comments
Open

[21pt] Add Alpha Testing to CI / CD (Step Functions) #1368

RobHanna-NOAA opened this issue Dec 10, 2024 · 0 comments
Labels
AWS Fix or Contribution for running HAND FIM in AWS CI/CD CI/CD - devOps related Sys Admin

Comments

@RobHanna-NOAA
Copy link
Contributor

RobHanna-NOAA commented Dec 10, 2024

Note: Now part of 1377 EPIC: FIM Sys Admin Tasks (and a few related FIM tasks)

Right now, our Step function system can not tack on auto running of synthesize_test_case and eval_plots. There were a number of reasons and part of the overall Step function is part of the issue.

A number of attempts were made with poor results.
Problems to be solved for this to be added

  1. Knowing when the fim_pipeline has completed in AWS. See issue 1367 (AWS Step Function notification System)
  2. A new tool to tell if fim_pipeline was successful and it is ok to move onto running alpha testing. We now have rolled up logs and folders in the output folder to help a tool determine if it is ok to proceed to alpha testing. Efforts have also been started recently with Matt via a QA tool that will help this process. 1279 QA Tool outline
  3. A separate tool that can review outputs or something from synthesize_test_case.py results before determining if it can proceed to eval_plots.py. Synthesize_test_cases.py has some logging that can be checked by this tool but we might need to upgrade it. At a min, we could just look and see if the final metrics csv file exists potentially.
  4. Kick off eval_plots.py

One of the biggest challenges is that the alpha testing can not be done in a fargate as it needs much, much more horsepower. However, it has been learned that it might be possible to add an EC2 instance to the end of the step function script. However, this will take a large amount of time to figure out and test potentially. It might not be as easy as it sounds. We already tried it but now have more knowledge to make this work via looking at HydroVIS tools.

@RobHanna-NOAA RobHanna-NOAA added CI/CD CI/CD - devOps related AWS Fix or Contribution for running HAND FIM in AWS labels Dec 10, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
AWS Fix or Contribution for running HAND FIM in AWS CI/CD CI/CD - devOps related Sys Admin
Projects
None yet
Development

No branches or pull requests

1 participant