test: e2e tests iteration 2

[gkc]

- What I did

• test: new e2e test harness. Design goals
  1. easy and fast to run locally
  2. can be run via github actions
  3. support multiple types of 'executables' (e.g. dart binaries, c binaries, docker containers)
  4. easily configurable matrix of daemon versions, client versions and tests to execute
• fix: allow sshnp >= 5.1.x client to be used in Linux scripting environments when using the -x flag

- How I did it
In tests/e2e_all:

• Entry point: scripts/main.sh, which then runs the following:
  • setup_binaries.sh which downloads artifacts and compiles local branch binaries
  • start_daemons.sh
  • run_tests.sh
  • stop_daemons.sh
  • cleanup_tmp_files.sh
• All the individual scripts use shared functions which are in tests/e2e_all/scripts/common/*.include.sh

in packages/dart/sshnoports/bin/sshnp.dart

• put try/catch around getting the original lineMode and echoMode from stdin (because the Dart VM barfs on Linux if the stdin isn't an actual terminal)
• and capture the error for later
• and throw the error if the terminal is actually required (which it won't be in a scripting environment)

- How to verify it

• e2e_all job runs, and tests pass
• run locally (MacOS or Ubuntu). See tests/e2e_all/README.md for instructions (tl;dr : execute tests/e2e_all/scripts/main.sh), as well as a description of how this test rig works.

Additional context
Shout-out and endless thanks to @JeremyTubongbanua who built the first iteration of the NoPorts e2e test rig

[XavierChanth]

Thanks for this @gkc !

[cconstab]

I cannot get the local tests to run on my M1 Mac...

╰$ ./run-act.sh 
INFO[0000] Using docker host 'unix:///Users/cconstab/.docker/run/docker.sock', and daemon socket 'unix:///Users/cconstab/.docker/run/docker.sock' 
WARN  ⚠ You are using Apple M-series chip and you have not specified container architecture, you might encounter issues while running act. If so, try running it with '--container-architecture linux/amd64'. ⚠  
[end2end_tests/e2e_test] 🚀  Start image=catthehacker/ubuntu:act-latest
INFO[0000] Parallel tasks (0) below minimum, setting to 1 
[end2end_tests/e2e_test]   🐳  docker pull image=catthehacker/ubuntu:act-latest platform= username= forcePull=true
[end2end_tests/e2e_test] using DockerAuthConfig authentication for docker pull
INFO[0001] Parallel tasks (0) below minimum, setting to 1 
[end2end_tests/e2e_test]   🐳  docker create image=catthehacker/ubuntu:act-latest platform= entrypoint=["tail" "-f" "/dev/null"] cmd=[] network="host"
[end2end_tests/e2e_test]   🐳  docker run image=catthehacker/ubuntu:act-latest platform= entrypoint=["tail" "-f" "/dev/null"] cmd=[] network="host"
Error: failed to start container: Error response from daemon: error while creating mount source path '/host_mnt/Users/cconstab/.docker/run/docker.sock': mkdir /host_mnt/Users/cconstab/.docker/run/docker.sock: operation not supported
╭─cconstab@tarial in ~/Documents/GitHub/@foundation/sshnoports/tools/local-e2e-test on gkc/new-e2e-tests ✘ (origin/gkc/new-e2e-tests)
╰$ 

The mount point used /host_mnt/ looks whacky to me..

'/host_mnt/Users/cconstab/.docker/run/docker.sock': mkdir /host_mnt/Users/cconstab/.docker/run/docker.sock: operation not supported

Spent a while on it after instaiing act via brew

No joy as yet..

running act in the top level directory seems to work fine however so act is ok I think.

╰$ act --list
INFO[0000] Using docker host 'unix:///var/run/docker.sock', and daemon socket 'unix:///var/run/docker.sock' 
WARN  ⚠ You are using Apple M-series chip and you have not specified container architecture, you might encounter issues while running act. If so, try running it with '--container-architecture linux/amd64'. ⚠  
Stage  Job ID                   Job name                                                                      Workflow name                                                         Workflow file                     Events                              
0      issue_opened             issue_opened                                                                  Auto Assign Issues to Triage column in Sprint Planning Project Board  autobug.yaml                      issues                              
0      analyze                  Analyze                                                                       CodeQL                                                                codeql.yml                        pull_request,schedule,push          
0      dependency-review        dependency-review                                                             Dependency Review                                                     dependency-review.yml             pull_request                        
0      docker                   docker                                                                        dockerhub_promote_to_latest                                           dockerhub_promote_to_latest.yaml  workflow_dispatch                   
0      docker                   docker                                                                        dockerhub_sshnpd                                                      dockerhub_sshnpd.yml              workflow_dispatch,push              
0      e2e_all                  e2e_all                                                                       end_to_end_tests                                                      e2e_all.yaml                      push,pull_request,workflow_dispatch 
0      e2e_release_test         e2e_release_test                                                              end2end_tests                                                         end2end_tests.yaml                workflow_dispatch,push,pull_request 
0      e2e_alternate_port_test  e2e_alternate_port_test                                                       end2end_tests                                                         end2end_tests.yaml                workflow_dispatch,push,pull_request 
0      e2e_test                 e2e_test                                                                      end2end_tests                                                         end2end_tests.yaml                workflow_dispatch,push,pull_request 
0      create-pull-request      create-pull-request                                                           Gitbook Review                                                        gitbook.yaml                      push                                
0      verify_tags              verify_tags                                                                   Multibuild                                                            multibuild.yaml                   workflow_dispatch                   
0      prod_rvd_test            prod_rvd_test                                                                 prod_tests                                                            prod_tests.yaml                   schedule,workflow_dispatch          
0      build                    Build package                                                                 Build and publish sshnpd PyPI package                                 python-sshnpd-build-publish.yml   workflow_dispatch,push              
0      analysis                 Scorecards analysis                                                           Scorecards supply-chain security                                      scorecards.yml                    branch_protection_rule,schedule,push
0      cli_tags                 cli_tags                                                                      unit_tests                                                            unit_tests.yaml                   workflow_dispatch,push,pull_request 
0      noports_core-unit_tests  noports_core-unit_tests                                                       unit_tests                                                            unit_tests.yaml                   workflow_dispatch,push,pull_request 
0      bump_requirements        bump_requirements                                                             Bump requirements.txt                                                 update_python_requirements.yml    pull_request,workflow_dispatch      
1      other_build              other_build                                                                   Multibuild                                                            multibuild.yaml                   workflow_dispatch                   
1      main_build               main_build                                                                    Multibuild                                                            multibuild.yaml                   workflow_dispatch                   
1      publish-to-pypi          Publish package to PyPI                                                       Build and publish sshnpd PyPI package                                 python-sshnpd-build-publish.yml   workflow_dispatch,push              
1      publish-to-testpypi      Publish package to TestPyPI                                                   Build and publish sshnpd PyPI package                                 python-sshnpd-build-publish.yml   workflow_dispatch,push              
2      notify_on_completion     notify_on_completion                                                          Multibuild                                                            multibuild.yaml                   workflow_dispatch                   
2      notify_on_failure        notify_on_failure                                                             Multibuild                                                            multibuild.yaml                   workflow_dispatch                   
2      github-release           Sign the Python distribution with Sigstore and upload them to GitHub Release  Build and publish sshnpd PyPI package                                 python-sshnpd-build-publish.yml   workflow_dispatch,push              

Detected multiple jobs with the same job name, use `-W` to specify the path to the specific workflow.
╭─cconstab@tarial in ~/Documents/GitHub/@foundation/sshnoports on gkc/new-e2e-tests ✘ (origin/gkc/new-e2e-tests)
╰$ 

[cconstab]

Also needs a bit more in the README about setting up the .env and SECRETS. Ho to create them is not obvious to me at least.
Is SECRETS just the contents of the atKeys file ? Assume so but is there ann easy way to do this perhaps ?

"JSON_FROM_KEYS"

(Quotes inside must be double delimited i.e. \" )

[gkc]

@cconstab refactored duplicate code to remove bug vectors; added expect stuff to allow testing of 'inline' (where the sshnp program itself runs the ssh session )

[cconstab]

Will take a look this eve

[XavierChanth]

Was able to pull and run locally!

[cconstab]

One improvement in the sshnp.expect script. At present it looks for the 'prompt' but there is no garrantee that the prompt actually matches the expect.
For instance I changed my shell to /bin/sh (chsh) and thuings failed as the match is not made.

So I change that line from
set expected "${env(REMOTE_USERNAME)}@${env(REMOTE_HOSTNAME)}"
to
set expected "Last login:"

This then is checking something that login provides and is standard across Linux/MacOs

I do love the Scooby Doo ref

[gkc]

One improvement in the sshnp.expect script. At present it looks for the 'prompt' but there is no garrantee that the prompt actually matches the expect. For instance I changed my shell to /bin/sh (chsh) and thuings failed as the match is not made.

So I change that line from set expected "${env(REMOTE_USERNAME)}@${env(REMOTE_HOSTNAME)}" to set expected "Last login:"

This then is checking something that login provides and is standard across Linux/MacOs

Thank you! Have made the change.

I do love the Scooby Doo ref
🐕

[gkc]

I made a few more changes

• simple single retry logic to cope with occasional network / atServer weather to cut down on number of manual GitHub action reruns we need to do
• Tweaked the test scripts' output for better readability
• Added some logic so that for already-released clients we don't run checks against already-released daemons - i.e. for already-released clients we only run forward-compatibility checks against the branch daemon
  • Conversely if the client is 'current' (i.e. built from the local git head) then it does compatibility checking with every daemon version
  • This has reduced the number of tests actually executed, with the current matrix of 3x3x7, from 27 of 63 to 19 of 63