Releases: google/testrun
v2.0.1
v2.0
Version 2.0 of Testrun provides new functionality to support new users hoping to submit a device for a pilot assessment. In addition, new features support a number of use cases for both Google and non-Google users.
New features
- Device create and edit modal has been updated and requires further device information
- Device testing type (Device qualification or pilot assessment) has been added to device object
- DHCP disconnect and DHCP disconnect IP change tests have been implemented
- Test module reports now provide further information
- Device testing type has been added to the reports list page
- Optional recommendations have been added to the pilot assessment device report
- TLS v1.0 test has been implemented
Improvements
- Documentation has been reviewed and updated
- UI is now built during package, not installation
- Informational test result color has changed
- PDF report design has been improved
- Unit testing has been enabled in GitHub actions
- Installation time is reduced by using slim Python images for test modules
Bug fixes
- DHCP NAK messages no longer trigger a Non Compliant result in the DHCP snooping test
- Steps to resolve are no longer present on tests with Error or Informational test result
- Single IP test will no longer detect the testing adapters
Known issues
- We currently have an accessibility issue in the create/edit device modal where some text does not display correctly when using accessibility tools.
- If you are using a python version lower than 3.9, you will experience the following error:
Traceback (most recent call last):
File "framework/python/src/core/test_runner.py", line 25, in <module>
from testrun import Testrun
File "/usr/local/testrun/framework/python/src/core/testrun.py", line 30, in <module>
from session import TestrunSession
File "/usr/local/testrun/framework/python/src/core/session.py", line 80, in <module>
class TestrunSession():
File "/usr/local/testrun/framework/python/src/core/session.py", line 638, in TestrunSession
def _remove_invalid_questions(self, questions: list[dict]) -> list[dict]:
TypeError: 'type' object is not subscriptable
To resolve, you will need to upgrade to 3.9 by doing the following:
-
Upgrade python:
apt install python3.9 -
Set python3.9 as default python installation by running the following:
sudo update-alternatives --install /usr/bin/python3 python3 /usr/bin/python3.9 1
sudo update-alternatives --install /usr/bin/python3 python3 /usr/bin/python3.9 2
sudo update-alternatives --config python3
- Confirm python3.9 is now the default:
python3 --version
Python 3.9.5
- Install 3.9 venv:
sudo apt install python3.9-venv
v1.4
During Q3 of 2024, Testrun will see a number of minor releases containing just a handful of new features, improvements and bug fixes. Whilst it is not required that you update your installation of Testrun, it is recommended so that you have a better experience with the application.
New features
- Addition of a web-sockets server for real-time communication between front and backend
- Receive a notification when an adapter is connected or disconnected
- Implemented automated unit testing to improve stability
- Addition of the informational dns.mdns test
- Added GRPC server to allow host network control
- TLS 1.3 server and client informational tests have been added
Improvements
- Duplicate DNS queries are now combined in the PDF report
- Improved security of the package workflow
- Disabled the UI during network only mode
- Disabled the monitoring period when in network only mode
- Tests always report Informational if they are informational tests
- All pylint issues have been resolved
- Google CLA, pylint and ESLint checks are now enforced
- Added a no profile option in the Testrun zip download
- Backend not responding error message has been rephrased
Bug fixes
- Individual test module configuration changes are no longer ignored
- DNS queries from the DNS server to 8.8.8.8 are now hidden
- Draft risk profiles can now expire after 1 year
v1.4-alpha
NOTE: This is a pre-release. It is not ready for official use. Feel free to give it a go and let us know if you find any bugs ahead of the release. For official use, please always download the latest version.
New features
- Addition of a web-sockets server for real-time communication between front and backend
- Implemented automated unit testing to improve stability
- Addition of the informational dns.mdns test
- Addition of the TLS client and server v1.3 tests
- TLS 1.3 server and client informational tests have been added
- Added risk level to each individual question in the PDF report
- Check for internet connectivity during a Testrun (wifi icon in top right)
Improvements
- Duplicate DNS queries are now combined in the PDF report
- Improved security of the package workflow
- Disabled the UI during network only mode
- Disabled the monitoring period when in network only mode
- Tests always report Informational if they are informational tests
- UI is now pre-built to speed up installation times
- All pylint issues have been resolved
- Google CLA, pylint and ESLint checks are now enforced
- Added a no profile option in the Testrun zip download
- Backend not responding error message has been rephrased
- Added automated unit testing
- More complete documentation
Bug fixes
- Individual test module configuration changes are no longer ignored
- DNS queries from the DNS server to 8.8.8.8 are now hidden
- Draft risk profiles can now expire after 1 year
v1.3.1
v1.3
NOTE: Please ensure ethool is available when using Testrun. Install using sudo apt-get install ethtool
Testrun v1.3 is one of our biggest updates yet! New additions to Testrun include an automated device risk assessment, certificate management and 3 new connection tests have been automated. See below for a detailed list of all changes:
New features
- Ability to upload local CA certificates into the user interface
- Answer questions to form a risk assessment for your device and attach to a ZIP export
- 3x new connection tests (port_link, port_duplex, port_speed)
- Mockoon and Postman files for the API have been shared
- Test modules are now loaded into the user interface dynamically
- The number of successful tests out of the total number executed is now in the PDF report
- Feature Not Detected test result has been added
- Informational test result has been added
Improvements
- Version numbers are no longer repeated in multiple locations in the code
- Test modules can be changed for one Testrun when quick launching a device test
- Added more exception handling in API endpoints
- Testrun version is now reported in the JSON report
- Updated documentation
- Move ZIP files to a temp folder
Bug fixes
- The device under test now has a working internet connection
- Testrun now stops if you disconnect the device during the monitor period
- ARP responses from link local addresses are now accepted
- QUIC UDP protocol is now allowed in the TLS 1.2 client test
- Fixed an issue where a second Testrun would be cancelled automatically
- Prevent creating a device where a folder for that device already exists
v1.2.2
Testrun v1.2.2 is a small release which adds some minor functionality and resolves some bugs:
Improvements
- Updated documentation on test statuses
- Ability to shutdown Testrun from the user interface
- Modify quick start icon on device tile
- Package script improved
- Shortened maximum device manufacturer and model name to 28 characters
- Sets default report limit to infinite
- Added steps to resolve to the PDF report
- PDF test report edge cases considered
Bug fixes
- Better error handling of test module results
- Testrun now stops if device is disconnected during monitor period
- Fix issue with TLS client test
- Increase API timeout when stopping Testrun
- Fixed issue when failed to fetch the latest version number
- Fixed issue with module reports duplicating on second test attempt
- Rendering issue with error test result resolved
v1.2.1
v1.2
Testrun v1.2 brings a number of user interface and testing improvements. Most notably, users will be provided with suggestions to resolve test failures (steps to resolve) and will find an updated report design with more in-depth information for some test modules.
Improvements
- Aligned with v4.5 test plan (removed recommended tests)
- Error message when configured port is missing
- Improved version update notification
- Ability to opt out of Google Analytics
- Improved accessibility of the user interface
- Ability to start a Testrun from the device repository
- Log level and monitoring period can be set from the user interface
- Steps to resolve test failures are suggested during a test
- Add export button on reports list to download ZIP file
- Improved report design with module specific data
- Addition of protocol module
- Addition of DHCP snooping test and ARP inspection test
- Completed documentation for test modules
- Updated README
- Modified services module to perform a full tcp/udp scan
Bug fixes
- #292 Improved handling of failed tests (will no longer disappear from the report)
- #241 Capture all module logs
- #190 Device names overflow correctly in the PDF report
- Report not removed from reports list when exceeding report limit
- Resolve TLS intermediate certificates for validation
- Ability to configure interfaces with just one interface
- Fixed some edge cases in the network listener
- Removed flaky test_tests automated action
v1.1.3
README: If upgrading from v1.1.1 or below, a clean installation will be required. This includes cleaning all existing Docker images. Please run the following command before upgrading: sudo docker system prune -a (this may take some time)
Bug fixes
#292 Missing exception for TLS 1.2 server test