Skip to content

Commit

Permalink
flashing: allow for different power off detection mechanisms
Browse files Browse the repository at this point in the history
Allows for the selection of either ethernet carrier or serial output to detect DUT power off

Change-type: minor
Signed-off-by: Ryan Cooke <[email protected]>
  • Loading branch information
rcooke-warwick authored and acostach committed Aug 22, 2024
1 parent d2f83a6 commit 3f9dbb1
Show file tree
Hide file tree
Showing 7 changed files with 177 additions and 102 deletions.
9 changes: 9 additions & 0 deletions lib/flashing/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -34,6 +34,15 @@ There are multiple possibilities here:
- If you do that, use `generic-flasher-boot-switch`.
- if testing balenaOS - flash the internal storage of the device once manually. Then leave the switch in the internal storage position. Then connect it to the autokit. Future boots, the device will automatically boot from a flasher image on the SD card if present, or internal storage otherwise. This method requires no automated boot switch toggling, but may be weak for testing BSP updates

#### Power of detection for flasher images

There are currently 2 ways to detect that a device running the flasher image has powered down after internal flashing:

1. `ethernet`: (default) The ethernet carrier signal is used - the autokit will use the interface configured via the `WIRED_IF` env variable. `POWER_OFF_TIMEOUT` can be used to control the number of cycles to wait and check the signal.
2. `serial`: The serial output of the DUT is monitored for a message `reboot: Power down` , configurable via the `POWER_OFF_MESSAGE` env variable. The interface defined via the `DEV_SERIAL` will be used, and this requires the `ftdi` serial implementation to be used. `POWER_OFF_TIMEOUT` can be used to set a timeout in ms (default 5 mins).

The method is selectable via `POWER_OFF_SELECTOR`, with `ethernet` or `serial`

### USB-BOOT / raspberry pi compute module devices (CM3/CM4)

For raspberry pi compute module devices, the autokit can bring them up into mass storage mode then write the OS image.
Expand Down
100 changes: 3 additions & 97 deletions lib/flashing/index.ts
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,7 @@ import * as sdk from 'etcher-sdk';
import * as fs from 'fs/promises';
import { BlockDeviceAdapter } from 'etcher-sdk/build/scanner/adapters';
import { Autokit } from '../';
import { waitForPowerOff } from './powerDetection';

/**
* Flash an image to a disk - this is the low level function used to flash a disk (SD card, USD storage device etc)
Expand Down Expand Up @@ -129,22 +130,6 @@ async function flashSD(filename: string, autoKit: Autokit){

}

/**
* Checks whether the DUT is powered using Ethernet carrier detection
**/
async function checkDutPower(autoKit:Autokit) {
const [stdout, _stderr] = await exec(`cat /sys/class/net/${autoKit.network.wiredIface}/carrier`);
const file = stdout.toString();
if (file.includes('1')) {
console.log(`DUT is currently On`);
return true;
} else {
console.log(`DUT is currently Off`);
return false;
}
}


const KEY_DELAY = Number(process.env.KEY_DELAY) || 500;
async function keyboardSequence(autoKit: Autokit, keyboard: [string]){
for(let key of keyboard){
Expand Down Expand Up @@ -181,53 +166,8 @@ async function flashFlasher(filename: string, autoKit: Autokit, jumper: boolean,

await autoKit.power.on();


await waitForPowerOff(autoKit);

// dut will now internally flash - need to wait until we detect DUT has powered off
// can be done through ethernet carrier signal, or through current measurement (or something else...)
// FOR NOW: use the network
// check if the DUT is on first
let dutOn = false;
while (!dutOn) {
console.log(`waiting for DUT to be on`);
dutOn = await checkDutPower(autoKit);
await delay(1000 * 5); // 5 seconds between checks
}
// once we confirmed the DUT is on, we wait for it to power down again, which signals the flashing has finished
// wait initially for 60s and then every 10s before checking if the board performed a shutdown after flashing the internal storage
const POLL_INTERVAL = 1000; // 1 second
const POLL_TRIES = 20; // 20 tries
const TIMEOUT_COUNT = 30;
let attempt = 0;
await delay(1000 * 60);
while (dutOn) {
await delay(1000 * 10); // 10 seconds between checks
console.log(`waiting for DUT to be off`);
dutOn = await checkDutPower(autoKit);
// occasionally the DUT might appear to be powered down, but it isn't - we want to confirm that the DUT has stayed off for an interval of time
if (!dutOn) {
let offCount = 0;
console.log(`detected DUT has powered off - confirming...`);
for (let tries = 0; tries < POLL_TRIES; tries++) {
await delay(POLL_INTERVAL);
dutOn = await checkDutPower(autoKit);
if (!dutOn) {
offCount += 1;
}
}
console.log(
`DUT stayted off for ${offCount} checks, expected: ${POLL_TRIES}`,
);
if (offCount !== POLL_TRIES) {
// if the DUT didn't stay off, then we must try the loop again
dutOn = true;
}
}
attempt += 1;
if (attempt === TIMEOUT_COUNT){
throw new Error(`Timed out while trying to flash internal storage!!`)
}
}
console.log('Internally flashed - powering off DUT');
// power off and toggle mux.
await delay(1000*10);
Expand Down Expand Up @@ -527,41 +467,7 @@ async function flashJetson(filename: string, autoKit: Autokit, deviceType: strin

if(nvme){
// wait for jetson to power off
const POLL_INTERVAL = 1000; // 1 second
const POLL_TRIES = 20; // 20 tries
const TIMEOUT_COUNT = 30000;
let attempt = 0;
let dutOn = true;
await delay(1000 * 60);
while (dutOn) {
await delay(1000 * 10); // 10 seconds between checks
console.log(`waiting for DUT to be off`);
dutOn = await checkDutPower(autoKit);
// occasionally the DUT might appear to be powered down, but it isn't - we want to confirm that the DUT has stayed off for an interval of time
if (!dutOn) {
let offCount = 0;
console.log(`detected DUT has powered off - confirming...`);
for (let tries = 0; tries < POLL_TRIES; tries++) {
await delay(POLL_INTERVAL);
dutOn = await checkDutPower(autoKit);
if (!dutOn) {
offCount += 1;
}
}
console.log(
`DUT stayted off for ${offCount} checks, expected: ${POLL_TRIES}`,
);
if (offCount !== POLL_TRIES) {
// if the DUT didn't stay off, then we must try the loop again
dutOn = true;
}
}
attempt += 1;
if (attempt === TIMEOUT_COUNT){
await autoKit.power.off();
throw new Error(`Timed out while trying to flash internal storage!!. Powered off DUT.`)
}
}
await waitForPowerOff(autoKit);
console.log('Internally flashed - powering off DUT');
}

Expand Down
67 changes: 67 additions & 0 deletions lib/flashing/powerDetection/ethernet.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,67 @@
import { Autokit } from '../../';
import { exec } from 'mz/child_process';
import { delay } from 'bluebird';


const TIMEOUT_COUNT = Number(process.env.POWER_OFF_TIMEOUT) || 30

/**
* Checks whether the DUT is powered using Ethernet carrier detection
**/
async function checkDutPower(autoKit:Autokit) {
const [stdout, _stderr] = await exec(`cat /sys/class/net/${autoKit.network.wiredIface}/carrier`);
const file = stdout.toString();
if (file.includes('1')) {
console.log(`DUT is currently On`);
return true;
} else {
console.log(`DUT is currently Off`);
return false;
}
}

export async function waitForPowerOffEthernet(autoKit:Autokit):Promise<void>{
// dut will now internally flash - need to wait until we detect DUT has powered off
// FOR NOW: use the network
// check if the DUT is on first
let dutOn = false;
while (!dutOn) {
console.log(`waiting for DUT to be on`);
dutOn = await checkDutPower(autoKit);
await delay(1000 * 5); // 5 seconds between checks
}

const POLL_INTERVAL = 1000; // 1 second
const POLL_TRIES = 20; // 20 tries
let attempt = 0;
await delay(1000 * 60);
while (dutOn) {
await delay(1000 * 10); // 10 seconds between checks
console.log(`waiting for DUT to be off`);
dutOn = await checkDutPower(autoKit);
// occasionally the DUT might appear to be powered down, but it isn't - we want to confirm that the DUT has stayed off for an interval of time
if (!dutOn) {
let offCount = 0;
console.log(`detected DUT has powered off - confirming...`);
for (let tries = 0; tries < POLL_TRIES; tries++) {
await delay(POLL_INTERVAL);
dutOn = await checkDutPower(autoKit);
if (!dutOn) {
offCount += 1;
}
}
console.log(
`DUT stayed off for ${offCount} checks, expected: ${POLL_TRIES}`,
);
if (offCount !== POLL_TRIES) {
// if the DUT didn't stay off, then we must try the loop again
dutOn = true;
}
}
attempt += 1;
if (attempt === TIMEOUT_COUNT){
await autoKit.power.off();
throw new Error(`Timed out while trying to flash internal storage!!. Powered off DUT.`)
}
}
}
15 changes: 15 additions & 0 deletions lib/flashing/powerDetection/index.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,15 @@
import { Autokit } from "../..";
import { waitForPowerOffEthernet } from "./ethernet";
import { waitForPowerOffSerial } from "./serial";

const powerOffSelector = process.env.POWER_OFF_SELECTOR || 'ethernet'

const powerOffFunctions: {[key: string]: (autoKit: Autokit) => Promise<void> } = {
ethernet: waitForPowerOffEthernet,
serial: waitForPowerOffSerial,
};

const waitForPowerOff = powerOffFunctions[powerOffSelector];


export { waitForPowerOff }
37 changes: 37 additions & 0 deletions lib/flashing/powerDetection/serial.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,37 @@
import { Autokit } from '../../';
import { ReadlineParser } from '@serialport/parser-readline';
import { Readable } from 'serialport';

const powerOffMessage = process.env.POWER_OFF_MESSAGE || 'reboot: Power down'
const timeout = Number(process.env.POWER_OFF_TIMEOUT) || 1000*60*5;

/**
* Checks whether the DUT has powered down by checking for a serial message
**/
export async function waitForPowerOffSerial(autoKit:Autokit):Promise<void> {
const serialport = await autoKit.serial.open();

if(serialport !== undefined){
return new Promise<void>((resolve, reject) => {

let timer = setTimeout(async () => {
await autoKit.serial.close();
reject(`Timed out while waiting for power down message over serial!`);
}, timeout)

// Examine each line of the serial output from the DUT as it comes in
const parser = serialport.pipe(new ReadlineParser({ delimiter: '\n' }));
parser.on('data', async(data:string) => {
console.log(`Serial line: ${data.toString()}`);
if(data.toString().includes(powerOffMessage)){
console.log(`### Detected power off message ###`)
await autoKit.power.off();
await autoKit.serial.close();
clearTimeout(timer);
resolve();
}
})
})

}
}
50 changes: 45 additions & 5 deletions package-lock.json

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

1 change: 1 addition & 0 deletions package.json
Original file line number Diff line number Diff line change
Expand Up @@ -23,6 +23,7 @@
"license": "Apache-2.0",
"dependencies": {
"@balena/usbrelay": "^0.1.4",
"@serialport/parser-readline": "^12.0.0",
"bluebird": "^3.7.2",
"bluebird-retry": "^0.11.0",
"dbus-next": "^0.10.2",
Expand Down

0 comments on commit 3f9dbb1

Please sign in to comment.