Firmware update via fwupd

[fbruetting]

Would it be possible, to make (all, even older) firmware versions available via fwupd? This way, we Linux users could update/downgrade firmware VERY easily and comfortable! :D

[JarrettR]

Seems that that would eliminate Windows support?

[fbruetting]

Not at all. Primarily, it would just enhance the Linux support and Windows procedure won’t be touched. BUT: You could run a Live Linux either inside a VM or via USB stick (or install it into a VM), start Gnome Software and update Bus Pirate there easily, too! So Windows support would also be enhanced! (And in fwupd, you can also downgrade.)

Furthermore, Bus Pirate would get a lot of promotion, because at the moment, there aren’t a lot vendors providing firmware via fwupd, so Bus Pirate would stand out. ;-)

I now get UEFI upgrades and Logitech receiver updates via just starting the Gnome AppStore and cklicking on update UEFI etc. That’s incredible for firmware updates, since I assume a lot of people don’t upgrade firmware, because it’s too complicated or they don’t want to accidentally brick their device.

[agatti]

Unfortunately this is not as easy as it might look like. Already-existing units probably have one of the following two issues that might prevent automated firmware installation: #38 and #69 (comment) . The latter may require having to clean the flash with an external PIC programmer to be worked around.

I'll still take a look at what fwupd can offer but considering that the devices pool that can make use of this is not really that big, I'm not sure whether this can be ever done.

[fbruetting]

Maybe you can ask the dev of ColorHUG (@hughsie) for advice – this is also just a microcontroller, which doesn’t seem to be that popular. :)

[agatti]

Perhaps I expressed myself incorrectly. The main issue here is that there is no way to be able to provide automated updates for Bus Pirate boards for more than 90% of the boards that are currently deployed due to a combination of problems in various releases of firmware/bootloaders that were installed before v7.1 (which is currently not yet released due to one of these issues blocking the whole lot).

Every board that has been sold and loaded with the official Dangerous Prototypes' firmware in the past few years cannot be updated without either shorting the PGC/PGD pins and attempt a firmware upload (which not all versions have it working!) or via a PICKit. The bootloader as it was (and still is), requires a board reset after reporting the board revision/firmware version (#70), and fixing that probably requires changing the firmware upload protocol making it incompatible with existing tools. This is quite critical as we do not have two assigned VID:PID pairs to uniquely identify the hardware, and adding metadata in the USB information packets is not an option for Bus Pirate v3 boards as they use a FTDI USB to UART adapter - hence no injection opportunity. The latter for v4 requires switching USB stack (#67), which is not something to be taken lightly either.

Moreover, Bus Pirate v3 boards have a quite confusing set of bootloaders/memory layouts with the real risk of bricking the device if the wrong update procedure sequence is performed, requiring clearing the flash memory via a PICKit or similar hardware tool and flash the latest bootloader to be able to flash the updated firmware. Bus Pirate v4 boards are a bit more sane on that regard but the latest Dangerous Prototypes' firmware has a number of issues making even the PGC/PGD pin short method not working for firmware updates.

So, we've got quite a long way to go to be able to integrate with fwupd. If external contributors are willing to step up and help maybe something can be done, one step at a time. Right now it's pretty much just me handling firmware maintenance and development, and there are bigger issues for this project for me to focus my not-so-copious spare time on.

That said, that project is quite interesting and it does make a lot of sense in improving the user experience when dealing with custom hardware. Hopefully more and more individual hardware vendors will take on that and make life easier for all of us :)

[hughsie]

That said, that project is quite interesting

Thanks! Could there be a plan going forward, i.e. designing future bootloader/firmware versions with fwupd support in mind? Updating using fwupd has to be 100% safe; we can't show the user a dialog saying "you need to go buy a hardware programmer..." so maybe trying to retrofit this into older bootloader/firmware versions isn't going to work. Out of interest, what firmware update protocol is being used in the new firmwares? If it's DFU it'll just work, anything custom will need some new code in fwupd.

[agatti]

Theoretically speaking, any board with firmware version 7.1 and the latest matching bootloader for its hardware type should be able to handle something like fwupd via an expect-like mechanism to be able to jump into bootloader mode and then talk directly over the serial port as if it was brought up in firmware update mode to begin with.

Right now the firmware uploading method is based on the open source version of ds30. DFU cannot be deployed on the more popular board type because it simply does not have an USB stack running (it sports an FT232R for USB<>Serial). The other type with a native USB stack should be able to handle it once the current USB stack is replaced with either Microchip's or with Signal11's.

Right now a Bus Pirate v4 is announced by the USB stack as 04D8:FB00:0002 (VID/PID/DEV) which I do not know if it's actually unique. I do not have a v3 board right now to see what is reported, but I have to check the schematics for the board to see if the current hardware allows to customise the VID/PID/DEV tuple reported by the FTDI adapter (but I'm not really confident that's possible).

[agatti]

Ok, turns out that the VID:PID pair associated with the Bus Pirate v4 seems to be indeed unique, according to this page, but the Bus Pirate v3 - whilst it can be configured with a custom VID:PID pair using FTDI's MPROG tool - does not have one assigned.

[hughsie]

Do you have any documentation on the protocol used in ds30? Thanks.

[agatti]

I haven't found any formal document but here's what I could surmise by looking at the source code of the firmware upload programs that come with the Bus Pirate. Please keep in mind that I don't know if anything has been altered from the original DS30 bootloader protocol, I'll take a look later for that, but hopefully this should get you going for the time being.

Handshake:

Write 0xC1 (1 byte)

Read [A] [B] [C] [D] (4 bytes)
A : Chip ID (f.ex. 0xD4 for Bus Pirate v3, 0xF1 for Bus Pirate v4)
B : Bootloader version major
C : Bootloader version minor
D : Result (0x4B means success, other means failure)

Erase Page command:

Write [A] [B] [C] 0x01 0x01 [D] (6 bytes)
A : Address bits 16-20
B : Address bits 8-16
C : Address bits 0-8
D : Command bytes 0 to 5 CRC (see below)

Read [A] [B] [C] [D] (4 bytes)
A : Result (0x4B means success, other means failure)
B : Unused
C : Unused
D : Unused

Write Data Row command:

Write [A] [B] [C] 0x02 0xC1 [D] [E] (198 bytes)
A : Address bits 16-20
B : Address bits 8-16
C : Address bits 0-8
D : 192 bytes worth of data to write
E : Command bytes 0 to 197 CRC (see below)

Read [A] [B] [C] [D] (4 bytes)
A : Result (0x4B means success, other means failure)
B : Unused
C : Unused
D : Unused

CRC Calculation:

uint8_t calculate_crc(uint8_t *buffer, size_t length) {
	uint8_t crc = 0;
	for (size_t offset = 0; offset < length; offset++) {
		crc -= *buffer++;
	}
	
	return crc;
}

Flashing operation sequence:

• Perform handshake, bail out if not successful.
• For each page (1536 bytes):
  • Send Erase Page command, bail out if not successful.
  • For each row in page (192 bytes) - 8 rows per page, hence 8 times:
    • Send Write Data Row command, bail out if not successful.

[hughsie]

That seems simple enough -- would it be possible to get a Bus Pirate so that I can write a fwupd plugin for the DS30 protocol? It doesn't have to actually work as a Bus Pirate, I just need the chip and the USB socket for https://blogs.gnome.org/hughsie/2017/11/07/hardware-ci-tests-in-fwupd/ . If this sounds like something you guys can sort, please grab me off-issue (richard_at_hughsie_dot_com) and I'll send you my address. Alternatively, I'd be happier still if someone with a Bus Pirate could contribute the missing plugin to fwupd themselves :)

[agatti]

Unfortunately the project itself has been effectively abandoned by the original designer, and whilst the hardware is currently being sold worldwide, this project is the only effort in bringing the firmware forward that we are aware of. Due to the apparent lack of interest from board resellers, we are using our own boards for development, after being dissatisfied on how the project was left to rot :| I am still relying on users to report issues on v3 boards since I don't have any of those around here, for example.

My old v4 board is currently en route to Europe so I cannot really test this for the next two to three weeks, but once I get hold of it I can easily add support for the board, albeit with a few caveats as described earlier. I might nag you on how to achieve some stuff though :)

Still, if somebody else with the necessary hardware is willing to step up and take on this, contributions are always welcome.

[hughsie]

I might nag you on how to achieve some stuff though

That's no problem at all. If you do IRC, we're #fwupd on freenode, or we have a mailing list too: https://groups.google.com/forum/#!forum/fwupd -- we're all friendly, so don't be afraid about asking questions. We have some initial docs about writing plugins here too: https://people.freedesktop.org/~hughsient/fwupd/plugin-reference.html

If you get stuck with compiling fwupd, please also let me know, although it should be pretty easy on any new-ish Linux.

[agatti]

Got it. That's something that I'll look at once I'm back to Europe and the 7.1 firmware is wrapped up.