UF2 works for data too.
The USB Flashing Format (UF2) was developed as a way to update firmware on microcontrollers, but it works equally well for data. Let’s look at how and why you might want to do that using the real world application of loading an FPGA.
The great thing about UF2 is that it emulates a USB thumb drive so images can be transferred by dragging and dropping the properly formatted file using the drivers that are standard in every operating system. The key is the specially formulated packets that you can read about in this blog:
https://makecode.com/blog/one-chip-to-flash-them-all
I also spoke about it at Crowd Supply Teardown 2023:
https://www.youtube.com/watch?v=AdoWZkSSaoA
While primarily created for firmware updates, it can transfer data just as easily. In the case of loading an FPGA, the firmware that loads the image into the FPGA does not need to change for each new image. One way to transfer the data is to convert the image to a C array and build it with the firmware, but that would require rebuilding the firmware for every new image which is not necessary. Instead of converting the image to a C array, you can convert it to UF2 packets so that it can be loaded directly into flash without rebuilding.
I did just this when I ported Altera’s JBC player code to the Raspberry Pi Pico board. The RP2040 implements a UF2 bootloader in its ROM, and the SDK produces a UF2 file when building the code. The same bootloader can be used to load an FPGA JBC image into the flash by converting the file to UF2 packets with the uf2conv utility.
The JBC player firmware needs to know where the JBC FPGA image is located, and the size. To make sure the firmware is compatible with the image, and make it easier to use, I decided to combine the firmware UF2 image with the JBC UF2 image. I created a custom UF2 converter that would take the firmware output from the RP2040 SDK, add a 256byte header with the JBC image size, and append the JBC image.
The header is 256 bytes to fill one UF2 block, so I added some additional information such as a description of the image. The output is a combined file similar to if you had included the JBC file as a C array, but in this case, to replace the JBC image, you do not need to have the RP2040 SDK installed, you only need to run a Python script.
You can learn more about the process and some quirks with the RP2040 ROM UF2 implementation in this Crowd Supply Teardown Session:
https://www.youtube.com/watch?v=Ns5RiT4F2Mw
The code is available on GitHub:
https://github.com/steieio/jbcuf2
And I used it in a workshop at Crowd Supply Teardown 2024:
https://github.com/gsteiert/pico-dirty-blaster-workshop
@steieio bit confused; isn't dfu the time-proven USB standard in that space?
@funkylab Great question. DFU was intended for this but everyone implements it differently and the drivers are not built into operating systems by default, so you need to write a custom application and build it for every operating system you want to support. MSC (Mass Storage Class) used by USB drives is supported in everything (even Android) without a custom app. The blog post I linked to explains why they developed UF2 at Microsoft
@funkylab @osmocom @LaF0rge I would argue that macOS and Android are even worse than Microsoft (iOS is fundamentally broken). It is very difficult to build an accessible platform agnostic solution with DFU. USB MSC can even work with Android or ChromeOS. That opens up cheap hardware that you could put in a middle school classroom.
@steieio @osmocom @LaF0rge I mean, you should be able to do DFU via WebUSB even, so I really am having a hard time agreeing, especially since "copy a file to an emulated storage device, but rely on the OS not doing out-of-order things or simultaneous file accesses, or checking what it wrote" is honestly just a terrible idea, as simple as it sounds to the user.
@funkylab @steieio @osmocom @LaF0rge iOS doesn’t have a USB framework accessible to third party apps, so that one’s definitely a no-go. iPadOS in theory provides DriverKit (at least on M1-M4 iPads) but you need code signing permission from Apple for every vendor ID, so there’s no way to implement a comprehensive generic solution.
I don’t see what the supposed issue with macOS is though, IOUSBLib doesn’t pose any major problems.
@pmdj @funkylab @osmocom @LaF0rge There is no issue with IOUSBLib, except that it is different than LInux, Windows and Android. The problem is that you have to develop a custom application on every platform you want to support. You are lucky if you get a DFU app for Windows and Linux. It is rare to see support for macOS let alone Android or ChromeOS.
@funkylab @osmocom @LaF0rge UF2 does not rely on the OS sending everything in order. Every packet includes the destination address so they can be sent in any order, as long as every packet is sent. I know Adafruit's TinyUF2 tolerates that, but I'm not sure the RP2040 ROM implemented that based on my experience concatenating data to a firmware image.
@piggo @funkylab I agree with the fragility of the mbed implementation, and so did Michal Moksal who came up with UF2. He came up with some clever ways to get the data through in spite of the OS. His blog post "One chip to flash them all" describes the motivation and method: https://makecode.com/blog/one-chip-to-flash-them-all
@steieio it does make a lot of sense from a Microsoft perspective, where software distribution still seems to be hard, indeed, and they for some reason don't like generic bulk device drivers. I've not had the worst experiences with dfu-util myself (thanks, openmoko/ @LaF0rge / @osmocom), and would honestly just have wished for more self- descriptor functionality bolted onto that protocol.