NORbert: SPI bus logging and TOCTOU attacks

spi-flash-tool monitor decodes the bus in real time:

TXN#   COMMAND            ADDRESS    INFO
------------------------------------------------------------
1      0x9F READ_JEDEC_ID
2      0x05 READ_STATUS
3      0x05 READ_STATUS
4      0xBB DUAL_IO_READ  0x001000
       end: 4097 bytes from 0x001000
5      0x9F READ_JEDEC_ID
6      0x05 READ_STATUS
7      0x05 READ_STATUS
8      0xBB DUAL_IO_READ  0x001000  ** DOUBLE READ (TOCTOU candidate)
  !! TOCTOU TRAP #1 FIRED at 0x001000 -- serving replacement data
       end: 4097 bytes from 0x001000

That's two back-to-back flashprog reads of the same 4 KB region. The first read arms the trap; the second read gets redirected to a completely different address in SDRAM. From flashprog's perspective the first read returned INNOCENT-0001- over and over and the second read returned MALICIOUS0101-.

The logger lives in src/logger.v. It takes four event signals out of spi_trx.v – all in the SPI clock domain – and synchronizes them through 3-FF synchronizers with edge detection into the system clock domain:

• spi_log_cmd_valid – pulse when the command byte is decoded
• spi_log_addr_valid – pulse when the address phase completes (for reads, writes and erases in all single/dual/quad I/O modes)
• spi_active – level, follows CS (deselect edge is the transaction-end event)
• trap_notify_strobe – pulse from the TOCTOU engine in glue.v when a redirect fires

Each event gets captured into a pending flag plus latched payload and serialized into a short binary packet in a 512-byte ring FIFO. The packet type bytes are all in the 0xA1-0xA4 range so they don't collide with the regular protocol ACKs (0x00, 0x01, 0x02):

0xA1 <opcode>                                  - SPI command decoded       (2 bytes)
0xA2 <addr3> <addr2> <addr1> <addr0>           - Address phase complete    (5 bytes)
0xA3 <count2> <count1> <count0>                - Transaction end + count   (4 bytes)
0xA4 <index> <addr3> <addr2> <addr1> <addr0>   - TOCTOU trap fired         (6 bytes)

host -> FPGA:  0x3A
FPGA -> host:  <log byte 0> <log byte 1> ... <log byte N-1> 0xA0

while running.load(Ordering::SeqCst) {
    let data = device.log_poll()?;
    if data.is_empty() {
        thread::sleep(Duration::from_millis(5));
        continue;
    }
    parse(&data);
}

The trap engine sits in glue.v and has four entries. Each entry is three 24-bit byte addresses:

• trap_start – the match value
• trap_mask – 1-bits must match, 0-bits are don't-care
• trap_replace – the replacement address base

On every log_addr_valid pulse, glue compares the read address against all four armed entries:

if (trap_armed[i] &&
    ((log_addr_sync & trap_mask[i]) == (trap_start[i] & trap_mask[i]))) begin
    if (!trap_triggered[i]) begin
        // First match: mark triggered, serve original data
        trap_triggered[i] <= 1;
    end
    else begin
        // Second+ match: activate address redirect
        redirect_active <= 1;
        redirect_mask   <= trap_mask[i][23:3];
        redirect_base   <= trap_replace[i][23:3];
        trap_notify_strobe <= 1;
        trap_notify_index  <= i[1:0];
        trap_notify_addr   <= log_addr_sync;
    end
end

The redirect is a combinational mux on the SDRAM burst address in top.v:

wire [22:0] spi_ram_addr_final = redirect_active ?
    ((redirect_base & redirect_mask) | (spi_ram_addr & ~redirect_mask)) :
    spi_ram_addr;

redirect_active is cleared when CS deasserts, so the trap only affects the current SPI transaction. Nothing about the SDRAM contents changes – it's purely an address rewrite on the fast path between spi_trx and the SDRAM controller.

0x01 SET index start_hi start_mid start_lo mask_hi mask_mid mask_lo \
         replace_hi replace_mid replace_lo     -- configure a trap entry
0x02 ARM index                                 -- start monitoring
0x03 DISARM index                              -- stop monitoring
0x04 RESET index                               -- clear triggered flag
0x05 RESET_ALL                                 -- disarm + clear all four

Hardware: a Tang Primer 25K with dock (for NORbert), plus an FT4222H as the SPI master for flashprog. The FT4222H's SPI pins go to NORbert's PMOD J5 (CS/CLK/IO0-3). You could equally well use a standalone FT2232H with -p ft2232_spi:type=2232H or any other flashprog-supported SPI master – it's just what I had hooked up.

git clone https://github.com/ArthurHeymans/NORbert
cd NORbert
nix develop          # or set up Gowin IDE, openFPGALoader, rustc by hand
make build
make prog            # volatile, or `make flash` for persistent
make tool

python3 -c "
img = bytearray(b'\xff' * 0x800000)
for sector in range(0x800):
    region = 'MALICIOUS' if 0x100 <= sector < 0x200 else 'INNOCENT-'
    pattern = f'{region}{sector:04X}-'
    img[sector*0x1000 : (sector+1)*0x1000] = \
        (pattern.encode() * (0x1000 // len(pattern) + 1))[:0x1000]
open('image.bin', 'wb').write(img)
"

00000000:00000fff bootblock
00001000:00001fff region1
00100000:00100fff evil_bootblock

PORT="-p /dev/ttyUSB1"          # Sipeed dock's UART channel
TOOL="./tool/target/release/spi-flash-tool"

$TOOL $PORT version              # should print "Protocol version: 4"
$TOOL $PORT load image.bin       # loads + starts emulation

# Trap 1: any read where addr[23:12] == 0x001 redirects to 0x101xxx.
# The low 12 bits are preserved, the high 12 are forced to 0x101.
$TOOL $PORT toctou set 1 0x001000 0xFFF000 0x101000
$TOOL $PORT toctou arm 1

$TOOL $PORT monitor

flashprog -p ft4222_spi -c "W25Q64BV/W25Q64CV/W25Q64FV" \
    -l layout.txt -i region1 -r read1.bin
flashprog -p ft4222_spi -c "W25Q64BV/W25Q64CV/W25Q64FV" \
    -l layout.txt -i region1 -r read2.bin

$ dd if=read1.bin bs=1 count=48 skip=4096 2>/dev/null; echo
INNOCENT-0001-INNOCENT-0001-INNOCENT-0001-INNOCE
$ dd if=read2.bin bs=1 count=48 skip=4096 2>/dev/null; echo
INNOCENTS0101-MALICIOUS0101-MALICIOUS0101-MALICI

$TOOL $PORT toctou reset-all
$TOOL $PORT stop

Mostly: characterizing a target SoC's boot ROM / bootloader. If you see the boot ROM read the same image twice, the trap primitive tells you whether there's a verify-then-use window you can exploit. If you see a single read, you know the target assumes flash is stable and TOCTOU isn't interesting on that code path. Either way, the logger gives you the sequence of commands, the addresses, and the byte counts to correlate against the ROM's behavior.

It also works as a cheap bus analyzer. The packet format is small enough that you can stream it at full FT245 rate if needed, and the 0xA1=/=0xA2=/=0xA3 packets give you transaction boundaries for free. For now I'm polling at ~200 Hz over UART, which is plenty for flashprog but would be the bottleneck for a live boot trace – that's what the FT245 path is for.

The trap engine has four independent entries so you can chain triggers, e.g. arm trap 0 on a config-block read and trap 1 on the boot image, and only flip the boot image once the config probe has happened. The on-chip logic is tiny; adding more entries is just a few LUTs per slot.

Everything is on GitHub: NORbert. The relevant files for this post are src/logger.v, src/glue.v (trap engine, LOGPOLL state machine), src/top.v (address redirect mux), and the monitor + toctou subcommands in tool/src/main.rs.