User guide¶

This tool can be run on real hardware (FPGAs) or in a simulation mode. As the rowhammer attack exploits physical properties of cells in DRAM (draining charges), no bit flips can be observed in simulation mode. However, the simulation mode is useful to test command sequences during the development.

The Makefile can be configured using environmental variables to modify the network configuration used and to select the target. Currently, 4 boards are supported, each targeting different memory type:

For board-specific instructions refer to Arty A7, ZCU104, DDR4 Datacenter DRAM Tester, LPDDR4 Test Board, DDR5 Tester and DDR5 Test Board chapters. The rest of this chapter describes operations that are common for all supported boards.

Network USB adapter setup¶

In order to control the Rowhammer platform an Ethernet connection is necessary. In case you want to use an USB Ethernet adapter for this purpose read the instructions below.

1. Make sure you use a 1GbE USB network adapter

2. Figure out the MAC address for the USB network adapter:

   • Run sudo lshw -class network -short to get the list of all network interfaces

   • Check which of the devices uses the r8152 driver by running sudo ethtool -i <device>

   • Display the link information for the device running sudo ip link show <device> and look for the mac address next to the link/ether field

3. Configure the USB network adapter to appear as network device fpga0 using systemd

   • Create /etc/systemd/network/10-fpga0.link with the following contents:

     [Match]
     # Set this to the MAC address of the USB network adapter
     MACAddress=XX:XX:XX:XX:XX
     
     [Link]
     Name=fpga0
     

4. Configure the fpga0 network device with a static IP address, always up (even when disconnected) and ignored by the network manager.

   • Run the following command, assuming your system uses NetworkManager

     nmcli con add type ethernet con-name 'Rowhammer Tester' ifname fpga0 ipv4.method manual ipv4.addresses 192.168.100.100/24
     

   • Alternatively, if your system supports legacy interfaces configuration file:

     1. Make sure your /etc/network/interfaces file has the following line:

        source /etc/network/interfaces.d/*
        

     2. Create /etc/network/interfaces.d/fpga0 with the following contents:

        auto fpga0
        allow-hotplug fpga0
        iface fpga0 inet static
                address 192.168.100.100/24
        

     3. Check that nmcli device says the state is connected (externally) otherwise run sudo systemctl restart NetworkManager

   • Run ifup fpga0

5. Run sudo udevadm control --reload and then unplug the USB ethernet device and plug it back in

6. Check you have an fpga0 interface and it has the correct IP address by running networkctl status

Controlling the board¶

Board control is the same for both simulation and hardware runs. In order to communicate with the board via EtherBone, the litex_server needs to be started with the following command:

export IP_ADDRESS=192.168.100.50  # optional, should match the one used during build
make srv

The build files (CSRs address list) must be up to date. It can be re-generated with make without arguments.

Then, in another terminal, you can use the Python scripts provided. Remember to enter the Python virtual environment before running the scripts! Also, the TARGET variable should be set to load configuration for the given target. For example, to use the leds.py script, run the following:

source ./venv/bin/activate
export TARGET=arty  # (or zcu104) required to load target configuration
cd rowhammer_tester/scripts/
python leds.py  # stop with Ctrl-C

Hammering¶

Rowhammer attacks can be run against a DRAM module. It can be then used for measuring cell retention. For the complete list of scripts’ modifiers, see --help.

There are two versions of a rowhammer script:

• rowhammer.py - this one uses regular memory access via EtherBone to fill/check the memory (slower)

• hw_rowhammer.py - BIST blocks will be used to fill/check the memory (much faster, but with some limitations regarding fill pattern)

BIST blocks are faster and are the intended way of running Rowhammer tester.

Hammering of a row is done by reading it. There are two ways to specify a number of reads:

• --read_count N - one pass of N reads

• --read_count_range K M N - multiple passes of reads, as generated by range(K, M, N)

Regardless of which one is used, the number of reads in one pass is divided equally between hammered rows. If a user specifies --read_count 1000, then each row will be hammered 500 times.

Normally hammering is being performed via DMA, but there is also an alternative way with --payload-executor. It bypasses the DMA and directly talks with the PHY. That allows the user to issue specific activation, refresh and precharge commands.

Attack modes¶

Different attack and row selection modes can be used, but only one of them can be specified at the same time.

• --hammer-only

  Only hammers rows, without doing any error checks or reports. When run with rowhammer.py the attack is limited to one row pair. hw_rowhammer.py can attack up to 32 rows. With --payload-executor enabled the row limit is dictated by the payload memory size.

  For example following command will hammer rows 4 and 6 1000 times total (so 500 times each):

  (venv) $ python hw_rowhammer.py --hammer-only 4 6 --read_count 1000
  

• --all-rows

  Row pairs generated from range(start-row, nrows - row-pair-distance, row-jump) expression will be hammered.

  Generated pairs are of form (i, i + row-pair-distance). Default values for used arguments are:

  argument	default
  --start-row	0
  --row-jump	1
  --row-pair-distance	2

  So you can run following command to hammer rows (0, 2), (1, 3), (2, 4):

  (venv) $ python hw_rowhammer.py --all-rows --nrows 5
  

  And in case of:

  (venv) $ python hw_rowhammer.py --all-rows --start-row 10 --nrows 16 --row-jump 2 --row-distance 3
  

  hammered pairs would be: (10, 13), (12, 15).

  In a special case, where --row-pair-distance is 0, you can check how hammering a single row affects other rows. Normally activations and deactivations are achieved with row reads using the DMA, but in this case it is not possible. Because the same row is being read all the time, no deactivation command would be sent by the DMA. In this case, --payload-executor is required as it bypasses the DMA and sends deactivation commands on its own.

  (venv) $ python hw_rowhammer.py --all-rows --nrows 5 --row-pair-distance 0 --payload-executor
  

• --row-pairs sequential

  Hammers pairs of (start-row, start-row + n), where n is from 0 to nrows.

  (venv) $ python hw_rowhammer.py --row-pairs sequential --start-row 4 --nrows 10
  

  Command above, would hammer following set of row pairs:

  (4, 4 + 0)
  (4, 4 + 1)
  ...
  (4, 4 + 9)
  (4, 4 + 10)
  

• --row-pairs const

  Two rows specified with the const-rows-pair parameter will be hammered:

  (venv) $ python hw_rowhammer.py --row-pairs const --const-rows-pair 4 6
  

• --row-pairs random

  nrows pairs of random rows will be hammered. Row numbers will be between start-row and start-row + nrows.

  (venv) $ python hw_rowhammer.py --row-pairs random --start-row 4 --nrows 10
  

• --no-attack-time <time>

  Instead of performing a rowhammer attack, the script will load the RAM with selected pattern and sleep for time nanoseconds. After this time, it will check for any bitflips that could have happened. This option does not imply --no-refresh!

  (venv) $ python hw_rowhammer.py --no-attack-time 100e9 --no-refresh
  

Patterns¶

User can choose a pattern that memory will be initially filled with:

• all_0 - all bits set to 0

• all_1 - all bits set to 1

• 01_in_row - alternating 0’s and 1’s in a row (0xaaaaaaaa in hex)

• 01_per_row - all 0’s in odd-numbered rows, all 1’s in even rows

• rand_per_row - random values for all rows

Example output¶

(venv) $ python hw_rowhammer.py --nrows 512 --read_count 10e6 --pattern 01_in_row --row-pairs const --const-rows-pair 54 133 --no-refresh
Preparing ...
WARNING: only single word patterns supported, using: 0xaaaaaaaa
Filling memory with data ...
Progress: [========================================] 16777216 / 16777216
Verifying written memory ...
Progress: [========================================] 16777216 / 16777216 (Errors: 0)
OK
Disabling refresh ...
Running Rowhammer attacks ...
read_count: 10000000
  Iter 0 / 1 Rows = (54, 133), Count = 10.00M / 10.00M
Reenabling refresh ...
Verifying attacked memory ...
Progress: [========================================] 16777216 / 16777216 (Errors: 30)
Bit-flips for row    53: 5
Bit-flips for row    55: 11
Bit-flips for row   132: 12
Bit-flips for row   134: 3

Row selection examples¶

• Select row pairs from row 3 (--start-row) to row 59 (--nrows) where the next pair is 5 rows away (--row-jump) from the previous one:

  (venv) $ python hw_rowhammer.py --pattern 01_in_row --all-rows --start-row 3 --nrows 60 --row-jump 5 --no-refresh --read_count 10e4
  

• Select row pairs from row 3 to to row 59 without a distance between subsequent pairs (no --row-jump), which means that rows pairs are incremented by 1:

  (venv) $ python hw_rowhammer.py --pattern 01_in_row --all-rows --start-row 3 --nrows 60 --no-refresh --read_count 10e4
  

• Select all row pairs (from 0 to nrows - 1):

  (venv) $ python hw_rowhammer.py --pattern 01_in_row --all-rows --nrows 512 --no-refresh --read_count 10e4
  

• Select all row pairs (from 0 to nrows - 1) and save the error summary in JSON format to the test directory:

  (venv) $ python hw_rowhammer.py --pattern 01_in_row --all-rows --nrows 512 --no-refresh --read_count 10e4 --log-dir ./test
  

• Select only one row (42 in this case) and save the error summary in JSON format to the test directory:

  (venv) $ python hw_rowhammer.py --pattern all_1 --row-pairs const --const-rows-pair 42 42 --no-refresh --read_count 10e4 --log-dir ./test
  

• Select all rows (from 0 to nrows - 1) and hammer them one by one 1M times each.

  (venv) $ python hw_rowhammer.py --all-rows --nrows 100 --row-pair-distance 0 --payload-executor --no-refresh --read_count 1e6
  

Cell retention measurement examples¶

• Select all row pairs (from 0 to nrows - 1) and perform a set of tests for different read count values, starting from 10e4 and ending at 10e5 with a step of 20e4 (--read_count_range [start stop step]):

  (venv) $ python hw_rowhammer.py --pattern 01_in_row --all-rows --nrows 512 --no-refresh --read_count_range 10e4 10e5 20e4
  

• Perform set of tests for different read count values in a given range for one row pair (50, 100):

  (venv) $ python hw_rowhammer.py --pattern 01_in_row --row-pairs const --const-rows-pair 50 100 --no-refresh --read_count_range 10e4 10e5 20e4
  

• Perform set of tests for different read count values in a given range for one row pair (50, 100) and stop the test execution as soon as a bitflip is found:

  (venv) $ python hw_rowhammer.py --pattern 01_in_row --row-pairs const --const-rows-pair 50 100 --no-refresh --read_count_range 10e4 10e5 20e4 --exit-on-bit-flip
  

• Perform set of tests for different read count values in a given range for one row pair (50, 100) and save the error summary in JSON format to the test directory:

  (venv) $ python hw_rowhammer.py --pattern 01_in_row --row-pairs const --const-rows-pair 50 100 --no-refresh --read_count_range 10e4 10e5 20e4 --log-dir ./test
  

• Perform set of tests for different read count values in a given range for a sequence of attacks for different pairs, where the first row of a pair is 40 and the second one is a row of a number from range (40, nrows - 1):

  (venv) $ python hw_rowhammer.py --pattern 01_in_row --row-pairs sequential --start-row 40 --nrows 512 --no-refresh --read_count_range 10e4 10e5 20e4
  

Utilities¶

Some of the scripts are simple and do not take command line arguments, others will provide help via <script_name>.py --help or <script_name>.py -h.

Few of the scripts accept a --srv option. With this option enabled, a program will start it’s own instance of litex_server (the user doesn’t need to run make srv to control the board)

Run LEDs demo - leds.py¶

Displays a simple “bouncing” animation using the LEDs on Arty-A7 board, with the light moving from side to side.

-t TIME_MS or --time-ms TIME_MS option can be used to adjust LED switching interval.

Check version - version.py¶

Prints the data stored in the LiteX identification memory:

• hardware platform identifier

• source code git hash

• build date

Example output:

(venv) python version.py
Rowhammer tester SoC on xc7k160tffg676-1, git: e7854fdd16d5f958e616bbb4976a97962ee9197d 2022-07-24 15:46:52

Check CSRs - dump_regs.py¶

Dumps values of all CSRs. Example output of dump_regs.py:

0x82000000: 0x00000000 ctrl_reset
0x82000004: 0x12345678 ctrl_scratch
0x82000008: 0x00000000 ctrl_bus_errors
0x82002000: 0x00000000 uart_rxtx
0x82002004: 0x00000001 uart_txfull
0x82002008: 0x00000001 uart_rxempty
0x8200200c: 0x00000003 uart_ev_status
0x82002010: 0x00000000 uart_ev_pending
...

Initialize memory - mem.py¶

Before the DRAM memory can be used, the initialization and leveling must be performed. The mem.py script serves this purpose.

Expected output:

(venv) $ python mem.py
(LiteX output)
--========== Initialization ============--
Initializing SDRAM @0x40000000...
Switching SDRAM to software control.
Read leveling:
  m0, b0: |11111111111110000000000000000000| delays: 06+-06
  m0, b1: |00000000000000111111111111111000| delays: 21+-08
  m0, b2: |00000000000000000000000000000011| delays: 31+-01
  m0, b3: |00000000000000000000000000000000| delays: -
  m0, b4: |00000000000000000000000000000000| delays: -
  m0, b5: |00000000000000000000000000000000| delays: -
  m0, b6: |00000000000000000000000000000000| delays: -
  m0, b7: |00000000000000000000000000000000| delays: -
  best: m0, b01 delays: 21+-07
  m1, b0: |11111111111111000000000000000000| delays: 07+-07
  m1, b1: |00000000000000111111111111111000| delays: 22+-08
  m1, b2: |00000000000000000000000000000001| delays: 31+-00
  m1, b3: |00000000000000000000000000000000| delays: -
  m1, b4: |00000000000000000000000000000000| delays: -
  m1, b5: |00000000000000000000000000000000| delays: -
  m1, b6: |00000000000000000000000000000000| delays: -
  m1, b7: |00000000000000000000000000000000| delays: -
  best: m1, b01 delays: 22+-08
Switching SDRAM to hardware control.
Memtest at 0x40000000 (2MiB)...
  Write: 0x40000000-0x40200000 2MiB
  Read: 0x40000000-0x40200000 2MiB
Memtest OK
Memspeed at 0x40000000 (2MiB)...
  Write speed: 12MiB/s
  === Initialization succeeded. ===
Proceeding ...

Memtest (basic)
OK

Memtest (random)
OK

Enter BIOS - bios_console.py¶

Sometimes it may happen that memory initialization fails when running the mem.py script. This is most likely due to using boards that allow to swap memory modules, such as ZCU104.

Memory initialization procedure is performed by the CPU instantiated inside the FPGA fabric. The CPU runs the LiteX BIOS. In case of memory training failure it may be helpful to access the LiteX BIOS console.

If the script cannot find a serial terminal emulator program on the host system, it will fall back to litex_term which is shipped with LiteX. It is however advised to install picocom/minicom as litex_term has worse performance.

In the BIOS console use the help command to get information about other available commands. To re-run memory initialization and training type reboot.

Example:

(venv) $ python bios_console.py
LiteX Crossover UART created: /dev/pts/4
Using serial backend: auto
picocom v3.1

port is        : /dev/pts/4
flowcontrol    : none
baudrate is    : 1000000
parity is      : none
databits are   : 8
stopbits are   : 1
escape is      : C-a
local echo is  : no
noinit is      : no
noreset is     : no
hangup is      : no
nolock is      : no
send_cmd is    : sz -vv
receive_cmd is : rz -vv -E
imap is        :
omap is        :
emap is        : crcrlf,delbs,
logfile is     : none
initstring     : none
exit_after is  : not set
exit is        : no

Type [C-a] [C-h] to see available commands
Terminal ready
ad speed: 9MiB/s

--============== Boot ==================--
Booting from serial...
Press Q or ESC to abort boot completely.
sL5DdSMmkekro
            Timeout
No boot medium found

--============= Console ================--

litex>

Perform memory tests from the BIOS¶

After entering the BIOS, you may want to perform a memory test using utilities built into the BIOS itself. There are a couple ways to do such:

• mem_test - performs a series of writes and reads to check if values read back are the same as those previously written. It is limited by a 32-bit address bus, so only 4 GiB of address space can be tested. You can get origin of the RAM space using mem_list command.

• sdram_test - basically mem_test, but predefined for first 1/32 of defined RAM region size.

• sdram_hw_test - similar to mem_test, but accesses the SDRAM directly using DMAs, so it is not limited to 4 GiB. It requires passing 2 arguments (origin and size) with a 3rd optional argument being burst_length. When using sdram_hw_test you don’t have to offset the origin like in the case of mem_test. size is a number of bytes to test and burst_length is a number of full transfer writes to the SDRAM, before reading and checking written content. The default value for burst_length is 1, which means that after every write, a check is performed. Generally, bigger burst_length means faster operation.

Test with BIST - mem_bist.py¶

A script written to test BIST block functionality. Two tests are available:

• test-modules - memory is initialized and then a series of errors is introduced (on purpose). Then BIST is used to check the content of the memory. If the number of errors detected is equal to the number of errors introduced, the test is passed.

• test-memory - simple test that writes a pattern in the memory, reads it, and checks if the content is correct. Both write and read operations are done via BIST.

Run benchmarks - benchmark.py¶

Benchmarks memory access performance. There are two subcommands available:

• etherbone - measure performance of the EtherBone bridge

• bist - measure performance of DMA DRAM access using the BIST modules

Example output:

(venv) $  python benchmark.py etherbone read 0x10000 --burst 255
Using generated target files in: build/lpddr4_test_board
Running measurement ...
Elapsed = 4.189 sec
Size    = 256.000 KiB
Speed   = 61.114 KiBps

(venv) $  python benchmark.py bist read
Using generated target files in: build/lpddr4_test_board
Filling memory before reading measurements ...
Progress: [========================================] 16777216 / 16777216
Running measurement ...
Progress: [========================================] 16777216 / 16777216 (Errors: 0)
Elapsed = 1.591 sec
Size    = 512.000 MiB
Speed   = 321.797 MiBps

Use logic analyzer - analyzer.py¶

This script utilizes the Litescope functionality to gather debug information about signals in the LiteX system. In-depth Litescope documentation is here.

As you can see in Litescope documentation, Litescope analyzer needs to be instantiated in your design. Example design with analyzer added was provided as arty_litescope TARGET. As the name implies it can be run using Arty board. You can use rowhammer_tester/targets/arty_litescope.py as a reference for your own Litescope-enabled targets.

To build arty_litescope example and upload it to device, follow instructions below:

1. In root directory run:

   export TARGET=arty_litescope
   make build
   make upload
   

   analyzer.csv file will be created in the root directory.

2. We need to copy it to target’s build dir before using analyzer.py.

   cp analyzer.csv build/arty_litescope/
   

3. Then start litex-server with:

   make srv
   

4. And execute analyzer script in a separate shell:

   export TARGET=arty_litescope
   python rowhammer_tester/scripts/analyzer.py
   

   Results will be stored in dump.vcd file and can be viewed with gtkwave:

   gtkwave dump.vcd
   

Simulation¶

Select TARGET, generate intermediate files & run simulation:

export TARGET=arty # (or zcu104)
make sim

This command will generate intermediate files & simulate them with Verilator. After simulation has finished, a signals dump can be investigated using gtkwave:

gtkwave build/$TARGET/gateware/sim.fst

1. Create the TUN interface:

   tunctl -u $USER -t litex-sim
   

2. Configure the IP address of the interface:

   ifconfig litex-sim 192.168.100.1/24 up
   

3. Optionally allow network traffic on this interface:

   iptables -A INPUT -i litex-sim -j ACCEPT
   iptables -A OUTPUT -o litex-sim -j ACCEPT