Update (2014-07): There is a new RPi model B+ with 4 USB ports (well 5 in fact when you include ethernet using a LAN9514 chip instead of 3 ports using the LAN9512 in the model B). Just to be clear the tests below were done on the model B. If and when I repeat the the tests with a model B+ I do not expect to see better performance.
Hardware Configuration:
Raspberry Pi model B - 512MB model, 128MB of RAM allocated to GPU, no overclockingInbuilt 10/100 NIC is connected.
Top USB port is a 2TB USB hard disk drive.
Bottom USB port is empty.
So no keyboard, no mouse. In an ideal world we would have the inbuilt USB NIC disabled (after setting the date and time) as well, but it appears that the RPi has the network module compiled directly into the kernel.
pi@raspberrypi ~ $ zgrep -i smsc95xx /proc/config.gz
CONFIG_USB_NET_SMSC95XX=y
pi@raspberrypi ~ $ sudo modprobe -r smsc95xx
FATAL: Module smsc95xx is builtin.
So to just have a hard disk and RPi (and internal 3 port USB hub) would require me to recompile the kernel, which would make the results slightly off standard
The HDD that I'm using has an Internal write rate of about 100MiB/sec, so USB throughput should be the cause of any performance bottleneck.
Prerequisites:
sudo apt-get install at expect expect-devext2, ext3, ext4, vfat file system write tests
I split a 2 TB disk into four equal sized partitions.
pi@raspberrypi ~ $ lsusb -t
/: Bus 01.Port 1: Dev 1, Class=root_hub, Driver=dwc_otg/1p, 480M
|__ Port 1: Dev 2, If 0, Class=hub, Driver=hub/3p, 480M
|__ Port 1: Dev 3, If 0, Class=vend., Driver=smsc95xx, 480M
|__ Port 2: Dev 4, If 0, Class=stor., Driver=usb-storage, 480M
pi@raspberrypi ~ $ sudo fdisk -l /dev/sda
Disk /dev/sda: 2000.4 GB, 2000398934016 bytes
255 heads, 63 sectors/track, 243201 cylinders, total 3907029168 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x1ba9000c
Device Boot Start End Blocks Id System
/dev/sda1 2048 976758827 488378390 83 Linux
/dev/sda2 976758828 1953515607 488378390 83 Linux
/dev/sda3 1953515608 2930272387 488378390 83 Linux
/dev/sda4 2930272388 3907029167 488378390 c W95 FAT32 (LBA)
pi@raspberrypi ~ $ sudo mkfs.ext2 -L ext2 /dev/sda1
pi@raspberrypi ~ $ sudo mkfs.ext3 -L ext3 /dev/sda2
pi@raspberrypi ~ $ sudo mkfs.ext4 -L ext4 /dev/sda3
pi@raspberrypi ~ $ sudo mkfs.vfat -n vfat /dev/sda4
pi@raspberrypi ~ $ cd /media
pi@raspberrypi /media $ sudo mkdir ext2 ext3 ext4 vfat
pi@raspberrypi ~ $ at midnight
warning: commands will be executed using /bin/sh
at> /home/pi/write-throughputtest.exp 2>&1 >/home/pi/at.output
at> ^D
pi@raspberrypi ~ $
pi@raspberrypi ~ $ cat write-throughputtest.exp
#!/usr/bin/expect -f
#
# This Expect script was generated by autoexpect on Sun Mar 30 16:43:25 2014
set force_conservative 0 ;# set to 1 to force conservative mode even if
;# script wasn't run conservatively originally
if {$force_conservative} {
set send_slow {1 .1}
proc send {ignore arg} {
sleep .1
exp_send -s -- $arg
}
}
set timeout -1
spawn $env(SHELL)
match_max 100000
expect "$"
send -- "/bin/bash"
expect "pi@raspberrypi"
send -- "sudo /bin//bash\r"
expect "root@raspberrypi"
send -- "script /home/pi/output-root.txt\r"
expect "Script started, file is /home/pi/output-root.txt\r"
send -- "mount /dev/sda1 /media/ext2\r"
expect "root@raspberrypi"
send -- "mount /dev/sda2 /media/ext3\r"
expect "root@raspberrypi"
send -- "mount /dev/sda3 /media/ext4\r"
expect "root@raspberrypi"
send -- "mount /dev/sda4 /media/vfat\r"
expect "root@raspberrypi"
send -- "/home/pi/write-throughputtest.sh /media/ext2\r"
expect "root@raspberrypi"
send -- "/home/pi/write-throughputtest.sh /media/ext3\r"
expect "root@raspberrypi"
send -- "/home/pi/write-throughputtest.sh /media/ext4\r"
expect "root@raspberrypi"
send -- "/home/pi/write-throughputtest.sh /media/vfat\r"
expect "root@raspberrypi"
send -- "exit\r"
expect "Script done, file is /home/pi/output-root.txt\r"
send -- "exit\r"
expect "pi@raspberrypi"
send -- "exit\r"
expect "$"
send -- "exit\r"
expect eof
pi@raspberrypi ~ $ cat write-throughputtest.sh
#!/bin/sh
DEST=$1
# 18 x 1GiB = 18GiB
# assuming maximum write 40MiB/sec this should take 8 minutes
# assuming poor write speed of 5 MiB/sec this should take 2 hours
# either way one pass should happen within 12 hours
echo touch $DEST/testbegin-write-starttime
touch $DEST/testbegin-write-starttime
date
echo sync
sync
date
for BS in 8 16 32 64 128 256 \
512 1024 2048 4096 8192 16384 \
32768 65536 131072 262144 524288 1048576
do
COUNT=`expr 1073741824 / $BS`
echo touch $DEST/${BS}b-write-starttime
touch $DEST/${BS}b-write-starttime
date
echo dd if=/dev/zero of=$DEST/${BS}b-data bs=${BS} count=${COUNT}
dd if=/dev/zero of=$DEST/${BS}b-data bs=${BS} count=${COUNT}
date
echo touch $DEST/${BS}b-write-endtime
touch $DEST/${BS}b-write-endtime
date
echo sync
sync
date
# any delay caused by sync will show up on next starttime
done
echo touch $DEST/testover-write-endtime
touch $DEST/testover-write-endtime
date
pi@raspberrypi ~ $ cat read-throughputtest.sh
#!/bin/sh
SRCE=$1
# 18 x 1GiB = 18GiB
# assuming maximum read 40MiB/sec this should take 8 minutes
# assuming poor read speed of 5 MiB/sec this should take 2 hours
# This script assumes that write-throughputtest.sh was run
# previously to generate the data files.
date
echo sync
sync
date
for BS in 8 16 32 64 128 256 \
512 1024 2048 4096 8192 16384 \
32768 65536 131072 262144 524288 1048576
do
COUNT=`expr 1073741824 / $BS`
date
echo dd of=/dev/null if=$SRCE/${BS}b-data bs=${BS} count=${COUNT}
dd of=/dev/null if=$SRCE/${BS}b-data bs=${BS} count=${COUNT}
date
echo sync
sync
date
done
date
pi@raspberrypi ~ $
Results
USB disk performance check Raspberry Pi normal USB (internal 3 port hub)
Raspberry Pi Model B USB disk write results
-----------------------------------------------------------
block filesystem type
size -------------------------------------------
(bytes)ext2 ext3 ext4 vfat
8 0.2 MiB/s 0.3 MiB/s 0.3 MiB/s 0.8 MiB/s
16 0.4 MiB/s 0.5 MiB/s 0.6 MiB/s 1.5 MiB/s
32 0.9 MiB/s 1.0 MiB/s 1.1 MiB/s 2.9 MiB/s
64 2.5 MiB/s 1.8 MiB/s 1.9 MiB/s 4.4 MiB/s
128 4.7 MiB/s 3.2 MiB/s 3.5 MiB/s 6.3 MiB/s
256 7.4 MiB/s 5.5 MiB/s 6.1 MiB/s 9.4 MiB/s
512 10.6 MiB/s 8.5 MiB/s 9.5 MiB/s 12.2 MiB/s
1k 14.9 MiB/s 11.3 MiB/s 13.8 MiB/s 13.1 MiB/s
2k 18.4 MiB/s 14.2 MiB/s 18.6 MiB/s 15.1 MiB/s
4k 20.8 MiB/s 17.7 MiB/s 22.5 MiB/s 17.0 MiB/s
8k 21.3 MiB/s 18.6 MiB/s 22.8 MiB/s 16.7 MiB/s
16k 21.3 MiB/s 19.0 MiB/s 22.4 MiB/s 17.0 MiB/s
32k 21.3 MiB/s 19.0 MiB/s 22.5 MiB/s 16.9 MiB/s
64k 20.7 MiB/s 18.7 MiB/s 22.9 MiB/s 16.9 MiB/s
128k 21.3 MiB/s 18.1 MiB/s 22.7 MiB/s 16.4 MiB/s
256k 21.3 MiB/s 18.4 MiB/s 22.4 MiB/s 16.5 MiB/s
512k 21.6 MiB/s 18.9 MiB/s 22.3 MiB/s 16.8 MiB/s
1M 20.9 MiB/s 19.2 MiB/s 22.4 MiB/s 16.9 MiB/s
For the small block sizes there is almost no disk activity, long pauses with no blinking RED LED's.
I'm begining to wonder if some options in Raspbian have been enabled to extend the life of the RPi SSD.
Raspberry Pi Model B USB disk read results
-----------------------------------------------------------
block filesystem type
size -------------------------------------------
(bytes)ext2 ext3 ext4 vfat
8 1.6 MiB/s 2.0 MiB/s 1.7 MiB/s 1.9 MiB/s
16 3.9 MiB/s 3.5 MiB/s 3.6 MiB/s 3.8 MiB/s
32 6.9 MiB/s 6.7 MiB/s 7.2 MiB/s 6.8 MiB/s
64 10.5 MiB/s 10.4 MiB/s 10.2 MiB/s 10.4 MiB/s
128 18.0 MiB/s 16.5 MiB/s 15.2 MiB/s 17.0 MiB/s
256 23.3 MiB/s 22.6 MiB/s 22.1 MiB/s 22.2 MiB/s
512 23.2 MiB/s 23.2 MiB/s 23.4 MiB/s 21.7 MiB/s
1k 23.6 MiB/s 23.2 MiB/s 23.6 MiB/s 22.0 MiB/s
2k 23.7 MiB/s 23.2 MiB/s 23.6 MiB/s 22.3 MiB/s
4k 23.6 MiB/s 22.9 MiB/s 23.6 MiB/s 23.3 MiB/s
8k 22.6 MiB/s 22.3 MiB/s 23.7 MiB/s 23.7 MiB/s
16k 23.6 MiB/s 23.6 MiB/s 23.7 MiB/s 23.7 MiB/s
32k 22.4 MiB/s 23.6 MiB/s 22.6 MiB/s 23.7 MiB/s
64k 22.5 MiB/s 23.6 MiB/s 22.7 MiB/s 23.7 MiB/s
128k 23.7 MiB/s 22.4 MiB/s 22.8 MiB/s 23.7 MiB/s
256k 23.7 MiB/s 22.5 MiB/s 23.8 MiB/s 23.7 MiB/s
512k 23.7 MiB/s 22.6 MiB/s 23.7 MiB/s 23.7 MiB/s
1M 23.7 MiB/s 22.4 MiB/s 23.7 MiB/s 23.7 MiB/s
USB disk performance check (midrange desktop PC) - dedicated USB port
For comparison with the above Raspberry Pi results , here are desktop PC results:midrange desktop PC disk write results (dedicated HS USB 2.0 port)
------------------------------------------------------------------
block filesystem type
size -------------------------------------------
(bytes)ext2 ext3 ext4 vfat
8 6.3 MiB/s 3.8 MiB/s 3.9 MiB/s 5.4 MiB/s
16 12.9 MiB/s 7.6 MiB/s 7.8 MiB/s 11.1 MiB/s
32 26.1 MiB/s 15.0 MiB/s 15.9 MiB/s 19.4 MiB/s
64 34.9 MiB/s 21.1 MiB/s 31.7 MiB/s 23.3 MiB/s
128 34.9 MiB/s 21.8 MiB/s 37.7 MiB/s 24.0 MiB/s
256 34.2 MiB/s 22.2 MiB/s 37.5 MiB/s 23.4 MiB/s
512 34.1 MiB/s 22.7 MiB/s 37.3 MiB/s 23.5 MiB/s
1k 34.1 MiB/s 22.8 MiB/s 37.1 MiB/s 24.7 MiB/s
2k 33.8 MiB/s 22.7 MiB/s 37.5 MiB/s 24.5 MiB/s
4k 34.0 MiB/s 22.3 MiB/s 37.2 MiB/s 24.4 MiB/s
8k 34.1 MiB/s 22.7 MiB/s 37.4 MiB/s 24.9 MiB/s
16k 35.1 MiB/s 22.9 MiB/s 37.5 MiB/s 23.7 MiB/s
32k 36.0 MiB/s 22.7 MiB/s 37.5 MiB/s 25.4 MiB/s
64k 35.4 MiB/s 23.5 MiB/s 37.6 MiB/s 24.8 MiB/s
128k 35.3 MiB/s 23.1 MiB/s 37.2 MiB/s 24.4 MiB/s
256k 35.6 MiB/s 23.0 MiB/s 37.6 MiB/s 25.5 MiB/s
512k 35.9 MiB/s 23.3 MiB/s 37.6 MiB/s 25.2 MiB/s
1M 35.9 MiB/s 22.7 MiB/s 37.6 MiB/s 25.2 MiB/s
For High Speed USB 2.0 the maximum data rate is about 40MB/sec (38.1MiB/sec) None of the tested filesystems hit this but ext4 did come closest, but not on the Raspberry Pi unfortunately.
('lsusb -t' does shows 480M, which in theory would correspond to 57.2MiB/sec, but there protocol overheads within USB. The maximum should be 53.248 MB/s, but this appears to be deliberately limited to around 30-42MB/sec to maintain interoperability with USB stacks in OS'es over throughput speed).
midrange desktop PC disk read results (dedicated HS USB 2.0 port)
-----------------------------------------------------------------
block filesystem type
size -------------------------------------------
(bytes)ext2 ext3 ext4 vfat
8 18.4 MiB/s 16.8 MiB/s 18.4 MiB/s 18.6 MiB/s
16 32.5 MiB/s 29.3 MiB/s 36.6 MiB/s 35.0 MiB/s
32 32.5 MiB/s 29.0 MiB/s 35.9 MiB/s 34.5 MiB/s
64 33.6 MiB/s 30.4 MiB/s 36.2 MiB/s 35.6 MiB/s
128 33.3 MiB/s 29.9 MiB/s 36.0 MiB/s 35.0 MiB/s
256 32.7 MiB/s 30.3 MiB/s 36.1 MiB/s 34.8 MiB/s
512 33.4 MiB/s 29.9 MiB/s 36.0 MiB/s 35.0 MiB/s
1k 33.5 MiB/s 30.2 MiB/s 36.2 MiB/s 35.9 MiB/s
2k 33.7 MiB/s 30.7 MiB/s 36.4 MiB/s 34.7 MiB/s
4k 34.0 MiB/s 30.5 MiB/s 36.1 MiB/s 35.9 MiB/s
8k 34.0 MiB/s 30.8 MiB/s 36.4 MiB/s 35.4 MiB/s
16k 34.0 MiB/s 30.9 MiB/s 36.4 MiB/s 35.5 MiB/s
32k 34.0 MiB/s 30.7 MiB/s 36.5 MiB/s 35.2 MiB/s
64k 33.7 MiB/s 30.4 MiB/s 36.4 MiB/s 35.2 MiB/s
128k 33.5 MiB/s 30.7 MiB/s 36.4 MiB/s 36.0 MiB/s
256k 33.4 MiB/s 30.5 MiB/s 36.4 MiB/s 34.8 MiB/s
512k 33.5 MiB/s 30.5 MiB/s 36.1 MiB/s 35.8 MiB/s
1M 33.2 MiB/s 30.5 MiB/s 36.5 MiB/s 35.7 MiB/s
I really wonder what the USB performance is for a Model A, which does not have an internal 3 port USB hub inside with a USB NIC connected to one of the ports. For all of the tests above there was basically no network traffic to/from the RPi, and even with that the performance of the USB is not all that great. But even so it does appear that USB is the fastest way to get data into and out of the Raspberry Pi.
Internal disk performance check (midrange desktop PC)
I reconnected the same 2TB disk to a dedicated 1.5Gbit/sec eSATA port on the back of the desktop PC just to confirm that USB was in fact the bottleneck for data transfers. And to verify that there were no problems with the hard disk that I used.midrange desktop PC disk write results (1.5 Gbit/sec eSATA port)
------------------------------------------------------------------
block filesystem type
size -------------------------------------------
(bytes)ext2 ext3 ext4 vfat
8 7.2 MiB/s 4.2 MiB/s 4.2 MiB/s 6.4 MiB/s
16 14.6 MiB/s 8.4 MiB/s 8.6 MiB/s 12.3 MiB/s
32 28.5 MiB/s 16.2 MiB/s 16.9 MiB/s 22.0 MiB/s
64 55.9 MiB/s 29.2 MiB/s 34.3 MiB/s 35.5 MiB/s
128 72.1 MiB/s 53.7 MiB/s 61.8 MiB/s 47.8 MiB/s
256 69.4 MiB/s 64.6 MiB/s 59.4 MiB/s 52.4 MiB/s
512 68.3 MiB/s 63.2 MiB/s 58.7 MiB/s 52.7 MiB/s
1k 66.9 MiB/s 62.1 MiB/s 58.4 MiB/s 54.1 MiB/s
2k 66.2 MiB/s 61.4 MiB/s 58.7 MiB/s 54.1 MiB/s
4k 64.8 MiB/s 60.7 MiB/s 58.1 MiB/s 54.1 MiB/s
8k 64.3 MiB/s 60.2 MiB/s 57.3 MiB/s 53.5 MiB/s
16k 64.5 MiB/s 60.6 MiB/s 57.2 MiB/s 52.5 MiB/s
32k 63.2 MiB/s 59.7 MiB/s 56.9 MiB/s 55.4 MiB/s
64k 63.4 MiB/s 60.7 MiB/s 56.8 MiB/s 53.3 MiB/s
128k 62.8 MiB/s 59.2 MiB/s 56.6 MiB/s 53.0 MiB/s
256k 62.6 MiB/s 59.6 MiB/s 56.4 MiB/s 53.1 MiB/s
512k 62.1 MiB/s 58.7 MiB/s 55.3 MiB/s 52.2 MiB/s
1M 61.6 MiB/s 58.3 MiB/s 53.9 MiB/s 52.3 MiB/s
midrange desktop PC disk read results (1.5 Gbit/sec eSATA port)
-----------------------------------------------------------------
block filesystem type
size -------------------------------------------
(bytes)ext2 ext3 ext4 vfat
8 18.7MiB/s 18.6MiB/s 18.5MiB/s 18.5MiB/s
16 36.4MiB/s 36.4MiB/s 36.8MiB/s 36.6MiB/s
32 71.2MiB/s 70.1MiB/s 72.5MiB/s 70.0MiB/s
64 117.3MiB/s 110.6MiB/s 98.2MiB/s 82.3MiB/s
128 116.3MiB/s 111.6MiB/s 98.2MiB/s 82.8MiB/s
256 117.3Mib/s 110.6MiB/s 98.2MiB/s 81.9MiB/s
512 117.3Mib/s 110.6MiB/s 99.2MiB/s 81.3MiB/s
1k 116.3MiB/s 109.7MiB/s 98.2MiB/s 81.3MiB/s
2k 118.3MiB/s 108.7MiB/s 97.3MiB/s 81.3MiB/s
4k 117.3Mib/s 108.7MiB/s 96.3MiB/s 81.3MiB/s
8k 117.3Mib/s 108.7MiB/s 95.2MiB/s 81.3MiB/s
16k 118.3MiB/s 108.7MiB/s 95.4MiB/s 81.3MiB/s
32k 118.3MiB/s 108.7MiB/s 95.4MiB/s 81.3MiB/s
64k 118.3MiB/s 108.7MiB/s 95.3MiB/s 81.3MiB/s
128k 118.3MiB/s 108.7MiB/s 95.3MiB/s 81.0MiB/s
256k 118.3MiB/s 108.7MiB/s 95.2MiB/s 81.4MiB/s
512k 118.3MiB/s 108.7MiB/s 94.7MiB/s 81.3MiB/s
1M 118.3MiB/s 107.8MiB/s 94.2MiB/s 81.3MiB/s
All the results say that the HDD was functioning within expected parameters. So all transfer limits (64+ bytes) were due to USB protocol limitation or the 3 port USB hub internal to the Raspberry Pi.
I was thinking about something similar, but thishow at least this time was saved. Thank you for making these tests!
ReplyDeleteI was doing the tests for my own knowledge and decided to share them in case it saved someone else from having to do the same. Good to know I didn't waste my time in posting my tests and results. (Results alone, I find, are not much good unless you know how they were actually generated).
Deletereally good to know, thanks
ReplyDelete