ZFS speed

So I ran some quick dd-based write numbers before choosing a filesystem for my new Proxmox based virtual server. I really wanted ZFS, but I need OpenVZ and KVM more, and ZFS-Fuse just does cut it.

In short - ext4 is the winner, though ZFS sure looks pretty damn fast. SolarisInternals has more info on ZFS write speed

Another thing to note, though the below numbers for ZFS-Fuse don't reflect it, is that ZFS performs better with Write-Back cache off on your Raid controller. Had I tested OpenSolaris with Write-Back cache on (boot up was ungodly slow with it on), I'm sure it would have been reflected in the numbers.

4 600GB, RAID 5, wb on, perc 7, ext4

proxmox2:~# dd if=/dev/zero of=/images/file bs=1024k count=1024 conv=fsync
1073741824 bytes (1.1 GB) copied, 2.53637 s, 423 MB/s

proxmox2:~# dd if=/dev/zero of=/images/file bs=1024k count=10240 conv=fsync
10737418240 bytes (11 GB) copied, 20.3379 s, 528 MB/s

proxmox2:~# dd if=/dev/zero of=/images/file bs=1024k count=102400 conv=fsync
107374182400 bytes (107 GB) copied, 192.164 s, 559 MB/s


---------------------------------------------------------------------------------------
4 600GB, RAID 5, wb on, perc 7, ext3

proxmox2:~# dd if=/dev/zero of=/images/file bs=1024k count=1024 conv=fsync
1073741824 bytes (1.1 GB) copied, 4.58121 s, 234 MB/s

proxmox2:~# dd if=/dev/zero of=/images/file bs=1024k count=10240 conv=fsync
10737418240 bytes (11 GB) copied, 37.0681 s, 290 MB/s

proxmox2:~# dd if=/dev/zero of=/images/file bs=1024k count=102400 conv=fsync
107374182400 bytes (107 GB) copied, 371.047 s, 289 MB/s


---------------------------------------------------------------------------------------
4 600GB, RAID 5, wb on, perc 7, reiser3

proxmox2:~# dd if=/dev/zero of=/images/file bs=1024k count=1024 conv=fsync
1073741824 bytes (1.1 GB) copied, 2.91366 s, 369 MB/s


proxmox2:~# dd if=/dev/zero of=/images/file bs=1024k count=10240 conv=fsync
10737418240 bytes (11 GB) copied, 28.923 s, 371 MB/s

proxmox2:~# dd if=/dev/zero of=/images/file bs=1024k count=102400 conv=fsync
Hmm Not sure what happened here...

---------------------------------------------------------------------------------------
4 600GB, RAID 5, wb on, perc 7, reiser4

proxmox2:~# dd if=/dev/zero of=/images/file bs=1024k count=1024 conv=fsync
1073741824 bytes (1.1 GB) copied, 2.87369 s, 374 MB/s

proxmox2:~# dd if=/dev/zero of=/images/file bs=1024k count=10240 conv=fsync
10737418240 bytes (11 GB) copied, 28.9769 s, 371 MB/s

proxmox2:~# dd if=/dev/zero of=/images/file bs=1024k count=102400 conv=fsync

Hmm Not sure what happened here...


---------------------------------------------------------------------------------------
4 600GB, RAID 5, wb on, perc 7, zfs-fuse (defaults)

proxmox2:~# dd if=/dev/zero of=/images/file bs=1024k count=1024 conv=fsync
1073741824 bytes (1.1 GB) copied, 15.0122 s, 71.5 MB/s

proxmox2:~# dd if=/dev/zero of=/images/file bs=1024k count=10240 conv=fsync
10737418240 bytes (11 GB) copied, 159.544 s, 67.3 MB/s


100GB - didn't bother

---------------------------------------------------------------------------------------
4 600GB, RAID 5, wb off, perc 7, zfs-fuse (defaults)


proxmox2:/# dd if=/dev/zero of=/images/file bs=1024k count=1024 conv=fsync
1073741824 bytes (1.1 GB) copied, 23.7539 s, 45.2 MB/s

---------------------------------------------------------------------------------------
4 600GB, RAID 5, wb off, perc 7, zfs-fuse (64k recordsize)

proxmox2:/# dd if=/dev/zero of=/images/file bs=1024k count=1024 conv=fsync
1073741824 bytes (1.1 GB) copied, 21.3764 s, 50.2 MB/s


---------------------------------------------------------------------------------------
4 600GB, raidz, wb off, perc 7, zfs-fuse

proxmox2:~# dd if=/dev/zero of=/images/file bs=1024k count=1024 conv=fsync
1073741824 bytes (1.1 GB) copied, 19.1704 s, 56.0 MB/s

proxmox2:~# dd if=/dev/zero of=/images/file bs=1024k count=10240 conv=fsync
10737418240 bytes (11 GB) copied, 187.938 s, 57.1 MB/s





---------------------------------------------------------------------------------------
4 600GB, raidz, wb off, perc 7, ZFS opensolaris (Nexenta)

root@proxmox2:/volumes# dd if=/dev/zero of=/volumes/images/file bs=1024k count=1024 conv=fsync
1073741824 bytes (1.1 GB) copied, 1.30164 seconds, 825 MB/s

root@proxmox2:/volumes# dd if=/dev/zero of=/volumes/images/file bs=1024k count=10240 conv=fsync
10737418240 bytes (11 GB) copied, 5.18124 seconds, 2.1 GB/s

root@proxmox2:/volumes# dd if=/dev/zero of=/volumes/images/file bs=1024k count=102400 conv=fsync
107374182400 bytes (107 GB) copied, 45.0769 seconds, 2.4 GB/s