# Frustrated with FreeNAS/ZFS performance...What now?



## max302

ZFS is notoriously hard on memory, IOPS and throughput are well known to be directly correlated to the amount of memory that you have. Using the SSD you already have for logs and cache should better performance, although from what I understand it'll probably be more IOPS than throughput.


----------



## PuffinMyLye

Quote:


> Originally Posted by *max302*
> 
> ZFS is notoriously hard on memory, IOPS and throughput are well known to be directly correlated to the amount of memory that you have. Using the SSD you already have for logs and cache should better performance, although from what I understand it'll probably be more IOPS than throughput.


Yea I know about ZFS being a memory hog which is why I got the 16GB which should be plenty for what I'm doing. I will setup the SSD as a cache drive though and see if it helps at all.


----------



## SamKook

I can at least say that with 32GB of RAM, I could fully saturate my gigabit network over CIFS and 6 WD red 3TB in RAIDZ2. I'm using vanilla FreeBSD though and use an ssd for the OS(not as a cache drive).
I now have double the drives and still no problem saturating the connection so I would think 16 should be enough and that the problem lies elsewhere, but I'm no expert so I have no idea where.
Other than the OS drive, the only other big difference would be that my CPU is much faster since I use it for a lot more stuff than a simple NAS so I would make sure the CPU usage isn't hitting 100%(I doubt it would though since it should be plenty fast for a NAS).


----------



## SweetAndLow

I would look into cpu utilization also. Samba on FreeNas should use a single core per connection. So clock speed is important. 3.4 Ghz should be plenty of clock speed for that. Are you using encryption, compression or dedup?


----------



## ButlerKevinD

You didn't mention what version of FreeNAS you're currently running. v9.1.1 just recently came out. If you're running one of the previous versions, try upgrading the installed distro. I'm about to fire one up as a NAS datastore for my ESXi whiteboxes. Would like to compare notes with our results as applicable.


----------



## PuffinMyLye

Quote:


> Originally Posted by *SamKook*
> 
> I can at least say that with 32GB of RAM, I could fully saturate my gigabit network over CIFS and 6 WD red 3TB in RAIDZ2. I'm using vanilla FreeBSD though and use an ssd for the OS(not as a cache drive).
> I now have double the drives and still no problem saturating the connection so I would think 16 should be enough and that the problem lies elsewhere, but I'm no expert so I have no idea where.
> Other than the OS drive, the only other big difference would be that my CPU is much faster since I use it for a lot more stuff than a simple NAS so I would make sure the CPU usage isn't hitting 100%(I doubt it would though since it should be plenty fast for a NAS).


Quote:


> Originally Posted by *SweetAndLow*
> 
> I would look into cpu utilization also. Samba on FreeNas should use a single core per connection. So clock speed is important. 3.4 Ghz should be plenty of clock speed for that. Are you using encryption, compression or dedup?


CPU utilization is less than 50% during large file transfers. I'm not doing any encryption, compression or dedup.
Quote:


> Originally Posted by *ButlerKevinD*
> 
> You didn't mention what version of FreeNAS you're currently running. v9.1.1 just recently came out. If you're running one of the previous versions, try upgrading the installed distro. I'm about to fire one up as a NAS datastore for my ESXi whiteboxes. Would like to compare notes with our results as applicable.


I'm running the latest version. I'm not using my FreeNAS box as VM datastore though as I have an SSD in my VM box for that and the speed is just no comparison. I do want to use my FreeNAS box for VM backups though.


----------



## swat565

I would ditch using Freenas entirely if performance is ANY consideration. Go download Openindiana and install Napp-it on top of it. I run this off an AMD E-350 integrated board with 8GB and x4 500gb barracuda drives (generation 7-8) and I am able to pretty close to saturate my gigabit line, there is no reason your configuration shouldn't.

I've scaled this setup from little boxes like the one I use to 10k builds for clients, Openindiana has been rock stable with ZFS and Napp-it web console makes it a breeze to manage.

Freenas is great and easy to set up, but performance is utter junk.


----------



## PuffinMyLye

Quote:


> Originally Posted by *swat565*
> 
> I would ditch using Freenas entirely if performance is ANY consideration. Go download Openindiana and install Napp-it on top of it. I run this off an AMD E-350 integrated board with 8GB and x4 500gb barracuda drives (generation 7-8) and I am able to pretty close to saturate my gigabit line, there is no reason your configuration shouldn't.
> 
> I've scaled this setup from little boxes like the one I use to 10k builds for clients, Openindiana has been rock stable with ZFS and Napp-it web console makes it a breeze to manage.
> 
> Freenas is great and easy to set up, but performance is utter junk.


Are the hardware requirements similar/the same for OpenIndiana as they are for FreeNAS?


----------



## CaptainBlame

To clarify, did you go from a 4 * 2-disk mirror to a 1 * 8-disk raidz2?


----------



## PuffinMyLye

Quote:


> Originally Posted by *CaptainBlame*
> 
> To clarify, did you go from a 4 * 2-disk mirror to a 1 * 8-disk raidz2?


I went from an 8 * 2TB RAID10 array on a Perc 6i to a 6 * 3TB RAIDz2 volume. Yes I knew there was going to be a write hit with the two parity drives I just didn't expect it to be this bad. I think I'm starting to realize first hand how much better performance is with RAID10 over parity RAID. I only went RAIDz2 because I wanted more space and didn't think the performance hit would be THIS bad. I have 4 more 3TB WD Reds in my possession though so I'm thinking of just going to a 8 * 2TB RAID10 setup. I'm just debating which OS to use. I want something that will fit on my 16GB USB Flash boot drive.


----------



## Jim888

I know freenas had problems with zfs/cifs speeds with 8.xx apparently they've not been fixed.

Try nas4free they broke off from freenas a few years back abd I've always liked them better, it's easy enough to give it a quick try just mount the same zfs disks/volumes and export through cifs.

About a year ago ppl were getting 40Mps with freenas web on the exact same system they were getting 100Mps on nas4free


----------



## PuffinMyLye

Quote:


> Originally Posted by *Jim888*
> 
> I know freenas had problems with zfs/cifs speeds with 8.xx apparently they've not been fixed.
> 
> Try nas4free they broke off from freenas a few years back abd I've always liked them better, it's easy enough to give it a quick try just mount the same zfs disks/volumes and export through cifs.
> 
> About a year ago ppl were getting 40Mps with freenas web on the exact same system they were getting 100Mps on nas4free


Interesting that's certainly worth trying. When you say say export through CIFS you mean what exactly?


----------



## Jim888

Quote:


> Originally Posted by *PuffinMyLye*
> 
> Interesting that's certainly worth trying. When you say say export through CIFS you mean what exactly?


just another way to say "Share"


----------



## swat565

Quote:


> Originally Posted by *PuffinMyLye*
> 
> Are the hardware requirements similar/the same for OpenIndiana as they are for FreeNAS?


Your hardware should work without hassle. Intel chipsets normally don't have any issues. Openindiana is open source version in essence of Solaris, which is what ZFS was natively written in.

I would download the "desktop" version to start out.
http://openindiana.org/
and once installed and updated you can install Napp-it with a simple command

*sudo wget -O - www.napp-it.org/nappit | perl*

simply set a static IP and you can manage it by putting in its IP followed by :81

I would defiantly give it a shot if your wanting to get better performance. Feel free to message me if you need any help getting it set up.


----------



## PuffinMyLye

Quote:


> Originally Posted by *Jim888*
> 
> just another way to say "Share"


Cool thanks! Might give this a shot this weekend so I have a good comparison.

Quote:


> Originally Posted by *swat565*
> 
> Your hardware should work without hassle. Intel chipsets normally don't have any issues. Openindiana is open source version in essence of Solaris, which is what ZFS was natively written in.
> 
> I would download the "desktop" version to start out.
> http://openindiana.org/
> and once installed and updated you can install Napp-it with a simple command
> 
> *sudo wget -O - www.napp-it.org/nappit | perl*
> 
> simply set a static IP and you can manage it by putting in its IP followed by :81
> 
> I would defiantly give it a shot if your wanting to get better performance. Feel free to message me if you need any help getting it set up.


Thanks a lot for this information! I'm going to give NAS4Free a shot this weekend since it seems I can easily mount my current ZFS volume into there and I can quickly test the peformance difference between it and FreeNAS. From there I very well may decide to give Openindiana a try and if I do I will let you know. I really appreciate you offering to help, I may very well take you up on that







.


----------



## CaptainBlame

Quote:


> Originally Posted by *PuffinMyLye*
> 
> I went from an 8 * 2TB RAID10 array on a Perc 6i to a 6 * 3TB RAIDz2 volume. Yes I knew there was going to be a write hit with the two parity drives I just didn't expect it to be this bad.


That's because of the way ZFS works, it stripes across all the vdevs in the pool. You only using one vdev and your performance is going to be that of a single disk. If you wanted a real comparison you should have configured raid10 in zfs as well. Even if you use OpenIndiana performance will still suck with 1 raidz vdev configuration.


----------



## PuffinMyLye

Quote:


> Originally Posted by *CaptainBlame*
> 
> That's because of the way ZFS works, it stripes across all the vdevs in the pool. You only using one vdev and your performance is going to be that of a single disk. If you wanted a real comparison you should have configured raid10 in zfs as well. Even if you use OpenIndiana performance will still suck with 1 raidz vdev configuration.


Yea I was considering configuring a RAID10 volume inside FreeNAS to see what I get. Unfortunately I only have room for 8 drives in my server so what was I supposed to do? Add another 2-drive mirrored volume so that I could stripe across the two vdevs?

What's really frustrated me is that I don't get any better write performance moving files around within the same volume. I have a CIFS share for the entire volume that has all my datasets (directories) within that share. If I move files from directory to directory the performance still isn't any better.


----------



## The_Rocker

I had 3 2TB Seagate Constellation CS's in a ZFS setup on freenas and I couldn't even saturate the 1Gig link to the desktop.

With a windows sever 2008, I can saturate the gig link with a single drive..........

Both on the same hardware.

On another note. I used Openfiler as a DIY SAN for my development virtual environment and performance is excellent over iSCSI. I havn't tried running samba shares off it yet, but id imagine its good.


----------



## CaptainBlame

Quote:


> Originally Posted by *The_Rocker*
> 
> I had 3 2TB Seagate Constellation CS's in a ZFS setup on freenas and I couldn't even saturate the 1Gig link to the desktop.


What configuration did you run?


----------



## SweetAndLow

So for reference, I have a 6 WD reds in a RaidZ2 and my write performance on a near empty volume is around 425MB/s when doing a local dd on the file system. My read performance is 475MB/s.

I would suggest reading through this thread, and see if your bottle neck is the disk or has something to do with the network.
http://forums.freenas.org/threads/write-performance-issues-mid-level-zfs-setup.13372/

here is the local dd I ran to get my numbers:
dd if=/dev/zero of=25G bs=1024k count=25k

over the network using cifs i get around 90MB/s, but i haven't really tested this that much.

my straight from disk reads are, this is skipping zfs:
Straight read from disk(no zfs):
ada0: 125 MB/s
ada1: 133 MB/s
ada2: 132 MB/s
ada3: 131 MB/s
ada4: 131 MB/s
ada5: 134 MB/s
Total: 788 MB/s


----------



## ndoggfromhell

I had similar issues with FreeNas on my HP Microserver. I eventually ditched it and used WHS2011 (got a copy for free from a friend who didn't use his) and I added an additional program called Drivebender to let me pool my 4 storage drives as 1 big volume. I fully saturate my gigabit link when I transfer files from my workstation. Since it's all files I have burned off elsewhere, I don't worry about keeping a backup or RAID for redundancy. It's got a very low power AMD chip, 8Gb of memory, and a 60Gb SSD. Specs are way less than your system, but it keeps up. The 4 drives in the removable bay are all Seagate 3Tb SATA models.


----------



## Pip Boy

NFS or Samba?

NFS is ridiculously fast. I have no problems here with ZFS using NFS.Samba is slower by far but easy peasy to setup (hence windows)


----------



## CaptainBlame

Quote:


> Originally Posted by *SweetAndLow*
> 
> dd if=/dev/zero of=25G bs=1024k count=25k


This is not a useful benchmark.


----------



## SweetAndLow

Quote:


> Originally Posted by *CaptainBlame*
> 
> This is not a useful benchmark.


why not? I don't believe ZFS treats this as a sparse file. I know if you have compression on then the numbers will be skewed, since 0* compresses quite well. You might be able to play with the bs to change the total throughput but that shouldn't make a huge difference.


----------



## ZachiF

PuffinMyLye - any progress with trying alternatives (NAS4Free, etc.)? Do you have any conclusions?
I have just setup my first FreeNAS (using the latest 9.2.1.1) and performance is really bad


----------



## Jeci

I can saturate my gigabit LAN with my NAS:

HP N40L | 8GB RAM | 6x2TB WD RED's RAIDz 12TB Total & 9.0TB Usable, you should certainly be able to as well... I running any dedup or encryption?


----------



## PuffinMyLye

Quote:


> Originally Posted by *ZachiF*
> 
> PuffinMyLye - any progress with trying alternatives (NAS4Free, etc.)? Do you have any conclusions?
> I have just setup my first FreeNAS (using the latest 9.2.1.1) and performance is really bad


I wound up changing my server to an UnRAID server since it's mostly for media anyways and it gave me the best options I felt for my needs in that area. I'm using an Asustor 4-bay NAS for my other needs (VM backups, etc.).

Quote:


> Originally Posted by *Jeci*
> 
> I can saturate my gigabit LAN with my NAS:
> 
> HP N40L | 8GB RAM | 6x2TB WD RED's RAIDz 12TB Total & 9.0TB Usable, you should certainly be able to as well... I running any dedup or encryption?


I wasn't doing any dedup or encryption no. I just had one 6 drive zpool and no matter what I did my performance over CIFS was very poor.

CIFS performance is literally all I care about as that's all I do.


----------



## johndough

Hi, I'm running Solaris 11 on a HP MicroServer N40L w/ 8 GByte RAM and 4 x 2 TByte RAIDZ (HGST 7K2000). Even though it has only a 1,5 GHz processor (AMD Turion II Neo) I'm saturating GBit LAN with big files over NFS. Local reading/writing gives me well above 400 MBytes / sec. Next I'll be trying FreeNAS on the very same hardware.


----------



## only1miller

Quote:


> Originally Posted by *johndough*
> 
> Hi, I'm running Solaris 11 on a HP MicroServer N40L w/ 8 GByte RAM and 4 x 2 TByte RAIDZ (HGST 7K2000). Even though it has only a 1,5 GHz processor (AMD Turion II Neo) I'm saturating GBit LAN with big files over NFS. Local reading/writing gives me well above 400 MBytes / sec. Next I'll be trying FreeNAS on the very same hardware.


johndough, I've got one of those HP MicroServers lying around- always figured them for too underpowered for FreeNAS so I'll be looking forward to your test results!

Cheers


----------



## Digidoc

Yeah I know this is an old thread, but I'm resurrecting it for a reason. If you're experiencing performance issues with FreeNAS and CIFS, make sure the bottleneck really isn't the server. That may sound obvious but here's why I'm saying it.

I just completed an upgrade on my FreeNAS server from 8.2.0 to 9.3. I've always been happy with the performance of my system under 8.2.0, but when I decided to upgrade to 9.3, I upgraded not only the OS but the firmware on my crossflashed M1015 card and I enabled LAGG on my dual-port Intel NIC. I also installed a dual port Intel NIC on workstation to see if I could boost the transfer speeds of the files I work on. The upgrade went smoothly (surprisingly). My FreeNAS server was already fairly beefy so I knew it'd handle 9.3 without an issue, but when I first tested the setup I wasn't happy with the speeds I was seeing.

Previously on 8.2.0 I was saturating the gigabit link. After the upgrade, when I tested, I was lucky if I'd hit 70MB on a 2Gb LAGG connection (on the server and workstation). Everything seemed zippier but the numbers weren't lining up with what I was seeing. Something was wrong but I couldn't put my finger on it. I beat my head against the wall at first but then it hit me what was wrong.

In short: I pulled a stupid.

The stupid was that when I tested my setup under 8.2.0, I made sure I copied my test files to a RAM disk that I set up on my workstation, and then copied the files to the server from the RAM disk. When I tested under 9.3 I was copying files to the server directly from my SSD's.

I then ran the test again, but this time I copied my test data to my RAM disk first then copied the files to the server from there. This time it peaked at 150MB/sec.









If you're seeing slow speeds, even if you have an SSD, try using a RAM disk on your workstation to hold the files you're copying over to eliminate the SSD from being the bottleneck. Here's a list for a few (some being free) if you want to try it. I'm using the SoftPerfect RAM Disk myself but any of them will do.

http://www.raymond.cc/blog/12-ram-disk-software-benchmarked-for-fastest-read-and-write-speed/
http://www.softperfect.com/products/ramdisk/


----------



## ucs75

Sorry to butt in here, but I have to dispel the FUD.

You're stating to the OP that he needs to ditch NAS4FREE in favor of another platform that you are not even able to saturate a 1Gbps link on. With a decent NAS4FREE box, you'll be needing 10Gbps links to handle the data flow. (and that's with compression enabled).

We use dual quad-core opterons with 32 or 64GB of RAM. 22 Desktop class spindles and two SSDs. (one for ZIL and the other for L2ARC).

This is used eBay class stuff, and it works VERY well.

Thank you for posting about OpenIndiana, it looks like something I'd be interested in checking out. But the demonization of NAS4Free is inaccurate.

...and after all that, I realized that I had confused FreeNAS, with NAS4Free...
So no, I don't have experience with FreeNAS, and can therefore not offer conclusive evidence one way or the other.


----------



## coachmark2

Just to chime in, I've got a RAIDZ2 array of 5 x 3TB WD Reds and a 64GB SSD for L2ARC. 16GB ECC system memory. My ZFS/iSCSI performance is complete trash, but... I don't really know what else to do with it.

It works, I guess, as a repository of stuff, but the performance over the wire is absolutely horrid.


----------



## tycoonbob

Quote:


> Originally Posted by *coachmark2*
> 
> Just to chime in, I've got a RAIDZ2 array of 5 x 3TB WD Reds and a 64GB SSD for L2ARC. 16GB ECC system memory. My ZFS/iSCSI performance is complete trash, but... I don't really know what else to do with it.
> 
> It works, I guess, as a repository of stuff, but the performance over the wire is absolutely horrid.


So I've actually been doing some research on ZFS (specifically FreeNAS) over the past couple of days, and I bet your issue is caused by the fact you have a L2ARC. Having one of those does not guarentee performance increase, and can actually kill performance in some scenarios. From what I've read, it's best to add more RAM to a system up to around 64GB, before bothering with a L2ARC.

I believe this is the post I read about why using L2ARC is not a good idea with 16GB RAM:
https://forums.freenas.org/index.php?threads/finding-suitable-zil-and-l2arc-drives.13561/

Maybe it can shed some light into your scenario. I'd be curious to hear how your performance changes by simply disabling your L2ARC.


----------



## coachmark2

@tycoonbob

I actually added the SSD as an L2ARC and ZIL at the suggestion of a NAS4Free developer...









Removing it doesn't help much, unfortunately.


----------



## nickster22

Hello everyone, I want to share a recent experience with 2 production servers I just completed. I am running these servers as a solution for Windows/Apple file server and Time Machine backups. I first read this set of posts before starting my projects. This is what I have experienced recently when using FreeNAS 9.3.

This is a wonderful feature rich software. I know that we are instructed that memory is important but, I would like to stress on point that I am learning. When I added a raid z-2 on my customers HP Proliant server ML110 G7, with 4 - 8terabyte Seagate archive drives things got sticky. My system choked the first time I started running tests. The fluid movement I am used to seeing in a new windows or Linux build was absent. Even with 8Gigs of RAM, my system was in distress. I was worried and needed this solution to work well. I went to my supplier and ordered the next Clock speed of RAM(PC-12800) and 32Gigs of it.

Server Stats: Xeon1220v2 3.3Ghz, 32Gigs DDR3 PC-12800, 4 - 8 Terabyte Archive drives Raid Z-2 (storage), Sandisk USB 3.0 16Gig flash (OS).

As I sit and run benchmarks you can see the usage on 32gigs skyrocket in a simple system. This server is running Time machine backups as well as proving CIFS file services that are reliable to apple as well. I have seen my usage in the first week of use as high as 26Gigs.

I would recommend more memory than 32Gigs for your custom build, especially if your a high performance type of person. An ideal build for a true performance seeker listed below.

1). CIFS is single threaded and needs a very fast processor 3.3 or greater will make great results.
2). 64Gigs of PC-12800 ECC - Can you afford not to go ECC when its your valuable info?
3). Hardware mirror set of SSD drives for the Operating system
4). Use the raid 10 option with 4 drives. All of the other Raid settings provide single disk speeds. Use identical disks, or you will suffer performance issues!

In my opinion FreeNAS is a useful system when purposed correctly. Please consider a different distro if extreme performance is your number one goal. The best way to use this wonderful distribution is for a backup server. Or maybe a place to safely house your most treasured info. This software is really setup to protect you from loss, use ECC memory to make sure the system is forever stable. After your hardware build run stress linux to test for any errors that may come about.

Good luck on your build, I hope you enjoy using this software as much as I do.

California Network Systems INC.


----------

