I have an old Excito Bubba B3-server that is a 2TB disk and a mirrored expansion slot. Excito went bust and the drives are getting old. On my main computer I have 4 HDD in a storage spaces with one copy mirroring. These drives are WD RE4 and very stable, but are getting old as well.
For backup of documents I use Onedrive, but the increase of ransomware made me think that it would be nice to also have a local but separate backup server. Here I would also like to archive my DVD collection etc.
As I am a Windows guy, I wanted to expore the option to run Windows 10 with storage spaces. The problem is that you are not allowed to create a parity storage space with dual party or more unless you run Windows server. And I dont plan to get a license for $500 just for a storage server.
There are some other alternatives. freeNAS that is a freeBSD based free solution, NAS4Free that is an older fork of freeNAS, but uses PHP instead of python. See a comparsion here.
There are also some other interesting solutions that are based on the concept that only the HDD that you read or write to should need to spin up. The problem with them was lack of proper file system and a very small community. That is why i choose freeNAS in the end.
Pre-buy testingBefore I did anything, I tried to install freenas in Hyper-V on Windows 10 1607. I failed!
After the installation it would just not boot freeNAS and it didnt matter what settings i tried.
I gave up and used my VMware Workstation (at work) instead. I created a new virtual machine with one boot disk and 5 virtual data disks (not true, I begun with 8 disks to simulate my future physical setup exactly, but after some testing I reduced this to 5 to keep it more simple).
After 10 minutes with VMware I was up and running! I spent the next two weeks (on my spare time mostly on weekends) playing with different scenarios in freeNAS. Replacing drives after unexpected loss of drives and planned replacements etc. I played with rsync & SAMBA settings, user management and ACLs. I learned a lot from this and I recommend any of you thinking of getting your first freeNAS server to do the same.
A Change of plansMy original plan was to begin with 4 HDD in a RAIDz2 and expand this to 8 HDD after 6 months or so. That way I would get different batch HDDs in case of a bad firmware etc.
But then I read about that you can not add more disks to the existing vdevs. In that case I had to make a new vdev with 4 drives, and since a RAIDz2 with 4 drives is just plain stupid when you could make a mirrored stripe instead, I decided to buy all 8 drives and go for RAIDz3. With a RAIDz3 you can loose any 3 drives at same time without loosing the vdev. I expect to only loose a single drive at a time, but during the resilver (rebuild after replacing a drive) I don't want anything bad to happen, with RAIDz2 you can safely loose two drives. If you loose another drive during resilver you are screwed! (I have lost a software RAID5 due to bad Seagate ES2 drives in my past, and I don't want that to happen again!)
HardwareIn the beginning of planning I was dedicated to find a good chassi with hot swap for the drives. I was very tempted to buy a "real" freeNAS Mini XL server but I couldn't find the hardware specs for the SATA controller etc and I want to buy from a Swedish retailer since we have some very nice laws when it comes to customer rights and warranty.
I couldn't find anything that had the right price and was available from a Swedish retailer.
After some time (some months or so I decided to choose a web store and select products they had.
In Sweden there are two main options for business products (the ECC parts isn't available at the gamer oriented retailers), Dustin or Misco. I've never had a bad experience with Dustin so I choose them.
Here is what I ended up with:
|Fractal Design Define R5 Black|
A very nice chassi with 8 internal easy access 3.5" slots and a lot of spots for fans. Good cable management and 2 SSD (2.5") mounting spots on the back of the mother board.
When I picked it up I realized how large the chassi is, It looks smaller on the pictures.
"Everything is included" even a small tool you can use to mount the motherboard spacers.
I've used Corsair power supplies for almost 10 years both at home and at work and I have never had any problems related to the power supply. It is modular and comes with a lot of cables.
Unfortunately my server have "too many" disks, and I needed 10 SATA power cables. The box contains 3 cables with 3 connectors. That was just enough for the 6 HDDs, and the SSD on the back.
For 2 drives I had to use the old style molex cables with SATA power adapters until I can find where to buy extra SATA power cables.
DO NOT USE CABLES FROM OLD GENERATION PSU FROM CORSAIR!!!
It seems they have changed their standard on the output. The physical ports have changed so that you shouldn't be able to make them fit, but I didn't even notice that until I made this text and watched the picture.
|Gigabyte Ga-X150m-Pro ECC C232 S-1151 m-ATX S-1151|
Another failure from my part was that this motherboard does not have a build in graphics card.
I'm using an old ATI HD4350 1GB DDR2 (passive cooling, uses about 20W) when I need a monitor. After som research I managed to run freeNAS with no graphics, read more about this below in the power consumption part.
|Intel Core i3 6100T / 3.2 GHz processor S-1151|
During my tests that included resilver and scrub, Ive never seen the CPU go above ~18% so this is powerful enough for a file server. If you plan to use the server for more than file sharing, you may want check up if it will handle the job. I think it does!
|Notice the extra chip that is proof of ECC capability.|
|Kingston Valueram 2x8GB 2133MHz DDR4 ECC (un-buffered)|
I made a big mistake when I bought the parts. Due to the poor filtering and lack of specifications I got registered ECC RAM instead of un-buffered. That made the computer not even boot to "BIOS".Dustin saved my day and I could send back the wrong memory and get new. Make sure you get the type your motherboard supports! ECC RAM have extra memory chips. Regular non-ECC RAM will have 8 chips and ECC RAM will have 9. I would also recommend that you ALWAYS run memtest after installing RAM even if you just added more or moved them. You just don't want to risk getting a memory corruption!
|WD Red SOHO NAS 3TB 3.5" SATA-600|
|LSI SAS 9207-8i|
|This is how the cable connectors look like in closeup|
After working with storage servers (among things) I've learned the hard way the importance of a proper disk controller. I knew I wanted a LSI controller but didnt know which one, so I sorted for best price and this is what I got. You want a non RAID controller and this is a very good choice.
The most confusing thing is that the LSI SAS 9207-8i is the new version of the LSI SAS 9211-8i. Higher numbers isnt always better it seems!
|LSI SATA cable|
LSI SATA Cables are not included in the box with the controller, so don't forget to buy them as well.
As with RAM, you don't want a bad cable that causes problems with your data!
Boot USB drive
|Corsair Flash Voyager Slider X2 16GB USB 3.0|
Why this USB storage? It has 5 years warranty and pretty good speed (50MB/s write and 200MB/read) And it is very small so it can sit in the USB3 ports on the back without the risk of getting stuck in cables etc.
Ive now put the boot USB drive in a motherboard USB adapter so it doenst stick out of the back of the server.
|APC Back-UPS ES 700|
System Dataset SSDOne thing that I did not plan for was separate "System Dataset" SSDs. In short, if you don't have separate drives for the "System dataset" your HDDs will never spin down. Read more about that below.
For this I use a pair of old 120GB+160GB Intel 320 SSD I had left over from before and I use them in a mirror.
Pictures from the install
|The SSD mount on the back of the motherboard|
The bad thing is that drives must be installed with ports facing down. This makes it very hard to connect cables. Also, all my cables are bent in the "wrong" direction. I could not use the molex to SATA power adapter here because it would not fit. I managed to use the SATA power cable from Corsair but there is no marginal at all.
|Power adapter cables|
|The box it was delivered in...|
|HDD mount with rubber thingis are included with the chassis.|
|All HDDs finally installed and the cables are somewhat in order.|
Some things I've learned about freeNAS and ZFSAfter reading the freeNAS intro to the system and ZFS I can summarize this with you (I think you should read it anyway)
- You don't need SSD for cache. Buy more RAM until you have maxed out the motherboard. Unless you are a business with lots of users you will do fine with 16GB. At the University institution I work at, we have some older Sun X4500 Thumpers with 48 HDD and 16GB RAM (they cant have more). One of the servers have been the primary "home server" for ~500 users for more than 10 years. When it was new it had 500GB drives and we then changed the drives for 2TB drives. It still works with "just" 16GB RAM (even though we would like more). If we can use it with 500 users you will do just fine with 16GB RAM at home.
- Unless you plan to use NFS you don't need a log (ZIL) SSD.
- You cant add more drives to a vdev after it has been created, plan ahead!
- It is very easy to export the configuration and to import existing ZFS pools in case you must re-install the server. Since you have no important data what so ever on the boot USB flash memory, its just to install the new version, boot the system, load the config and import the ZFS pools (if they didnt load with the config).
- Why did I choose RAIDZ3 shouldnt RAIDZ2 be enought? At the moment you will probably be just fine with RAIDZ2. After some reading and talking to some UNIX people at work it seems that RAIDZ2 is fine for systems with <=3TB HDDs. With larger drives the resilver(rebuild) times is longer and during the stress another disk failure is more probable.
Wait a minute, didnt I say I had 3TB HDDs? That mean I should be fine with RAIDZ2 and get more data from the pool! Yes, I would.. for now. But I plan to use this server for many years to come, and when I replace my drives the next time, they will probably be 10TB or more. Then I really need RAIDZ3. I will have plenty of space even with "only" 12TB data (80% of 15TB is 12TB, never fill a ZFS pool more than 80%, it become painfully slow when the system tries to find free sectors to write data and trying to avoid fragmentation at the same time.)
Things that have gone wrong
- First of all, I just bought 4 HDDs because I thought that I could add more drives later.
- I got the wrong ECC type RAM. Make sure you get the correct RAM for your motherboard.
- When you use an external HDD controller the freenNAS GUI built in settings for HDD power management does not work. But there is a fix for that!
You can not have the System Dataset on the HDDs you want to spin down. Use another disk or two in a mirror and move the system dataset here.
Set "Advanced Power Management" to "127" on all HDDs and then run a command like this in the shell (make sure you change to fit the correct HDD names:
"camcontrol standby /dev/da0 -t 600"
You can add a crontab task that runs every now and then to set this on all drives at the same time, this example will do this for da0 -> da7:
"/bin/bash -c 'for x in `seq 0 7`; do camcontrol standby da$x; done'"
I plan to run this command as a crontab scheduled task so that I can have different spindown times depending on the time of day and if it is weekday or weekend.
- Every attempt to upgrade to the latest version have ended up with the same error. I'll just have ot wait for a new version to become released. The good thing is that the new version have the same problem even in VMware, so in the future Ill make sure to try the upgrade there before I upgraade my real server.
- At first I used an old Intel x25m 80GB SSD triggered an error because freeNAS could not read the S.M.A.R.T data. So I had to replace this with a 160GB Intel 320 SSD.
Things that took longer than expected
When I first installed the HDDs to the LSI controller it took like 5 minutes to pass the boot steps before you could even enter BIOS. I found a new firmware but I just could not make out how to flash it. First I made a boot floppy (since I choose the DOS/Windows version). Windows 10 cannot create boot floppies anymore, so I had to do it from a virtual Windows 7. Then I could not make the flash util to show the help command. After some time I just decided to install Windows on the server. I already had the ISO in my Zalman SSD USB with CD/DVD emulator so the installation took 10 minutes. It then turned out I still had to use the "DOS" version but I managed to get it work from inside Windows CMD.
With the new firmware the boot times was shortened to seconds, not minutes and all is well.
Compare this with the way you flash BIOS on the Gigabyte Motherboard.
- Download firmware
- Extract bin to a USB device with FAT32 or ExtFAT
- Boot the computer and select the QFlash option.
- Select the bin file and start the upgrade!
Power consumptionI have now done som measuring of the power consumption. This was done with a Fibaro Wall plug that have built in measurement feature. I don't know how precise it is but seems good enough for me,
With graphics card: ATI HD4350 1GB DDR2
All disk idle: 67.4W
All disk spins: 86.6W
All disk spin up: ca 150W
After I activated some power management features in BIOS:All disk idle: 62.2W
All disk spins: 86.1W
All disk spin up: 138W
With the computer offline I noticed that it used more power than I expected:
UPS only: 4.9W
UPS + NAS shutdown: ca 8W
To make freeNAS work without a graphics card, you need to make sure serial ports are enabled in the "BIOS", and check the box "Use serial console" in freeNAS System>Advanced.
No graphics (serial console output only)
All disk idle: 44.8W
All disk spin: 67.2W
All disk spin up: 118.8W
List of hardware
- Motherboard: Gigabyte Ga-X150m-Pro ECC C232 S-1151 m-ATX S-1151
- CPU: Intel Core i3 6100T / 3.2 GHz processor S-1151
- RAM: Kingston Valueram 2x8GB 2133MHz DDR4 ECC (un-buffered)
- HDD: WD Red SOHO NAS 3TB 3.5" SATA-600
- HDD Controller: LSI SAS 9207-8i
- SATA Cables: LSI SATA cable
- Boot USB: Corsair Flash Voyager Slider X2 16GB USB 3.0
- Chassis: Fractal Design Define R5 Black
- PSU: Corsair RM750x
- UPS: APC Back-UPS ES 700
Some useful links: