Project freeNAS server

Background

I have an old Excito Bubba B3-server that is a 2TB disk and a mirrored expansion slot. Excito went bust and the drives are getting old. On my main computer I have 4 HDD in a storage spaces with one copy mirroring. These drives are WD RE4 and very stable, but are getting old as well.
For backup of documents I use Onedrive, but the increase of ransomware made me think that it would be nice to also have a local but separate backup server. Here I would also like to archive my DVD collection etc.

As I am a Windows guy, I wanted to expore the option to run Windows 10 with storage spaces. The problem is that you are not allowed to create a parity storage space with dual party or more unless you run Windows server. And I dont plan to get a license for $500 just for a storage server.
There are some other alternatives. freeNAS that is a freeBSD based free solution, NAS4Free that is an older fork of freeNAS, but uses PHP instead of python. See a comparsion here.
There are also some other interesting solutions that are based on the concept that only the HDD that you read or write to should need to spin up. The problem with them was lack of proper file system and a very small community. That is why i choose freeNAS in the end.

Pre-buy testing

Before I did anything, I tried to install freenas in Hyper-V on Windows 10 1607. I failed!
After the installation it would just not boot freeNAS and it didnt matter what settings i tried.
I gave up and used my VMware Workstation (at work) instead. I created a new virtual machine with one boot disk and 5 virtual data disks (not true, I begun with 8 disks to simulate my future physical setup exactly, but after some testing I reduced this to 5 to keep it more simple).
After 10 minutes with VMware I was up and running! I spent the next two weeks (on my spare time mostly on weekends) playing with different scenarios in freeNAS. Replacing drives after unexpected loss of drives and planned replacements etc. I played with rsync & SAMBA settings, user management and ACLs. I learned a lot from this and I recommend any of you thinking of getting your first freeNAS server to do the same.

A Change of plans

My original plan was to begin with 4 HDD in a RAIDz2 and expand this to 8 HDD after 6 months or so. That way I would get different batch HDDs in case of a bad firmware etc.
But then I read about that you can not add more disks to the existing vdevs. In that case I had to make a new vdev with 4 drives, and since a RAIDz2 with 4 drives is just plain stupid when you could make a mirrored stripe instead, I decided to buy all 8 drives and go for RAIDz3. With a RAIDz3 you can loose any 3 drives at same time without loosing the vdev. I expect to only loose a single drive at a time, but during the resilver (rebuild after replacing a drive) I don't want anything bad to happen, with RAIDz2 you can safely loose two drives. If you loose another drive during resilver you are screwed! (I have lost a software RAID5 due to bad Seagate ES2 drives in my past, and I don't want that to happen again!)

Hardware

In the beginning of planning I was dedicated to find a good chassi with hot swap for the drives. I was very tempted to buy a "real" freeNAS Mini XL server but I couldn't find the hardware specs for the SATA controller etc and I want to buy from a Swedish retailer since we have some very nice laws when it comes to customer rights and warranty.
I couldn't find anything that had the right price and was available from a Swedish retailer.
After some time (some months or so I decided to choose a web store and select products they had.
In Sweden there are two main options for business products (the ECC parts isn't available at the gamer oriented retailers), Dustin or Misco. I've never had a bad experience with Dustin so I choose them.

Here is what I ended up with:

Chassi

Fractal Design Define R5 Black

A very nice chassi with 8 internal easy access 3.5" slots and a lot of spots for fans. Good cable management and 2 SSD (2.5") mounting spots on the back of the mother board.

When I picked it up I realized how large the chassi is, It looks smaller on the pictures.

"Everything is included" even a small tool you can use to mount the motherboard spacers.

Power supply

Corsair RM750x

I've used Corsair power supplies for almost 10 years both at home and at work and I have never had any problems related to the power supply. It is modular and comes with a lot of cables.
Unfortunately my server have "too many" disks, and I needed 10 SATA power cables. The box contains 3 cables with 3 connectors. That was just enough for the 6 HDDs, and the SSD on the back.
For 2 drives I had to use the old style molex cables with SATA power adapters until I can find where to buy extra SATA power cables.

DO NOT USE CABLES FROM OLD GENERATION PSU FROM CORSAIR!!!
It seems they have changed their standard on the output. The physical ports have changed so that you shouldn't be able to make them fit, but I didn't even notice that until I made this text and watched the picture.

Motherboard

Gigabyte Ga-X150m-Pro ECC C232 S-1151 m-ATX S-1151

I wanted a motherboard that had support for ECC RAM. ECC is a must when you use ZFS since all your data will pass through the RAM and you don't want memory corruption to be a possibility. Its not even that expensive. This was one of the few motherboard Dustin had with full ECC support and

Warning! A lot of motherboards on price checkup sites and web shops have "ECC support" marked on them. That is true in the way that they will work with ECC RAM, but they will not USE the ECC features. Make sure you get a motherbord with full ECC support!
Another failure from my part was that this motherboard does not have a build in graphics card.
I'm using an old ATI HD4350 1GB DDR2 (passive cooling, uses about 20W) when I need a monitor. After som research I managed to run freeNAS with no graphics, read more about this below in the power consumption part.

CPU

Intel Core i3 6100T / 3.2 GHz processor S-1151

I wanted a server that used as little power as possible and the T-version have a TDP value of 35W compared to 51W for the non-T-version.
During my tests that included resilver and scrub, Ive never seen the CPU go above ~18% so this is powerful enough for a file server. If you plan to use the server for more than file sharing, you may want check up if it will handle the job. I think it does!

RAM

Notice the extra chip that is proof of ECC capability.

Kingston Valueram 2x8GB 2133MHz DDR4 ECC (un-buffered)

I made a big mistake when I bought the parts. Due to the poor filtering and lack of specifications I got registered ECC RAM instead of un-buffered. That made the computer not even boot to "BIOS".

Dustin saved my day and I could send back the wrong memory and get new. Make sure you get the type your motherboard supports! ECC RAM have extra memory chips. Regular non-ECC RAM will have 8 chips and ECC RAM will have 9. I would also recommend that you ALWAYS run memtest after installing RAM even if you just added more or moved them. You just don't want to risk getting a memory corruption!

HDD

WD Red SOHO NAS 3TB 3.5" SATA-600

Previously I have used Western Digital RE4 edition drives. They are Enterprise class and the same we have in some of our file storage servers at work. freeNAS recommends Western Digital Red series but I was suspicious of the quality. I changed my mind after reading a lot of tests and the price for Enterprise rate disks combined with the number of disks I wanted made me go for the Red series.

HDD Controller

LSI SAS 9207-8i

This is how the cable connectors look like in closeup

After working with storage servers (among things) I've learned the hard way the importance of a proper disk controller. I knew I wanted a LSI controller but didnt know which one, so I sorted for best price and this is what I got. You want a non RAID controller and this is a very good choice. 

The most confusing thing is that the LSI SAS 9207-8i is the new version of the LSI SAS 9211-8i. Higher numbers isnt always better it seems!

LSI SATA cable

LSI SATA Cables are not included in the box with the controller, so don't forget to buy them as well.

As with RAM, you don't want a bad cable that causes problems with your data!

Boot USB drive

Corsair Flash Voyager Slider X2 16GB USB 3.0

Why this USB storage? It has 5 years warranty and pretty good speed (50MB/s write and 200MB/read) And it is very small so it can sit in the USB3 ports on the back without the risk of getting stuck in cables etc.

Ive now put the boot USB drive in a motherboard USB adapter so it doenst stick out of the back of the server.

UPS

APC Back-UPS ES 700

I've used APC UPS for 20 years and this series have normal wall plug outlets and also a master/slave feature. In short, connect server to master and monitor etc to slave. When server is off the monitor will also be completely turned off, not just standby. For this server I don't expect to use this feature, but on my main computer at home I have speakers, monitors & LED lights connected to the slave port. I used to have the "tower" version of the Back-UPS series but the later models have a high pitch sound that I could live with. The only drawback with this model is that the power cord is attached to the UPS, you cannot replace or change the cable without voiding the warranty.

System Dataset SSD

One thing that I did not plan for was separate "System Dataset" SSDs. In short, if you don't have separate drives for the "System dataset" your HDDs will never spin down. Read more about that below.

For this I use a pair of old 120GB+160GB Intel 320 SSD I had left over from before and I use them in a mirror.

Pictures from the install

The SSD mount on the back of the motherboard

The bad thing is that drives must be installed with ports facing down. This makes it very hard to connect cables. Also, all my cables are bent in the "wrong" direction. I could not use the molex to SATA power adapter here because it would not fit. I managed to use the SATA power cable from Corsair but there is no marginal at all.

Power adapter cables

The box it was delivered in...

HDD mount with rubber thingis are included with the chassis.

All HDDs finally installed and the cables are somewhat in order.

Some things I've learned about freeNAS and ZFS

After reading the freeNAS intro to the system and ZFS I can summarize this with you (I think you should read it anyway)

• You don't need SSD for cache. Buy more RAM until you have maxed out the motherboard. Unless you are a business with lots of users you will do fine with 16GB. At the University institution I work at, we have some older Sun X4500 Thumpers with 48 HDD and 16GB RAM (they cant have more). One of the servers have been the primary "home server" for ~500 users for more than 10 years. When it was new it had 500GB drives and we then changed the drives for 2TB drives. It still works with "just" 16GB RAM (even though we would like more). If we can use it with 500 users you will do just fine with 16GB RAM at home. 
• Unless you plan to use NFS you don't need a log (ZIL) SSD.
• You cant add more drives to a vdev after it has been created, plan ahead!
• It is very easy to export the configuration and to import existing ZFS pools in case you must re-install the server. Since you have no important data what so ever on the boot USB flash memory, its just to install the new version, boot the system, load the config and import the ZFS pools (if they didnt load with the config).
• Why did I choose RAIDZ3 shouldnt RAIDZ2 be enought? At the moment you will probably be just fine with RAIDZ2. After some reading and talking to some UNIX people at work it seems that RAIDZ2 is fine for systems with <=3TB HDDs. With larger drives the resilver(rebuild) times is longer and during the stress another disk failure is more probable.
  Wait a minute, didnt I say I had 3TB HDDs? That mean I should be fine with RAIDZ2 and get more data from the pool! Yes, I would.. for now. But I plan to use this server for many years to come, and when I replace my drives the next time, they will probably be 10TB or more. Then I really need RAIDZ3. I will have plenty of space even with "only" 12TB data (80% of 15TB is 12TB, never fill a ZFS pool more than 80%, it become painfully slow when the system tries to find free sectors to write data and trying to avoid fragmentation at the same time.)

Things that have gone wrong

• First of all, I just bought 4 HDDs because I thought that I could add more drives later. 
• I got the wrong ECC type RAM. Make sure you get the correct RAM for your motherboard.
• When you use an external HDD controller the freenNAS GUI built in settings for HDD power management does not work. But there is a fix for that!
  You can not have the System Dataset on the HDDs you want to spin down. Use another disk or two in a mirror and move the system dataset here.
  Set "Advanced Power Management" to "127" on all HDDs and then run a command like this in the shell (make sure you change to fit the correct HDD names:
  "camcontrol standby /dev/da0 -t 600"
  You can add a crontab task that runs every now and then to set this on all drives at the same time, this example will do this for da0 -> da7:
  "/bin/bash -c 'for x in `seq 0 7`; do camcontrol standby da$x; done'"
  I plan to run this command as a crontab scheduled task so that I can have different spindown times depending on the time of day and if it is weekday or weekend.
• 
  
• 
  

When I tried freeNAS in VMware everything was just fine. When I had installed freeNAS on the server I got stuck on "loading kernel modules". I didn't have a clue on what to do or why it happened. Then I realized that there had been a new version released just the days before my server parts came. I installed the old version (the same as in VMware) and it worked just fine. I expect this to be fixed "soon". The good thing is that I get the same problem in VMware so I will be able to try the new version there before I upgrade my real server.

• Every attempt to upgrade to the latest version have ended up with the same error. I'll just have ot wait for a new version to become released. The good thing is that the new version have the same problem even in VMware, so in the future Ill make sure to try the upgrade there before I upgraade my real server.
• At first I used an old Intel x25m 80GB SSD triggered an error because freeNAS could not read the S.M.A.R.T data. So I had to replace this with a 160GB Intel 320 SSD.

Things that took longer than expected

When I first installed the HDDs to the LSI controller it took like 5 minutes to pass the boot steps before you could even enter BIOS. I found a new firmware but I just could not make out how to flash it. First I made a boot floppy (since I choose the DOS/Windows version). Windows 10 cannot create boot floppies anymore, so I had to do it from a virtual Windows 7. Then I could not make the flash util to show the help command. After some time I just decided to install Windows on the server. I already had the ISO in my Zalman SSD USB with CD/DVD emulator so the installation took 10 minutes. It then turned out I still had to use the "DOS" version but I managed to get it work from inside Windows CMD.

With the new firmware the boot times was shortened to seconds, not minutes and all is well.

Compare this with the way you flash BIOS on the Gigabyte Motherboard.

1. Download firmware
2. Extract bin to a USB device with FAT32 or ExtFAT
3. Boot the computer and select the QFlash option.
4. Select the bin file and start the upgrade!

Power consumption

I have now done som measuring of the power consumption. This was done with a Fibaro Wall plug that have built in measurement feature. I don't know how precise it is but seems good enough for me,

With graphics card: ATI HD4350 1GB DDR2
All disk idle: 67.4W
All disk spins: 86.6W
All disk spin up: ca 150W

After I activated some power management features in BIOS:All disk idle: 62.2W
All disk spins: 86.1W
All disk spin up: 138W

With the computer offline I noticed that it used more power than I expected:
UPS only: 4.9W
UPS + NAS shutdown: ca 8W

To make freeNAS work without a graphics card, you need to make sure serial ports are enabled in the "BIOS", and check the box "Use serial console" in freeNAS System>Advanced.

No graphics (serial console output only)
All disk idle: 44.8W
All disk spin: 67.2W
All disk spin up: 118.8W

List of hardware

• Motherboard:    Gigabyte Ga-X150m-Pro ECC C232 S-1151 m-ATX S-1151
• CPU:     Intel Core i3 6100T / 3.2 GHz processor S-1151
• RAM:    Kingston Valueram 2x8GB 2133MHz DDR4 ECC (un-buffered)
• HDD:    WD Red SOHO NAS 3TB 3.5" SATA-600
• HDD Controller:    LSI SAS 9207-8i
• SATA Cables:    LSI SATA cable
• Boot USB: Corsair Flash Voyager Slider X2 16GB USB 3.0
• Chassis:    Fractal Design Define R5 Black
• PSU:    Corsair RM750x
• UPS:    APC Back-UPS ES 700

Some useful links:

https://forums.freenas.org/index.php?threads/slideshow-explaining-vdev-zpool-zil-and-l2arc-for-noobs.7775/

http://www.ixsystems.com/freenas-mini

http://www.freenas.org/blog/a-complete-guide-to-freenas-hardware-design-part-ii-hardware-specifics/

https://forums.freenas.org/index.php?threads/s-m-a-r-t-smart-best-practice.40340/

https://forums.freenas.org/index.php?threads/scripts-to-report-smart-zpool-and-ups-status-hdd-cpu-t%C2%B0-hdd-identification-and-backup-the-config.27365/ 

checkfreediskspacemsg.cmd

Why I created the script

I needed a script that could notify users about their hard drive having to little free disk space. Many of our users didn't know or care about this and it became a problem when software updates could not become installed (many of them use laptops and are travelling a lot, so we could not force them to use file server storage).

What is does

When you run the script you provide it with the amount of needed disk space in GB and in %. If the free disk space is lower than either of these values, the script will send a message to the logged on user.
At the moment I only care about the system drive.

For this picture I set very high requirements to get the msg (100GB or 80% free)

How I run it

I run this script as a scheduled task that runs on a 30 second logon delay and every hour when a user is logged on. It also runs as the logged on user to make sure the script can sent the MSG as the user to the user. To avoid the black cmd screen from popping up, I launch it with the invisible.vbs script.

The script

@ECHO OFF
:: Script get data from input and returns 0 if conditions met.
:: davpe67 2016-02-11
IF "%~1" == "" GOTO :SYNTAX
IF "%~2" == "" GOTO :SYNTAX

SET GBNEEDED=%~1
SET PERCENTNEEDED=%~2
SET TIMEOUT=%~3
IF "%~3" == "" SET TIMEOUT=1440

SETLOCAL ENABLEEXTENSIONS
SETLOCAL ENABLEDELAYEDEXPANSION
FOR /F "tokens=1-3" %%n IN ('WMIC LOGICALDISK GET Name^,Size^,FreeSpace^|findstr /i "c:"') DO (
 SET FreeBytes=%%n
 SET TotalBytes=%%p
)
SET /A TotalSpace=!TotalBytes:~0,-9! 
SET /A FreeSpace=!FreeBytes:~0,-9!
SET /A TotalUsed=%TotalSpace% - %FreeSpace%
SET /A PercentUsed=(!TotalUsed!*100)/!TotalSpace!
SET /A PercentFree=100-!PercentUsed!
::ECHO Total space: %TotalSpace%GB
::ECHO Free space: %FreeSpace%GB
::ECHO Used space: %TotalUsed%GB
::ECHO Percent Used: %PercentUsed%%%
::ECHO Percent Free: %PercentFree%%%

IF %FREESPACE% LSS %GBNEEDED% SET NEEDMOREDISK=1
IF %PERCENTFREE% LSS %PERCENTNEEDED% SET NEEDMOREDISK=1


IF "%NEEDMOREDISK%" == "1" (
 ECHO %COMPUTERNAME% is running low on diskspace on C: ^(%FREESPACE%GB ^/ %PERCENTFREE%%%^)>%TEMP%\msgfile.tmp
 ECHO If there is not enough free space the computer will become slow >>%TEMP%\msgfile.tmp
 ECHO becuse it needs more time to find sectors to write the data.>>%TEMP%\msgfile.tmp
 ECHO A golden rule is that at least 10%% and^/or 10GB ^(whatever is biggest^)>>%TEMP%\msgfile.tmp
 ECHO of free space is needed by workstations.>>%TEMP%\msgfile.tmp
 ECHO.>>%TEMP%\msgfile.tmp
 ECHO Some programs wont install or upgrade ^(old versions may be removed^)>>%TEMP%\msgfile.tmp
 ECHO until you have cleared at least %GBNEEDED%GB and^/or %PERCENTNEEDED%%%.>>%TEMP%\msgfile.tmp
 ECHO.>>%TEMP%\msgfile.tmp
 ECHO Most users have way more data stored locally than they need to.>>%TEMP%\msgfile.tmp
 ECHO Move files you are not working with to the home- or nobackup-server.>>%TEMP%\msgfile.tmp
 ECHO If you need help to make more space, contact support@yourcompany.com>>%TEMP%\msgfile.tmp
 MSG %USERNAME% /TIME:%TIMEOUT%<%TEMP%\msgfile.tmp
 EXIT /B 911
)

EXIT /B 0
GOTO :EOF
:SYNTAX
ECHO.
ECHO %~nx0 ^<GBNEEDED^> ^<PERCENTNEEDED^> ^<TIMEOUT^>
ECHO.
ECHO GBNEEDED is number of GB och free disk needed
ECHO PERCENTNEEDED is how many percent of total disk need to be free
ECHO TIMEOUT is how long the user message will remain on the screen