Introduction

Virtualization is cool. Last year, I set out to put together an all-in-one home server using ESXi and ZFS. I'm using this page to collect information about my experiences. Links to helpful resources I've found will also be provided.

History

I purchased a Dell PowerEdge T110 II on sale in 2013 on a whim. I'd been looking to buy a good server; however, as I'm too lazy to boot into Windows on the Mac Pro, I ended up buying a replacement power supply (with PCIe power) and a GTX 650Ti to turn it into a gaming machine. I daydreamed about making it a true all-in-one using VGA pass-through and KVM or Zen, but that plan was stymied by the lack of passthrough support on Nvidia's consumer cards.

I built a real gaming machine recently, though, and shifted the T110 II to its original intended duty as a virtualized home server.

Hardware

Dell T110 II

Acquired on sale in 2013, it has:

• Xeon E3-1240v2
• 32 GB of RAM
• 8x Western Digital 3TB Red hard drives
• Intel DC S3500 80GB SSD (SLOG)
• Intel PRO/1000 2-port Ethernet card
• IBM ServerRAID M1015 SAS HBA (flashed to LSI IT firmware)

I've been impressed by Dell's server hardware, unlike their consumer stuff. It's well put-together and you can find some excellent deals on it.

IBM M1015 ServerRAID

These used to be found on eBay cheaply, but demand for use in ZFS NAS devices has driven the prices up. The card is essentially an LSI 9480-8i. It's relatively simple to flash it to become an LSI 9211-8i and act as a dumb HBA. This makes it ideal for use as a passthrough storage controller for a ZFS pool.

Western Digital Red 3TB

8 largish spinning disks to host the bulk of the storage data. In multiple mirrored vdevs with 8 3TB drives, this gives around 12TB of usable space.

Intel S3500 80GB

A small, reliable SSD with good power-loss protection and lots of longevity. This is used as the SLOG device for the pool. Over-provisioned with hdparm to 10GB.

Intel PRO/1000 2-port Ethernet card

Unfortunately the T110 II includes a mediocre Realtek network adapter. The Intel card can be found cheaply used, can do NIC teaming, and has better performance than the Realtek. Intel's newer PCIe Ethernet cards, though much more expensive, can passthrough individual ports to VMs and have significantly lower power consumption.

Software

NAS host

I considered two options here: Napp-it (based on OmniOS) and FreeNAS. There are a few good comparisons between the two. Benjamin Bryan has a good one.

Napp-it

Built on OmniOS (which is based on Solaris), it is a fast, efficient, technically impressive distribution with a terrible and limited web UI, little in the way of comprehensible free documentation, and relatively fewer free features than some of its competitors. I like the tech, but don't like the hard to read docs, and smaller community.

FreeNAS

Built on FreeBSD, FreeNAS has been around for many years. They had a fork somewhat recently do to changes in the commercial-side of the business. NAS4Free is built on the old FreeNAS 7 codebase. I have a lot of experience with FreeNAS, since I've been using it an old Intel SS-4200. I prefer the web interface, it has a large community, lots of good-quality free documentation, and a lot of built-in features. It's not built for a VM environment, so it needs some tweaking to run well and you'll hear a lot of loud admonitions about losing all of your data if you do try to use it in a VM, but many people do so without issue.

Ultimately I ended up going with FreeNAS.

Setup

I installed ESXi 6.0 on a USB drive on the motherboard of the T110 II. Once ESXi was booted and ready, connect with a vSphere client and install FreeNAS. I followed an excellent guide for FreeNAS 9.3 on ESXi 6.0 from Benjamin Bryan.

Storage

Unfortunately the virtual machine hosting the ZFS pool cannot host its own storage, so some local datastore or external NAS is required for installing the ZFS VM. I used an older Intel 40GB SSD plugged into a 1x PCIe adapter (which needed some tinkering to get the unsupported SATA controller to work with ESXi.

Virtual machines

sipXcom

Formerly sipXecs, I like this a lot more as a SIP server than the various Asterisk distributions. It's cleaner and easier to work with and I like the High Availability and scalability features that I will never, ever use.

Ubuntu Server

Mostly I use this as a web server for testing sites and playing with web platforms and tools. It's also a back up for my local DNS server.

ownCloudNextcloud

Another Ubuntu install with ownCloud installed. Stores ownCloud data on the ZFS NFS share.

Update (February 11th, 2020): I moved my cloud storage to Linode in 2018 and switched to Nextcloud, which was forked from ownCloud in 2016. It seems like a more robust functional product based on my experience with it.

Miscellaneous play VMs

I've been playing with VyOS (open source fork from Vyatta). Considering it as a replacement for pfSense.