In my last blog I explained how and why I ended up with the hardware I have. Here’s how I made it work.
So ESXI is nice and simple to install onto a USB drive, especially when you have Supermicro’s iKVM with remote media mounting. Stick the USB stick into the mobo, log into the iKVM, mount the ESXI installer iso as a remote CD and boot from it. Simple. Really, it “just works”. Reboot, unmount the remote media and boot from the USB. All without having to actually move from the keyboard, except the initial plugging in.
Initially I wanted to make sure things would work before I bought the NVMe M.2 SSD, so I used the unused space on the boot USB for my datastore. Its pretty easy to do, you can find how on the internet, but I’m not going to tell how you ‘cos its a terrible idea and it caused ESXI to crash, a lot. It did show me that ESXI would work though, so I bought the SSD.
The first issue with ESXI (I’m using 6.5 here) is that it does not want to let you pass through the C610 SATA controller. It makes sense I guess, ESXI wants it for using with data stores. But I have the NVMe for that, so how can we pass the SATA through? Well, this chap knows http://www.kaperschip.nl/2016/08/ahci-controller-passthrough-with.html so thats easy.
Next up the 10Gbit ethernet doesn’t show up under ESXI. It only sees the two GBit NICs. Once again the internet has the answer, https://tinkertry.com/how-to-install-intel-x552-vib-on-esxi-6-on-superserver-5028d-tn4t. Grab vib, install, reboot, woohoo!
Installing a Linux VM was easy, it “just works”. Nice. So pfSense. Just pass through the two i350 NICs. Err, ESXI won’t let me. For some reason it will only allow me to pass through one of the i350 NICs and one of the X552’s. Not both of either. Weird. I did some digging about pass through and how to force things to work (kinds like for the SATA pass through above). In the end I worked out that I just needed to add these two lines to /etc/vmware/passthrou.map
# Enable both i350 NICs to pass
8086 1521 d3d0 false
8086 1521 d3d0 false
Reboot. Although the first NIC is still greyed out in the GUI, it’ll pass through to a VM just fine. The only confusing thing is you have two devices called exactly the same thing to select in the VM PCI pass through config, so it may take a couple of tries to guess which to select to avoid passing the same device twice.
Thats pretty much it. I’ll not bore you with the oddities of FreeNAS nfs4 sharing to ESXI for a data store (NFS3 works much more easily), or why the Linux VM gets a much lower frame rates in Zoneminder than my old dedicated Linux box did (not solved that yet!). You get the idea though. It all works with a bit of tinkery.
Leave a Reply