Network Monitoring with NEMS, NAGIOS and Rasberry Pi

Part of running my lab is ensuring that services I expect to run 24×7 are, in fact, running and doing just what I expect them to do. Unifi offers some level of monitoring around my internet connection but I wanted to go one further and be notified when services are unavailable or if a resource is consumed over a defined threshold.

Up until recently, I had planned on running this on a Linux box until I came across a Rasberry Pi B+ a good low power monitoring solution might be just what I need.

I’ve not gone into detail as to my hypervisor setup at home but running a monitoring solution on the exact server I want to monitor seemed to me like putting too many eggs in one basket. After some Google-Fu I came across NEMS Linux and open source project that offers a scaled-down version of NAGIOS to run on a Rasberry Pi.

This does come with some disadvantages right out of the gate. The Pi though very cost-effective and low power is using an SD card as it’s storage medium. Logging vast quantities of data to an SD card isn’t the brightest of ideas as they tend to burn out quickly with many prolonged writes. However, NEMS seems like a decent lightweight version of NAGIOS so I figured for the price of an SD card I’d give it a go. This version for NAGIOS is also somewhat limited with not as many features or support as the full NAGIOS Core for which are are not running here.

However, even with the disadvantages listed above the fact I’ve got a low power reliable device that’s able to monitor services within the network in a standalone way and gets me working with consoles more is a win-win.

The distro supplied was incredibly easy to install, as simple as installing Raspian OS to the SD card. Equally simple was it’s initial setup (excuse me for lack of photos of this). After that things get a little more complicated, as someone who’s never used NAGIOS monitoring before the UI was confusing at first but several configuration videos later and we’ve got full monitoring of key devices in the network.

I’m fairly comfortable with the consoles and was able to make minor tweaks to the application through it’s shell configuration (getting email notifications working was tricky). It does a healthy job of monitoring my switches, VM’s, hosts and website all through one system.

One of my concerns long term is the overall reliability of the SD card. One thing I know they really don’t like is lots of small reads / writes which is something that a service like this will do. As far as I understand it however with the optimisations applied to this distro it’s been reported as pretty reliable. Only longer term use will show if that’s anything to go by.

Given the low cost of a Rasberry Pi and high efficency I’m happy to have this running and keeping an eye on my network. I think if I expanded the number of endpoints I’d be temped to go for Nagios Core on it’s own dedicated box but for now this seems like a good idea.

Switching & Routing (Part 3)

The last post covered the basic setup of the home network. Getting the WiFi up and running and connected ASAP was of top priority so that the network devices that I don’t trust could get functioning as soon a possible. This time around we’re covering the home office networking and how I’ve got the VLAN networks configured.

Starting with the main piece of kit is the Dell PowerConnect 2824, I picked this up from eBay at a pretty decent price. I found a seller that was looking to get rid of it and I messaged offering slightly lower than what was asked and mentioned that it was for a “home lab” style project. Needless to say, the final price I picked the switch up for was £45.00. It’s not the worlds most current switch but it has VLAN support which is super important for what I want it to do.

My primary concern was to break out my network into logical segments that allowed me to easily switch kit in between networks should I need to without configuration. Handily the switch offers 3 banks of 8 ports which I used for this logical configuration as follows:

      • Home VLAN (ports 1-8) – No VLAN tagging
      • Work VLAN (ports 9-16) – Tagged VLAN 10
      • DMZ VLAN (ports 17-24) – Tagged VLAN 20

Depending on the growth of my Home VLAN I may need to expand its capacity and take ports away from the Work or the DMZ VLAN to accomodate for this but for now it’s easy to understand where things go.

This particular switch is located in the main office with all my other equiptment. I have to say that the audio levels are tolerable given I work in here everyday but that might not be the case for everyone. It’s louder than my home PC put it that way.

For connectivity (my office is located 1 floor above the Unifi USG) I’ve run CAT6 cable downstairs and tagged it to the skirting boards. It’s a rental property so drilling/wall routing isn’t an option and I’ve got to reverse it when I leave. This actually works really well as when I’m gaming /downloading/working as I’ve never got to worry about any lag or range issues that come with typical WiFi. As an additional plus it also negates any transfer speed issues giving me full 1Gbit ethernet speeds upstairs away from the incoming fibre connection to the house. I’ve used the wall plugs that use the existing copper of a building and have found that you’re luck & reliability can vary.

You’ll see above that I’ve split out the networks into appropriate components.

The Dell PowerConnect comes with a decent user manual and fairly straight forward configuration. It was easy to setup VLAN tagging for my other home networks (Work and DMZ). Setting up the networks in the USG is fairly straight forward.

What’s really important here is adding networking rules to the USG to prevent inter-VLAN communication. This is entirely down to my preference, as I didn’t want my work equipment talking to my home network and vice-versa. A really handy post from UniFi helps with this UniFi – USG Firewall: How to Disable InterVLAN Routing

I can now allow inter VLAN connections by adding the rules with a higher priority than the blocking rules. This allows my home LAN to connect into the other networks for sensible things like RDP or SSH.

Switching & Routing (Part 2)

In my last post, I gave a brief introduction to the networking for a homelab. In this part, I’ll talk about the other pieces of equipment that helped make it work.

Though your situation may vary here’s what I’ve used:

  • Unifi USG (approx £100)
  • Unifi Controller (free unless if you host yourself or you can buy the device from USG)
  • Either a managed or unmanaged switch (Unifi have a good offering for this) otherwise for VLAN support you’ll want a Layer 3 capable managed switch I’ve used a Dell PowerConnect 2824 (£45 eBay) for this. If you are not expecting any VLAN’s at this point unmanaged routing might be the way to go then a standard 5 or 10 port ethernet switch will work just fine.
  • If you’re expecting to setup WiFi within the home then a spare home broadband router (free), Google WiFi (£100+) or again Unifi’s AP range.
  • Plenty of ethernet cables.

First off I got started with the installation and setup on the Unifi USG and the Unifi Controller. The best way to go about this is to set up the controller first, I found that this would run best on one of my existing Virtual Servers.

Safe to say I’m running Windows Server 2016 and the installation of the controller software is really straight forward. Make sure that when you set it up, configure it to run as a service to avoid the controller software stopping when you log-off and if you’ll have the ability to configure the controller to auto-start on reboot. Don’t worry if you haven’t got Windows server licences hanging around they offer support for Linux distro’s as well which may be better suited in some cases.

Your Unfi Controller software must be able to see the USG and vice versa as you’re USG will be polling your controller (see Unifi documentation for this one). Once you’ve got the controller installed and the basic setup has been completed then you need to move onto configuring the USG. I use an existing modem/router from my broadband supplier, but if you’re doing this you’ll want to avoid any double NAT’ing this can cause all sorts of issues in routing traffic, it’s most certainly possible but not advisable unless you’re happy with double NAT within you’re network.

I’m running home fibre so I’m required to use my modem from my broadband supplier for the handoff of data to the network. If possible you’ll want to turn your router into a bridge mode (if it has one) this means that it’ll take care of the modem side of traffic (talking to your ISP) but you’ll be responsible for the rest. You’ll be taking full responsibility for ensuring that the firewall is installed and correctly configured.

You’ll want to configure your WAN settings before you switch your kit out to the new stuff, otherwise, you’re may lose internet connection before you’ve had the chance to google troubleshooting steps. The USG can be adopted prior to wiring it up but it’s a bit of a faff. I’ve found it’s always handy to have a mobile hotspot available to search for solutions to any issues you may encounter while the internet is off.

From there connectivity was simple. For my Google Home WiFi I’ve connected that via ethernet to the WAN2/LAN2 port with a different subnet to the rest of the house. I picked a 172.16.0.0 subnet for the Guest WiFi as I intend to use a 10.0.0.0 network in the home. Nothing wrong with 192.0.0.1 subnets but 10.0.0.0 networks are easier to type, remember and have a greater range of available addresses (as if I’d ever use them all).

You’ll see below that there’s no requirement for VLAN tagging at this point. The reason being as this is a physical port connected to the USG. As my other networks will be operating over 1 shared ethernet connection (they’ll be sharing LAN1 port) and so VLAN would be needed to segment those correctly.

Controller Settings for USG Guest Network

From here we can configure additional settings in the Google WiFi Android app. Google WiFi can and likes to operate as a router unto itself which you’ll have to switch off, the downside of this is that you’ll lose some of the advanced features that the WiFi has to offer but I was fine with that managing my settings through the USG.

Google WiFi App
Google WiFi App

Whilst some settings had been lost such as family control and splitting the AP into different subnets directly on the device itself I’ve still retained a few of the nice features with Google WiFi. The speed tests can be run, remote access to see who’s on the network and what is consuming the resources are all handy little tools in case the Unifi software goes wrong. You’ll also note that the USG isn’t aware that this is an access point (something you’d get with an Unifi AP). That loss of functionality is fine with me as it’s still available in the app and I can see who the clients are by IP, and MAC within the controller software.

One final point on the use of Google WiFi over the Ubiquiti Unifi offering was simply down to what equiptment I already had. The Google WiFi has been excellent for such a small smart looking device and the range, plus the 5 GHz & 2.4 GHz connectivity and switching has been excellent. However I have lost all the nice functionality that comes with Unifi Access Points and I may switch my homelab network over to it in the future.

Switching & Routing (Part 1)

We exist in an ever-changing threat environment, as a typical tech user I’ve got friends who visit my home and know my WiFi password, I’ve got IoT devices that run inside my home network, Alexa, Google Home, Firestick, TV’s, and Printers. Any of these could have a vulnerability.

So I work from home. Nice right? Great and all but I’m a firm believer of keeping work and home life separate. That’s difficult when you’re sat on a typical home broadband router with 1 WiFi access point.

We exist in an ever-changing threat environment, as a typical tech user I’ve got friends who visit my home and know my WiFi password, I’ve got IoT devices that run inside my home network, Alexa, Google Home, Firestick, TV’s, and Printers. Any of these could have a vulnerability (of course these get patched as per the manufacturer’s recommendations).

How comfortable are you writing code for an organisation working in this way? Though I trust myself enough to not do silly things on my network how can I expect that of anyone who visits? Same goes for what run’s on my work laptop is it going to have a product that scans my home network? I trust my business sure, but I’m subject to their IT security policies.

Finding a way to keep both sides of the fence happy can be tricky. I know that work would probably appreciate it if I segmented their equipment away from my home LAN and I’d be happy protecting my data in my home network.

In the past I’ve run networking for a small business, and I know that logical separation of concerns not only falls under a programming paradigm but also a networking capacity. VLAN seems the most appropriate option here, segmenting network devices so that I’ve got a Guest Network, Home LAN, Work LAN and have inter-VLAN rules to protect the assets either side of the fence seems a good approach in this scenario.

How to achieve such a goal? First port of call is to move away from the stock equipment supplied by broadband suppliers a quick Google search shows why you should consider doing this anyway. Besides that, for a reasonable price, you can obtain either older hardware and set up your own home-brew firewall using something like pfSense (my original approach to this issue). However, I like being inside support and my livelihood relies upon having a stable network connection not just for work but to keep my partner connected while I’m away.

I’ve been running with Ubiquiti’s offering of the USG which is a small dedicated firewall appliance that can sit at the edge of my network. I run speeds of 200Mbps down and 20Mbps up and this firewall has easily kept pace with all of that.

A few other pieces of equipment are required when exploring the Ubiquiti line-up, most notable is the lack of any WiFi capability in the router at all. I’m never a fan of a single appliance performing too many concurrent roles, and so prior to purchasing this I already had my Google Home Wifi which sets up a perfect guest network on the WAN2 port of the USG and has accompanying settings to ensure the guests are isolated in their own. That’s great as the first part of the network segment has been configured and all fits inside of a tiny package.

One thing that you’ve got to be mindful of is ensuring that you either purchase (at additional cost) a Unifi Controller which allows you to track, configure and deploy the USG or multiple USG’s over different networks or have a dedicated device (home server) that can run that software for you.

I’ve got to give credit to Unifi USG their software is pretty easy to get on with for a starting user. The USG itself has console access to be able to access some advanced configuration but for the not so console happy user the Controller software makes deploying configurations and upgrades really easy.

I’ll cover more of that in Part 2

Homelab – Inception

What is a homelab?

In short a homelab is a way to play around with old enterprise server equipment in the home.

It’s puzzling in this day and age why anyone would want to spend any hard earned money on older server enterprise hardware and give it a new lease of life in a home environment where it mostly acts as a space heater or even worse a space heater that generates enough noise that you can hear 2 rooms away…

A growing number of tech-savvy people world over will run their own home lab projects with each doing so for their own reasons be it a learning platform, charity, fun, hobby or all of the above. In an age of cloud services, and ever higher electricty bills it can be hard from the outside to see the appeal yet after only 8 months of running my homelab I find myself drawn to spending time building on it and coming up with new and inventive ways of getting the most out of what I’ve got

See now here’s the catch I’m a software developer by trade for a mid sized business in the UK. I work hard day to day developing for my work and while I learn alot from my time in the office there’s knowledge and expierence that I’ve gathered only through my time spent working on servers, hardware and software that’s outside of my comfort zone. Much of that expertise was never gathered in the workplace, in fact the whole reason I got into development was because I ran my own Battlefield 2 servers and php websites back in the day.

Grind can be key to developer burnout, focusing just on the latest and greatest products coming from Silicone Valley can be difficult and the noise of that work is admittedly hard to focus your downtime on. One way I find of testing new products and services on offer is through a homelab environment. If you’re learning as part of a hobby what’s there to stop you right?

Another way to draw expierence from running a homelab is that I’ve got the opportunity to test, experiment and play without the red-tape of the corporate world (which rightly has to be there). I have the chance to try new things and spectacularly fail without (hopefully) the publicity of my work colleagues picking up on my failings. It’s a fantastic environment to learn and progress.

This post, in fact, this blog is intended to be my musings and scribbles about setting up my homelab the reasons for doing so and my interests in keeping it going despite what some might think that it’s a bad idea.