Networking: VLANs!

Up until now, there was no real network segmentation in my homelab. Everything was on the same network, which made it difficult to manage and secure. It became even more relevant when I planned on doing a “cyberlab” (more on that later) and this server absolutely needed to be fully isolated from the rest of the network.

VLANs to the rescue

I never touched VLANs in my life before this, so it was a fun little journey especially with the difference in configuration between Cisco and Mirkotik.

Different VLANs were created for different purposes:

• Wifi VLAN: All wifi devices are on this VLAN, isolated from the rest of the network. They can only access the internet.
• Cyberlab VLAN: This is the “dangerous” VLAN. It can only access the internet, and is reserved for all the VMs that will be used to experiment with some malware/Windows AD and other fun stuff.
• Server VLAN: This is the main VLAN for all servers.
• Management VLAN: This is the VLAN for all management interfaces (Proxmox, UPS, Router, etc.).
• Workstation VLAN: This is the VLAN for all trusted PCs.

I started using colored cables to make it easier to identify the different devices, but I quickly realized this would not be enough. I will have to start thinking about a proper labelling system at some point!

Main server: Hardware upgrades and moving to Proxmox

Before that, this was a simple server running on Debian with consumer hardware. This is served me well for the last few years, but as my side projects grew, I started running low on RAM: 120GB was simply not enough anymore. It was time to start thinking bigger. The hardware has been changing to the following:

• CPU: AMD EPYC 7413 (24c/48t). This is a older platform CPU, but fits my need and is pretty affordable on the second hand market.
• RAM: 512GB DDR4 ECC.
• Motherboard: Supermicro H12SSL-I-O. This was actually the most expensive part of the build, but server motherboards are not cheap. This finally gives me IPMI support which was one of my main requirements!
• Storage: 2x 4TB NVME Micron 7400 PRO 3.84TB M.2 in RAID 1. 4TB was a must-have and I was able to get those new for fairly cheap. That also gives me room for expansion with U.2 drives in the future.
• Network: 2x 10GbE SFP+ ports. As I started working with weather model data, 10GbE started to be a requirement.
• CPU Cooler: Noctua NH-D9 TR5-SP6 4U. This is a SP6 socket cooler but luckily it has the same physical dimensions as SP3 and is properly oriented for my case exhaust - unlike the Noctua SP3 cooler which was blowing hot air into the PSU. Noctua kindly sent over mounting bracket adapters for the SP3 socket!

Proxmox

Migrating to Proxmox was a bit painful at first (baremetal to hypervisor is really not a fun process), but I am now really happy with it. I have been using it for a few months now and I am really happy with the performance and stability. I have been able to finally split my workload into different VMs, making it easier to manage and more importantly more secure.

This has allowed to move all the non-critical ALS workloads into this Proxmox host, reducing hosting costs at the same time. This also hosts all my other side projects (weather stuff, monitoring VMs…) while still leaving some headroom for future projects.

I’m also using Proxmox Backup Server for offsite encrypted backups as the integration was pretty much seamless. My trusty Synology NAS still handles the local backups within a 2x18TB drives RAID 1 volume.

Electricity cost

It’s not cheap. The whole rack costs around 70€ a month to run, but to put that into perspective, a similar spec server (so that’s just the main server, not all the network equipment/NAS) would be just below 700€/month at OVH. Sounds like to great deal to learn all the surrounding stuff :)

UPS & NUT

I was finally time to replace the old dying APC UPS. After testing a APC SMX1500, I was really disappointed by the electrical noise this thing was making. I sent it back and finally went with a Eaton 5PX 1500 G2. It has been a great unit so far, while being near silent in normal operation and allows for network management. I really wanted a rack mounted one to fit everything inside it and avoid cables going in and out everywhere.

Network UPS Tools management

In case of power failure, I need something to easily shut down all my servers when the battery runs low. I went with Network UPS Tools that is running on a Raspberry PI 4, with all my servers listening to it. When the battery runs low, it will send a shutdown command to all the servers. All servers and the UPS itself are configured to automatically power back on when power is restored, which should ensure fully automated recovery. The network card in the UPS also allows me to power cycle each outlet groups, just in case.

Cyberlab

I always wanted to have a server to play around with cybersecurity related stuff. I got a second hand Dell R340 for this purpose, which is only powered on when needed (those things are loud). Fun fact, the service tag returns some McDonald’s related custom configuration… maybe this was actually used in McDonald’s before? I’ll be able to play with Windows Active Directory etc. while being fully isolated from the rest of the network, when I’ll want to start playing around with some attack scenarios. I’ll probably have more on this in the near future!

With 22 servers running overall for Apex Legends Status (=ALS) needs, having a centralized logging platform has quickly became mandatory to be able to access all logs when needed from a single place.

The dev blog will detail the most important parts of the logs flow, starting at Filebeat/RabbitMQ level, Logstash and finally the ELK stack.

The infrastructure could be summed up as this, in a very simplistic way:

Data collection

RabbitMQ & AMQP

Most of the time, collecting logs from different sources is trivial (for example, with filebeat for Apache2 logs - which we’ll talk about shortly after). However, sometimes, when using more complex/home-made solutions, you have to think a bit outside the box. We’ll take a concrete example of ApexWS, the websocket server used within DGS (statistics platform for Apex Legends tournaments) that sends and receives events from the game client.

First of all, AMQP (Advanced Message Queuing Protocol) is a way for different systems to send messages to each other reliably. RabbitMQ is a popular tool that uses AMQP to manage and route these messages, here relaying messages between the DGS websocket server and Logstash. Hardware resources impact is very limited, and it can handle hundreds if not thousands of messages per seconds without issues.

Each time the DGS websocket server receives an event, the full payload is encoded into JSON and sent to the local RabbitMQ instance. It contains everything we want to log: the game event, user ID, current game ID, DGS server identifier, etc. RabbitMQ keeps that message into an internal RAM buffer until it is fetched and acknowledged by the remote Logstash instance. On DGS, all dedicated servers having their own complex systems will have their own RabbitMQ instances, linked to Logstash through distinct pipelines. It’s really important to set a max queue size on the rabbitMQ server, to avoid unwanted resources issues: in the event of a Logstash server failure, the messages would pile up in RabbitMQ… until the OOM killer comes by. I personally have a max queue of 10,000 (this should be adapted according to your events per second rate), which is enough to buffer messages while the Logstash server is restarting for example.

RabbitMQ comes with a wonderful web interface, as shown below. You can basically do everything you need from there and avoid using the CLI. It’s also great to have a quick overview on all the existing connections, the size of the buffer and the overall health of the system. UI doesn’t feel that modern, but it’s not really made to watch all day and it does the job perfectly.

For DGS websocket needs, all messages are sent to the same queue (ApexWS-prod which you will see below in the Logstash conf), having a unique identifier depending on the DGS server instance and the environment (prod, dev, ALGS).

Filebeat

For more common systems, such as the Apache2 web server, a simple Filebeat instance is deployed on all the needed hosts.

If we take ALS Apache2 logs as an example, it’s as easy as this:

filebeat.inputs:
- type: log
  enabled: true
  paths:
    - /var/log/apache2/apache2-access.log
  fields:
    log_type: apache2
    hostname: als-core

Filebeat will continuously read all the logs coming from the Apache2 access logs which are in a standard format, and forward it to the remote Logstash instance. The logstash step is not always needed, but it felt easier to implement with the pre-existing rabbitMQ infrastructure.

Data reception & Formatting

This part is handled by Logstash, on the core Elastic server. Each distinct log source has its own logstash pipeline listed in the pipelines.yml file. If we go back to our previous example of DGS websocket server, the pipeline is loaded as follows:

- pipeline.id: apexws
  path.config: "/etc/logstash/conf.d/logstash-apexws.conf"

Which points to the following config file:

input {

        rabbitmq {
                id => "rabbitmq_logs_apexws"
                host => "RABBITMQ_SOURCE_IP"
                port => RABBITMQ_SOURCE_PORT
                vhost => "/"
                queue => "ApexWS-prod"
                ack => false
                user => "RABBITMQ_USER"
                password => "RABBITMQ_PASSWORD"
                codec => "json"
        }
}

filter {
        mutate {
                remove_field => ["[event][original]"]
        }
}

output {
        elasticsearch {
                hosts => ["ELK_IP"]
                index => "dgs-apexws-%{+YYYY.MM.dd}"
                user => "LOGSTASH_USER"
                password => "LOGSTASH_PASSWORD"
        }
}

This config will fetch all the events in the RabbitMQ buffer, decode the JSON, and output it into ELK in the designated index. The config file is roughly the same for all pipelines, other than the input which may change, and the destination index.

Data indexing & Viz

Now that our data is indexed into Elastic, we can make some really cool stuff with it. We start by creating index patterns, which will basically allow us to query all matching indexes as the same data source: it’s usually recommended to create a separate index for each day for performance and ease of data retention policies. By taking the example logstash config above, dgs-apexws-%{+YYYY.MM.dd} data will be indexed into the index with the current date. All the entries are then visible in the Discover section of Elastic:

But it doesn’t stop there! The most interesting part is the ability to create custom visualizations, and most importantly: dashboards.

This is an example of a dashboard made using the DGS websocket server data. We can filter data in each individual visualisation, with a lot of different types of viz. Possibilities are limitless, and can really help on a security or data analysis point of view.

On the performance side of things, everything has been running pretty smoothly so far on a single ELK node. The node ingests between 100 and 500 events per second most of the time without much trouble, storage being one of the main issues: most of the logs are kept for 30 days only (which is more than enough in most cases), while some others have a much longer retention period to comply with obligations. The Logstash + ELK combo eats quite a lot of CPU power and RAM however, this kind of traffic wouldn’t run on a Raspberry PI.

As a sidenote, data retention is handled using what we can Index Lifecycle Policies, which moves data in a specific storage tier (hot, warm, cold, delete) depending on how old the data is. You would typically move it down gradually through warm and cold, and finally delete when you want to get rid of that data.

Why not Opensearch

Quite a long time ago, I decided to give a try to Opensearch, a fork based on ELK before they changed their licensing. While it works, my main issue was the lack of updates and how late it was on new features compared to ELK. The UI/UX also feels much better overall on ELK compared to Opensearch. Hopefully I’ll be able to jump back to Opensearch after the product grows!

Online resources are also much more available for ELK compared to OS, where it’s sometimes quite painful to find any documentation or info about what you’re trying to achieve.

ELK feels overall much better on a day-to-day usage.

Routing: Mikrotik CCR2004-1G-12S+2XS

Overview

The most important piece of hardware: the router! This replaces the good old ISP router (which was not powerful enough… it was unfortunately not able to handle a large amount of concurrent packets - and why would you make something easy if you can make it harder? :)). In the past, this was a EdgeRouter 4, which was changed after I decided to go with a multi-gigabit ISP connection. The Mikrotik CCR2004 is a perfect router for this, with 12 SFP+ (10Gb/s) ports and 2 SFP28 ports (25Gb/s), having plenty of headroom for future upgrades, while also being cheaper than most high-end brands. A bit noisy at first, a fans replacement was needed. Fortunately, it’s really easy to open and swap the stock fans with a pair of Noctuas, making it dead silent. A thermal repaste was also necessary to cool this beast down, with a 20°C drop - which is really worth it and easy to do. This also acts as the firewall to/from the outside world, only keeping open ports for services needing it. Great thing from this, it barely goes above 10% CPU usage when having high traffic!

ISP Router replacement - the “fun” part

My main requirements when chosing this router was it to handle 10Gb/s, but also being able to put my ISP router back in its cardboard box. French ISP have a good history of using non-standard protocols to authenticate their clients, a fully customizable router was necessary for this. We’re lucky to have a few people who worked on reverse-engineering the auth flow, allowing people to tinker around and avoid using their ISP router by setting the correct DHCP client parameters (basically, making your ISP believe it is their router trying to authenticate with the network), and some specific VLANs. The base configuration is quite straightforward, but it gets really hard when needing additional services, such as TV. If you want to dive into this, lafibre.info is a great resource to get started. It gets even funnier (not really) when having to replace the ONT: it involves reprogramming the optic to replace the serial number and its vendor ID, which can be done by using some FS.com optics. Doing all this gives way more liberty, while also being dependant on any ISP network changes (auth changes, etc.). It’s really beneficial in my case as I have a 2Gb/s plan with my ISP, which I would only be able to use up to 1Gb/s per port with their provided router.

Configuration can be done through the RouterOS CLI, or by using the old RouterOS GUI (Winbox). This is one of the main downsides of Mikrotik products, the software feels really old.

Core Switch: CRS 317-1G-16S+

An other great piece of Mikrotik hardware, with 16 SFP+ ports (10Gb/s). It handles all 1Gb/s+ switching needs, with a 10Gb/s DAC uplink to the router. After swapping fans and thermal paste for the same reasons as the latter, it also runs dead quiet. Unfortunately, SFP+ RJ45 adapters had to be used due to constraints on the connected devices (lack of compatible SFP ports, mostly). It’s really important when using such adapters to space those evenly, as those get REALLY hot when running. This can cause early failure, and most importantly make the fans ramp up to maximum when having heavy traffic! It’s not used as its full potential at all for now, not having any other 10Gb/s devices linked to it at this time. This however still allows me to use my full bandwith from my workstation, which only supports up to 2.5Gb/s - some improvement to be made there in the future, maybe? :-)

1Gb Switch: Cisco SG200

This is a really old piece of Cisco hardware with 24 gigabit ports - and it was also my first switch in my homelab, a few years ago! It still works wonderfully, and its main benefit is to be passively cooled. It’s connected to the core switch, and distributes network connectivity to all gigabit devices behind it, through a patch panel right below it. Small patch cables make it tidy, which is way different than the back of the rack!

NAS: Synology RS422+

Until I started putting all my stuff in a rack, I was running a desktop format NAS (DS418+). It was great, and I really enjoyed the DSM OS. I only use it for storage (no docker containers or anything running on it), so I upgraded to a simple RS422+ after moving to a rack. It consists of 48TB raw storage capacity, spread across 2 nodes in RAID 1 each. I hate deleting stuff, and tend to always keep storing more and more - this gives some headroom, but I’m not sure it will last long!

A first “production” server, Reese

Named after a Person of Interest character, this is my main homelab server where I throw most of my workload. This is encased in a 4U Textorm case (I absolutely do not recommend it, it’s a pain to work with its rails), using desktop-grade hardware. This is still plently of power to run things, while having the benefits of low noise levels with Noctuas fans. Its currently config is a Ryzen 9 5950X CPU (16 cores/32 threads, more than enough), 128GB of DDR4 RAM, a GT710 GPU for video output and a random set of SSDs/NVMEs for storage. Most important part of storage is a Intel Datacenter grade SATA SSD for boot/heavy write data, and a pair of Samung NVME SSD in RAID 1 for main storage. My main issue when using SSD is their write lifespan, as I’ve already destroyed a few SSD that way. The Intel one is specifically made for this kind of usage, with a high TBW. A typical consumer-grade SSD from this period would have a 400TB write lifespan, while the Intel one is at 1.2 PB - on top of all other data integrity features on such kind of datacenter-grade SSD. One of the main downside when using such hardware is not having any IPMI control (= controling your server when it is offline, allowing you to boot, stop, connect virtual medias etc. to it). The GPU output is connected to a PiKVM box that does all this. It allows to have a KVM for this server, and also control its ATX I/O which makes it perfect to remotely manage if necessary. This server handles a few of my side project, and still one non-critical ALS component: a Redis database used for leaderboards generation. Redis barely uses any CPU resources for this usage, but needs large amounts of RAM (~80GB as of now); this would be really expensive if hosted through a dedicated server provider or in the cloud. It also handles some other components, such as Opensearch/Logstash used for logs ingestion and management (which will be the next devblog subject!).

The lab server, Finch

Named after yet an other Person of Interest character, this is a Dell Poweredge R720 server with 2 Xeon CPUs (6 cores/12 threads total),128GB of RAM, and a few SAS drives running a Proxmox install. It’s a really power hungry server, that I’m only using when having to play around and learning new tech. It’s also quite noisy, but this old version of iDRAC allows you to control the fan speed manually through impitool, which make it barely audible when running while still having decent temps.

… the nasty part: UPS, PIs, and random stuff

Cable management is OK in the front of the rack, but it’s a whole other story when looking at its back. This is where all the random PIs and other domotic things are stored (temp sensors, light controls, some other secret stuff…) and the wifi AP. All the critical devices are plugged into a APC UPS that gives around 20mins of running time in case of a power outage, which has always been more than enough for now. When its battery runs low, it sends a signal to all running servers to make them shutdown gracefully, ensuring data integrity - and all servers are automatically restarting when the power comes back.

The best thing about this rack is that it still has 4U of empty spaces - what’s next? :-)

What is the goal of this system?

The primary aim is to provide Twitch viewers with real-time stats during a tournament, ensuring the data is concise enough not to overwhelm viewers yet informative enough to enhance their viewing experience. This includes displaying a live scoreboard, minimap, and player stats/inventory without distracting from the main stream.

This has a huge value for viewers that want to be engaged in the stream, allowing them to easily have a look at the players healing items, weapons or stats, being important factors in a competitive Battle Royale game.

The tech behind Live Stats for Livestream ("Shaw")

Quick overview

Live Stats for Livestream (code name Shaw, being the internal name of this project - a Person of Interest reference) is a PHP backend that hooks to the existing Redis live game data cache of the ApexWS server, and exposes endpoints to get formatted data being returned to end users.

DGS Uplink

The core of the system remains ApexWS, the homemade websocket server that powers the existing DGS (“Detailed Game Stats”) feature on the Apex Legends Status website. As a quick reminder, this is a PHP websocket server that talks with the game client of the game admin. This allows the game client to send events live to the server (player damage, items pickup, kills, etc.) either in a JSON or a Protobuf format. In this case, DGS (more precisely the ApexWS server) is not directly handling client queries. By separating critical components, we ensure the stability of the tournament ecosystem: ApexWS is a critical part, and any crash could result in the loss of ongoing tournament data. However, Shaw isn’t as critical; if it crashes, it can quietly restart without significantly impacting the tournament or the end-users.

End user <-> Shaw Compute

First, the end user will make a request every second to the dgs-shaw-lb.apexlegendsstatus.com endpoint. This goes to Cloudflare first, that handles the Web Application Firewall, before being proxied to the Shaw Compute server, that will check if a game is in progress for the provided TournamentID and if Live Stats is allowed for this tournament by querying the DGS MySQL database. If none found, we return a waiting message to the user with a matching description message.

Redis Shaw Cache

If a game is currently running, a query will be sent to the Redis Shaw Cache (which isn’t the same cache as ApewWS) and check the latest update timestamp. If the latest data is newer than 1 second, we simply return the saved JSON payload to the user, saving precious computing resources and processing time. The overall HTTP request takes about 35ms in this specific case (client <-> server).

ApexWS query

If the data is older than 1 second, Shaw Compute will send a query to the Redis DGS Live cache for the corresponding internal ID. This bypasses the need to connect to ApexWS using websocket, which would have a much longer processing time (due to the websocket protocol & authentication on ApexWS). Here, we simply hit our Redis cache in a readonly mode. The data is then parsed by Shaw Compute, and the resulting payload is saved into Redis Shaw cache with a TTL of 1 second, before being sent to the end user. Doing so, the overall processing time is about 140ms.

In the end of the process, the JSON payload is parsed by the user browser (Javascript), and the DOM is updated accordingly.

Twitch channel status

To check the status of a Twitch Channel, a service simply calls the Twitch Helix API every minute, and updates the DGS MySQL database. If the channel is online, we also save the viewers count, which is also displayed on the DGS user interface. A quick mockup PHP code could be as follows:

$curl = curl_init();

curl_setopt_array($curl, array(
    CURLOPT_URL => 'https://api.twitch.tv/helix/streams?'.$channelToCheck,
    CURLOPT_RETURNTRANSFER => true,
    CURLOPT_TIMEOUT => 0,
    CURLOPT_FOLLOWLOCATION => true,
    CURLOPT_CUSTOMREQUEST => 'GET',
    CURLOPT_HTTPHEADER => array(
        'Client-ID: Twitch app client ID',
        'Authorization: Bearer '.$twitch_access_token
    ),
));

$response = json_decode(curl_exec($curl), true);

var_dump($response);

curl_close($curl);

Scaling & current limitations

In its current state, Shaw is able to handle about ~2000 concurrent requests (with a 1 second interval) on a single server. The main bottleneck isn’t computing power as the caching does a pretty good job for this, but mostly bandwidth. Bandwidth is expensive.

It would however not be hard to scale the current system, as it was made with this in mind. Depending on the success of the feature, more processing power could be added, but will come to the question of funding: DGS is entirely free to use in its current state, and if this feature were to be used for a tournament of the size of ALGS for example, it wouldn’t be possible without external funding.

There are also some easy workaround to mitigate this to some extent: reducing the payload size by optimizing it, or just increasing the delay between each HTTP calls to Shaw compute.

In the end, I believe this is a good proof of concept to show what could be made to enhance the Apex Legends viewing experience on the esports side of things, and I really hope we’ll see this kind of feature for ALGS at some point.