Being currently assigned to the issue about ARM support for Resonite headlesses (GH-2555), time for an update since there hasn’t been one in some time.
First off, everything is looking great, current status being:
6 PRs are currently open (FreeImage, Opus, Crunch, Assimp, MSDFGen, RNNoise)
1 PR has been merged š (Brotli)
1 repo is missing (Freetype)
This marks the first ARM-related PR being reviewed and merged into an official repository, being the PR #1 on the Brotli repo, which bundled Windows, Linux x64 and Linux ARM CI/CD builds.
As a reminder, I am currently providing a complete package of all libraries built directly for ARM on my website.
Next steps would be to:
Get an official fork of the Freetype repository (requested on 2025/04/09)
Create a container image bundling my libraries and a way to download the headless easily on ARM machines
Get all the current PRs reviewed and merged
The second one is more important as distribution is probably the biggest issue for complete ARM support of the headless, SteamCMD not supporting this architecture.
I am very confident to say that we will reach official support very soon, given how well this has been going so far.
My plans after this feature is shipped is to work on the macOS support (GH-1412) as it’s also marked as ācommunity help wantedā.
Today, from approximately 16:30 UTC to 17:45 UTC, the Resostats Dashboard which provides various public metrics on Resonite was offline.
Background
Routine maintenance was being done on the machine hosting Resostats, namely updating the packages, containers, cleaning up some debugging tools. Configuration changes were committed to try and have the TSDB sync faster to the S3 storage bucket that backs the whole instance.
Metrics stored on the Mimir instances do not have any set expiration.
The S3 bucket itself is fully replicated and backed up using Restic in multiple places, including rsync.net as an external one.
The cause
While committing changes to the mimir configuration, the compactor_blocks_retention_period configuration key was swapped from 0 to 12h.
The compactor_blocks_retention_period configuration key in mimir specifies the retention period for blocks. Anything older than the set amount will get marked for deletion, then cleaned up. You can read more about this in the official mimir configuration documentation.
This prompted the mimir instances to start marking blocks older than 12h for deletion, thus cleaning inadvertently years of historical data.
Restoration
The error in the configuration was quickly spotted and corrected, but the blocks already marked for deletion were already being cleaned up regardless. Given the backup hosted on rsync.net was the closest and fastest for this available server, the decision was taken to restore everything from there.
The restoration process was easy enough, given Restic provides a nice command for this:
Most of the time spent was the stressful wait for the backup to be downloaded onto the machine.
In the end, about 12h of metrics were lost, which is not that much considering the scale of the outage.
Learnings
From now on, a backup will be done before starting any maintenance. The current backup strategy has also been proven robust enough to withstand an event like this one.
Turns out having a proper backup strategy is damn effective.
Given sessions in Resonite are hosted by the players themselves, IPv6 is very useful in this context as there are no needs to battle with CGNAT or other network shenanigans and restrictions ISPs might put in place to save up on IP space.
As full native IPv6 support is currently being worked on (see GH-143 for a more in-depth status), some parts already do support it.
Joining any session using a network string like lnl://[<IPv6 address>]:<port>/ is already supported, which only leaves the relays and bridges needing IPv6 support, a mod made by a community member already existing to solve this issue until official support is added.
In the end, Iām very confident that we will see full native IPv6 support land in Resonite this year, if not already in Q1, given this is actively being worked on.
Once those those two issues (relays and bridges + AAAA records missing) are addressed, the only thing missing IPv6 will beā¦ the bug tracker, GitHub, which I already talked about in this article (spoiler, we aināt seeing IPv6 from them anytime soon).
Also special thanks to ProbablePrime for looking into it!
If you are a Resonite player and are in the Discord guild, you might be familiar with the #active-sessions channel, in which a bot displays the 10 most popular sessions on Resonite as well as some stats.
What you might not know, is that Iām the author of this bot, that I originally started as just a small oneshot project to have some fun.
For some background, when Neos was still a thing (technically still is, but in what state), a bot like this was in the Discord guild, showing sessions and game stats like server metrics and session counts.
When Resonite was released, the channel was there, however, no metrics or posts were ever made, saying that the bot would be revived at some point in the future(TM).
At the time, I was a bit bored and wanted a new project, so I decided to start experimenting with .NET Discord bots and in term set myself the objective to re-create the original bot.
Why .NET? One reason being that I use .NET all the time as itās fairly easy to use, the other one being that most of the Resonite community also knows .NET due to being used to make mods and whatnot.
The bot itself is fairly simple and divided in multiple parts built around the .NET Hosted Services:
Discord client service – handles the connectivity to Discord
Interaction service – provides the command handler for the few commands the bot has
Startup service – sets up some bot must-haves like the logger and other data
Message service – the core functionality of the bot that runs all the logic that makes the magic happen
Healthcheck service – this one is optional, but important when hosting the bot in Docker
Letās go through all of those in detail and see why they were made that way.
The Discord-related services
Iām gonna group those together as they belong to the same component really: handling Discord-related shenanigans.
The bot has a few commands:
/info which shows some info about the bot
/setchannel which sets the channel in which you want the notifications
/which which shows which channel is set as the notification one
/unregister which unsets the notification channel
/setting allows you to toggle some settings like if you want thumbnails or not
All of those commands (with the exception of /info) are admin-only.
This part is honestly fairly boring and straightforward, the rest just passes a Discord bot token, connects to the chat service and tries to log in.
First off, the bot uses a relatively simple SQLite DB to avoid everything being hardcoded. The first versions were using direct channel IDs, but this is far from ideal if you want something modular without having to host multiple copies.
The DB basically stores the guild ID, channel ID and settings for the bot to send the updates in the right place, containing what we want.
Speaking of settings, there is only one so far: show the session thumbnails or not. The reason for this is a difference between the Discord & Resonite ToS. While nipples arenāt considered sexual on Resonite, they are on Discord, meaning having a session thumbnail showing nipples on Discord without gating the channel to 18+ users would be violating the ToS of the platform.
One thing I am quite proud of is how stable the bot it, nowadays it rarely, if ever, crashes alone, which it used to do quite often.
The bot is made to handle errors gracefully and never shut down or crash the program unless something really, really warrants it. When running this bot, all the errors that would normally crash it are instead logged with an error message and stacktrace to make it easy to debug.
Another thing to note is that the database schema hasnāt been updated since the bot basically released and touching it is considered a very last resort thing. Having the DB break during an upgrade would be disastrous, requiring all admins to re-set the notifications channel. As they say, if it aināt broken, donāt fix it.
Out of all the variables in the mix, Discord is the most unstable one, having lots of outages, sometimes lasting hours at a time, just being slow for no reason whatsoever or thinking existing things (such as channels or messages) donāt exist even though they do.
This is why the whole checking logic exists, it will first check if the channel exists, if it does will check if the message exists, and if it does, try to update it. If it fails at any point, it will try again for a while then delete the message, try to re-send it, and ultimately, if this fails, delete the channel from the DB and the admin will have to re-set the notification channel again.
The re-try was implemented after some issues raised from Discord:
The bot would forget about the message (because Discord said it didnāt exist anymore) and would send a second one in the same channel or over and over until restarted
Sometimes the checks would fail on first try and delete everything gracefully without notifying anybody
Bot would crash because everything āceasedā to exist
On the Resonite side, if an error happens while contacting the API, the bot will just skip this cycle and try updating the next time (one minute by default). This used to crash the bot (whoops) in the early days.
The latest addition made was the Docker healthcheck, given recently the bot crashed in the main Resonite guild (GH-3521) and no monitoring was triggered.
Now the bot has a small HTTP server running, simply returning the date that a curl instance will check every 30 seconds.
The CI/CD
Itās no secret that I love GitLab as a software. I also work daily on and with it in my day-to-day job.
The CI/CD used in this project is extensive, but classic:
Secret detection
SAST scan
Dependency scanning
NuGet build
NuGet deployment
ARM64 and x86 Docker builds
Release publishing
The first three are kinda explicit, and will warn if any secrets have been pushed, if any unsafe codepaths are detected or if any dependencies needs updating.
Now the most important thing to highlight are the separated Docker builds for the two architectures. I originally tried combining the builds into a single one as you would do by specifying multiple architectures in buildx, however this did not work.
An error when building ARM on x86 (with virtualization) and vice versa would always arise, though the same command would work for other projects.
To avoid errors when doing things manually, the release process is also automated, triggering when a tag is detected. It will basically build as usual with the version tag and then publish the release following a markdown template. It will also automatically fill-in some details like image tags, etc from the changelog.
Now for the self-criticism: if I had to restart the project from scratch, I would probably opt into an even less complex design. Some services are really huge files that only increased in complexity with time.
Currently nothing too bad, but I think a refactor would be warranted to decrease complexity and make it more maintainable.
I wouldnāt touch the language though, since the botās footprint is really small, only about 30Mb to 50Mb of RAM used total during normal runtime.
In the end, this bot is a really fun project to make and maintain, and Iām extremely happy that it got canonized and used in an official manner.
As you may know, I made a mod for the platform Resonite that allows you to export metrics from the game as OpenMetrics data readable by Prometheus amongst other software.
In this post, weāre gonna see:
How to setup a basic Resonite headless on Windows
How to install the Resonite mod loader
How to install the Headless Prometheus Exporter mod
How to install and configure Grafana and Prometheus to scrape the metrics
Setting up a headless in Windows is really easy. To first get the headless files, there are two ways that all begin the same way, sending /headlessCode to the Resonite Bot while logged in-game. This will give you the code needed to activate the beta branch for the headless.
Using the Steam client is the easiest. Just right-click on Resonite, hit āPropertiesā, then āBetasā, enter the code you previously got into the field and click on āCheck codeā. You should now be able to select the Headless branch in the small dropdown and will download it automatically to your Steam game folder.
When using SteamCMD, unpack the zip file it comes in in a directory, hold SHIFT and right-click, then select āOpen PowerShell window hereā. Once the PowerShell open, you can use the following command to download the headless (replace your account name, password and headless code):
You should now be able to find the headless within the resonite\Headless directory near where SteamCMD is unpacked.
Now, to run the mod itself, the headless is not enough, we need to extend it via the Resonite Mod Loader. Its installation is straightforward, as outlined by their README file:
Download ResoniteModLoader.dll to Resoniteās Libraries folder (C:\Program Files (x86)\Steam\steamapps\common\Resonite\Libraries). You may need to create this folder if itās missing. Place 0Harmony.dll into a rml_libs folder under your Resonite install directory (C:\Program Files (x86)\Steam\steamapps\common\Resonite\rml_libs). You will need to create this folder. Add the following to Resoniteās launch options: -LoadAssembly Libraries/ResoniteModLoader.dll. If you put ResoniteModLoader.dll somewhere else you will need to change the path. Optionally add mod DLL files to a rml_mods folder under your Resonite install directory (C:\Program Files (x86)\Steam\steamapps\common\Resonite\rml_mods). You can create the folder if itās missing, or launch Resonite once with ResoniteModLoader installed and it will be created automatically. Start the game. If you want to verify that ResoniteModLoader is working you can check the Resonite logs. (C:\Program Files (x86)\Steam\steamapps\common\Resonite\Logs). The modloader adds some very obvious logs on startup, and if theyāre missing something has gone wrong. Here is an example log file where everything worked correctly.
Those same instructions also apply to the headless software. On certain Windows version, you might want to right click and open the properties of ResoniteModLoader.dll and 0Harmony.dll then check the āunblockā checkbox as it will prevent the mod loader from functioning correctly.
Once this is done, head over to the releases tab of the Headless Prometheus Exporter mod and download the HeadlessPrometheusExporter.dll file. You will need to move this file in the rml_mods folder that should be located in the headless directory. If this folder isnāt there, you can create it. Also check the properties of this dll as well for the unblock checkmark.
Now that the mod installation is done, we have one last step: configuring our headless. This step is also incredibly easy, being documented on the Resonite Wiki. I can recommend going in the Config folder then copying the example configuration file provided to fit your needs. It is not recommended to start out with a āminimalā configuration file, that might lack some essential settings and result in the headless not working as intended or not starting at all. Once you are familiar with what goes where and does what, feel free to trim the configuration file following your needs.
If you still have your PowerShell window open, you can type cd resonite\Headless to navigate to where the executable is and then use the following to start it with the mods:
After waiting a bit for it to start, you should be able to visit https://localhost:9000 in a web browser and see some metrics being displayed such as some world stats, engine FPS and many others.
Now we can tackle the last issue: how to display those metrics. For this, weāre going to use Prometheus in combination with Grafana, which are hands-on probably the best solution for this.
Weāre gonna use my pre-made minimal Grafana setup for this. You can obtain the files by either using git clone https://g.j4.lc/general-stuff/configuration/grafana-minimal-setup.git or by downloading a ZIP or tarball of the source.
Getting started with it is extremely easy, but first, letās go through each configuration file and what it does.
This one is fairly simple. This configures Prometheus, whose job is to aggregate the metrics the mod is exposing. In this particular configuration, we are telling it to scrape our headless at the address host.docker.internal:9000 every 15 seconds.
Note that in most cases, you would need to use localhost:9000 or container_name:9000; host.docker.internal is only used because the headless is not in a container and on the host machine.
Grafana, which: ; Will store all of its data in the directory ./grafana ; Will be accessible on the port 3000
Prometheus, which: ; Will store all of its data in the directory ./prometheus/data ; Have access to the configuration file mentioned earlier ; Pass the localhost of the machine inside (as well as passing some command-line configuration arguments defining the configuration file path and the storage path)
Now that this is out of the way, open a PowerShell window in the directory where the docker-compose.yml file is located and make sure Docker is launched. There, just write docker-compose up -d and watch Docker automatically pull and start the images. If you are curious, you can use docker compose logs -f to see all the logs in real time.
After waiting for a minute or two, you can visit http://localhost:3000 in a web browser to setup Grafana. The default login should be admin and admin for both user and password.
Once in Grafana, head to the menu on the left, go in āConnectionsā then āData sourcesā. There, select āAdd new data sourceā and select Prometheus. You will only need to set the URL of said source to http://prometheus:9090, you can then go on the bottom and click on āSave & Testā.
You can now either select to go to the explore view or create a brand new dashboard on the top right of the screen. I can recommend playing with the Explore view for a bit before starting to build dashboards, as it will teach you the different types of visualisation as well as some useful queries.
