FrooxEngine, Resonite & CI/CD

Today, we’re going to have a look at the CI/CD pipelines for Resonite, and admittedly, I’m at my third attempt at writing this blog post.

I started writing this post back when work started on the pipelines, trying to keep it in sync. However, while I was adding more features, and patching others, I didn’t have the time to update the draft which ended up completely desynchronized from the pipelines themselves.

The release of the Unity renderer as an Open-Source project also changed plans quite a bit, as the original post was written while the project was still internal.

Now that I finally found some time, and energy, to write it, let’s do a quick dive into Resonite and the pipelines powering it.

The anatomy of a pipeline

This paragraph is for those who have no idea what CI/CD is. To keep it short, CI/CD is the practice of automating testing and builds when a change is made (or pushed) to a project’s code. Those workflows can even be used to publish a new version of the software, which is what is done for Resonite.

Pipelines themselves work in multiple layers:

• Pipeline: a collection of individual workflows. Think of it as the collection of stuff to do.
• Workflow: a collection of jobs; a way to concentrate a bunch of related tasks together.
• Job: a specific collection of actions made to achieve a specific goal, for instance, testing Resonite.
• Step: a single action within a job, for instance, running a command to install a software.

On each level, you can define conditions for execution, input & output values, and tons of other parameters that make those kinds of pipelines very powerful tools.

In the case of Resonite, some of those workflows are launched manually, as to avoid pushing a new version accidentally or spending too much money on the pipelines themselves.

All those steps are executed on a remote machine called a “runner” in this context.

Cost for CI/CD is calculated in “minutes” which is equal of time spent by a runner to run its workflow. For instance, if you have a single workflow that runs for 15 minutes, you will consume 15 CI minutes. If you have two workflows running in parallel that each consume 5 CI minutes, your total will be 10 CI minutes.

Lastly, the output of those workflows are “artifacts” which can basically be any kind of file.

External libraries

Before tackling anything else, there was already an aspect already done: the native libraries. Resonite is a very complex piece of software, relying on tons of external software libraries.

Luckily, most of those already had CI/CD when I joined, but some things could still be done. Through Composite Actions and Workflows, the pipelines used for those repositories were unified as to ease maintenance (I want to avoid as much as possible having to go through everything to push a single update).

The only major update that happened since was to add steps to compile libraries under ARM, which was needed for the Headless Server Software on ARM.

The renderer

As mentioned in the intro, this part has evolved quite a bit since I originally made the pipelines for it. The renderer for Resonite is Unity, and an outdated version at that. Building it sucks. It has since been released as an Open-Source project, which saves us money on compute, as CI/CD in Open-Source repositories is free.

To build it, you first have to install the whole Unity Editor, which weighs several gigabytes, even with limited options, then activate it using a licence. That licence is probably the most annoying aspect when building automatically or testing; for now, it’s just using a dummy account created for that purpose.

The only real optimization possible on that side is caching the build, even though GitHub remains incredibly slow in downloading and decompressing it.

      - name: 'Install Unity'
        uses: buildalon/unity-setup@ab626823a2d32033d5a7811169d0ce92e8010d38 # v2.3.0
        id: unity-setup
        with:
          unity-version: '2019.4.19f1'
          modules: |
            windows-mono
          cache-installation: 'true'

In the end, building the Unity renderer takes around 15 minutes for a cached build. Luckily, we don’t have to pay for any of this.

The FrooxEngine project

FrooxEngine itself is pretty large: a .NET solution containing 41 projects in total, ranging from the engine itself to the cloud library and other utilities.

The codebase itself is not that difficult to build, but requires some steps: clone it, run the build tasks on first run, build the whole repository, and you should be greeted with a copy of Resonite. The last thing to do is to drop the renderer into the folder.

To build FrooxEngine, you also need to have a few things in the same directory of the repository: locales and ProtoFlux. If you ever wondered how the locale stats are updated, this is done on build.

As a side note about the ProtoFlux repository, I was working on migrating it to an internal NuGet package, but facing issues and time constrains, this work has been postponed and is still ongoing.

The final process looks like so:

• Fetch all repositories needed
• Run the build tasks
• Build FrooxEngine itself and the Headless Server Software
• Assemble and package Resonite and the Headless Server Software
  • Download the renderer from the repository
  • Place it in the right place
  • Compress everything with ZStandard
• Upload artifacts to a S3 bucket

Now, if we want to publish a release, there are a few more steps involved:

• Create a new GitHub release and upload the compressed artifacts there
• Publish the client and Headless Server Software to Steam

And that’s about it, you will now get a fresh version of Resonite published everywhere needed.

Versioning

As you may have noticed, Resonite uses a date time-based versioning system, which is based upon when the engine was built.

Because of this, the workflows will read the versioning file and take this as the authoritative source to create tags, versions, etc on its own.

          RESONITE_VERSION=$(cat Windows/Build.version)
          echo "resonite-version=$RESONITE_VERSION" >> $GITHUB_OUTPUT

Troubles

As of now, we are using GitHub and GitHub Actions to host Resonite’s code and run its CI/CD. To avoid unnecessary questions: yes, we know there are huge issues with GitHub, yes we considered migrating somewhere else, no we can’t currently due to costs and time constrains, yes we’re still keeping it as an option.

If you’ve read any of my GitHub Actions-related blog posts, you probably know I’m far from being a fan of this platform. Its implementation is hazardous at best, has huge security issues, and is, in general, awfully managed.

The standard ubuntu-latest runner is supposed to have 72Gib of storage, which in theory is more than enough to build the FrooxEngine project. Enters GitHub’s wisdom to ship everything in the default image, bringing the tangible available storage down to 17Gib.

But even with this storage, in theory, a build of FrooxEngine is only a few gigs at most, but on the runner, everything would fail because of storage constrains regardless.

Why? Well, did I mention that working with Unity sucked? Turns out that there is no real light way to reference Unity, making us need to download the whole thing on the runner, using storage that could be spared otherwise. Amazing, isn’t it?

Luckily for me, someone made a really useful Action that cleans up the useless stuff from the runner, bringing the available storage to around 42Gib.

      - name: 'Free Disk Space'
        uses: jlumbroso/free-disk-space@54081f138730dfa15788a46383842cd2f914a1be # v1.3.1
        with:
          android: true
          haskell: true
          large-packages: true
          docker-images: true

The script above is used to remove a few things:

• The Android development kit (generally used to make phone apps)
• The Haskell runtime (used to interact with the programming language of the same name)
• Docker images (environments for a pretty popular development tool)
• Large packages (unused programming languages, databases and software)

Overall, the step takes a few minutes that could be spared if the default image was optimized correctly, but of course, we can’t have nice things.

The costs

Now that we have a working pipeline, how much does it actually costs to run all of this? Luckily for us: absolutely nothing.

On our current GitHub plan, we get 3000 CI minutes a month for free and 2Gib of artifact storage for private repositories. The full release workflow taking from 17 to 40 minutes (the latter if the cache randomly decides to purge Unity), we could publish between 75 and 176 versions of Resonite a month, which is in my opinion more than enough.

The only real issue in those limits was the artifact storage. A fully built version of Resonite is around 600Mib, plus the headless alone being around 400Mib. Now you can see how just a few versions can fill up the 2Gib quota quite rapidly.

The solution: bypass the middleman and manage storage ourselves. All artifacts are now uploaded to a S3-compatible bucket, which saves us quite a lot of trouble. Not only S3 storage is incredibly cheap, but we now have a super reliable build history that we can just browse on our own if the need ever arises.

Overall, our pipelines costs close to nothing, which I’m satisfied with.

The funny issues

Quickly after the first release done with the new pipelines, probably the funniest bug I ever caused happened: Resonite was now a Linux exclusive platform.

How did this happen? Given Resonite is built using Linux runners, I forgot to set a parameter forcing to generate Windows executable files when building the client and utilities. Naturally, since I use a Fedora Linux workstation to work on Resonite, I never noticed that issue as it was launching perfectly fine on my end.

    <DefaultAppHostRuntimeIdentifier>win-x64</DefaultAppHostRuntimeIdentifier>

While I quickly patched it, I still maintain that this was the right move and that nobody understands my genius yet.

Word of the end

While the new pipelines have been a tremendous help to publish Resonite, I can’t help but be frustrated at some aspects that could be handled better, but that just can’t, the first culprit being of course Unity.

I predict that the renderer switch away from Unity will tremendously reduce the CI minutes used in all builds, making our process even cheaper.

This will also allow for more platforms to be targeted like native Linux, macOS and ARM builds.

The security aspect of GitHub Actions also scares me quite a lot, as it was built without this consideration at all. For instance, when referencing an Action tag v4, the runner will download on its own the latest v4.X tag, meaning that if a threat actor pushes a new release, it will be downloaded without second thoughts or approvals.

To avoid that, you need to pin Actions to a specific Git commit SHA, but then makes is a hassle to update large workflows.

In the end, this work is still a good step towards automating more parts of the production of Resonite, so our team can focus on engineering rather than menial tasks that can easily go wrong.

If you’ve made it this far in the devlog, thanks for reading it whole. This is now a more personal note that I want to share.

You may have noticed that I am not present in the Resonite Discord any more, but worry not, I am still actively part of the team.

I ended up leaving for multiple reasons, among them being that I abhor Discord as a platform, but also because of some recurring behaviours I noticed.

On multiple occasions, I have seen people pile up on new users bringing valid criticism of the platform. This is not a culture I wish to see on the Discord; you are welcome and even encouraged to criticize Resonite. If you think we did something wrong, please let us know, the only requirement is to stay polite and constructive.

While I know only a very little minority engages in this behaviour, it still reflects on the larger community and platform, which is something that demotivates me; hence why I left this place.

While I’m not a moderator nor your mother, if you have engaged in this type of behaviour, jumping on criticism to “defend” Resonite before, I respectfully ask: please don’t.

I know bad faith actors sometimes join, but assuming malice in every criticism is not the way to go. If you think someone is acting maliciously, use the moderation portal to report the user instead of launching a witch hunt, those last for a long time, tend to end up in a circle, and suck for everybody involved.

And to new users, the same applies to you, if you notice someone trying to instigate harassment for bringing up a concern or feedback, make use of the moderation portal. While not all reports lead to actions, those are logged and are used if it becomes a recurring behaviour.

Thanks for listening to my ramblings, I’ll hopefully see you in the next post. Stay purple.