Ephemeral Servers (pop-up servers)

Hello teem. I am the only one who knows how these pop-up servers REALLY works in our group. The pop up servers are battle-tested over years now at this point with Jamulus and Barotrauma being the tip of the spear (:hot_face:). The popups are made from a protocol or framework that I wanted to describe in code - with barotrauma-popup repo being the gold star project for this.

The protocol is as follows: Create a server in AWS with Terraform, and allow Ansible to SSH onto the server and install docker on it, then make all files and folders required for persistence or configuration, including a docker-compose file which contains the docker container image name to run, often we will use 3rd party docker containers for the game server code. The last thing it does is runs the container and then sends a notification to a AWS Lambda function which determines where it should send a notification (telegram/discord).

To make things as automated as possible we use Gitlab pipelines to run this process. Since the pipelines also run in a docker environment we use another docker image called Vereto Build Image. the VBI contains everything the pipeline runner needs to run the code, like ansible, terraform and such. The VBI refreshes itself daily automatically.

On the server side we need an operating system to run all this shit. OS’s change all the time so to make sure we’re not trying to hit a moving target every time we run the pipeline to create the server we created the Vereto Launch Platform which is an AMI (Amazon Machine Image) which is basically a Linux ISO you can give to AWS. We then run all pop ups based on this AMI. The VLP is a slightly modified version of Ubuntu 22 and from time to time we need to go in and update the VLP versions manually. Like when Ubuntu 20 reached the end of support. Locking to our own custom version of ubuntu means we KNOW for a fact which software is available and that nothing has been really moved around on the OS level.

The last step in the protocol is clean-up. We have the ability to run pre-destroy tasks like grabbing the latest save from the game server and saving it to AWS s3. Once thats completed we completely remove everything to do with the server from AWS.

This is to exploit how AWS pricing works. AWS servers are charged by the minute and the fewer minutes you run for - the cheaper the service! AWS is really fast at making new servers and so this means we can use very powerful servers, since we only use them for a few hours.

We’re using the same process for making an Arma Reforger for events. Currently it fucking sucks though because some software really hates to be run like this.

Ok now you know now.

The Vereto AWS Notification Service (VANS) is a lambda function which posts notifications to Telegram and Discord. The lambda trigger is a custom CloudWatch Notification that I made which listens to all EC2* events in an AWS account plus a custom event. The default events you get from AWS are the current state of the server - being created, setting up, started, stopping and destroyed. I add another event here called ‘setup-completed’ which is triggered by the Ansible code in the popups. Only when this setup-completed event occurs does it actually post about the IP information and the stuff you’ve seen already in Telegram… If its a service it knows about. There is a list in the repo of all services, with some nice little messages and the picture that appears in telegram. If its some random server it doesnt know about then it posts to another channel and doesnt wait for ‘setup-completed’. A lot of the code in this project is to make the output nicer, like saving the message ID of the first message it sends so it can edit in-place. It saves all message information into DynamoDB. I tried to make this with defensive programming so that you dont really need to modify the code just the configuration file.

*Ec2 is AWS server creation service (AWS does a lot more than just make servers)