Setting up GitHub Runners on DigitalOcean

Setting up GitHub Runners on DigitalOcean – Using GitHub Actions is free for private repositories. It comes with a certain amount of free minutes and storage. An organization using GitHub Free has 2 000 minutes (per month) at its disposal. This is also related to the architecture on which the job is executed. 

Jobs that run on Windows and macOS runners that GitHub hosts consume minutes at 2 and 10 times the rate that jobs on Linux runners consume. For example, using 100 macOS minutes would consume 1 000 minutes included in the account, which can be described using a simple formula:

100 used minutes = 10 building minutes x 10

An option that emerges as the best solution is using a self-hosted GitHub runner. Self-hosted runners can be used with a repository at an organization or enterprise level. In this article, we will focus on self-hosted runners at the organizational level. 

Using self-hosted runners on an organizational level enables configuring processes for multiple repositories. It offers more control over hardware, operating system, software, etc. Still, it also bears some other responsibilities (previously handled by GitHub) like updating software and OS and security issues.

With self-hosted runners, there is no need for a clean instance for every job execution (like with GitHub-hosted runners).

Self-hosted runners open connections to GitHub to check if any jobs are queued for processing. This way, there is no need to allow GitHub to make inbound connections to self-hosted runners.

Setting up DigitalOcean Droplet

Digital Ocean droplet is a Linux-based virtual machine (VM) that runs on top of virtualized hardware.

Adding public key to DigitalOcean account

To add an SSH public key to the Digital Ocean account, log in to the control panel. In the “Account” section click “Settings”, then click the “Security” tab at the top of the page. In the “SSH keys” section, click “Add SSH Key”.

Copy the public key and paste it into the “SSH key content” field.

There is a field for the key’s name which will be used to identify the added key in the DigitalOcean control panel. Upon key creation, it will be displayed on the “Security” page and it will be automatically used in the droplet-created process, if the “SSH keys” option is selected for the “Authentication” method. Now, instead of using a set root password, a private key will be used to sign in to Droplet.

Creating a new Droplet using DigitalOcean web interface

Process of creating a DigitalOcean droplet is fairly simple using a control plane. Once logged in DigitalOcean, in the top right corner there is the “Create” drop-down which allows creation of different resources like droplets, databases, volumes, firewalls, load balancers and many more.

Once the Droplet is selected, a new page with all available options for the droplet’s creation will be triggered containing the following steps:

1. Select one of multiple Linux images like Ubuntu, CentOS, Fedora, etc.:

2. Select a cost plan according to your needs (for this article, we used a simple machine for demo purposes):

3. Select datacenter region (the recommended is to use the region closest to you). VPC Network will be automatically created according to the selected region:

4. Select the authentication method (we will use the SSH keys setup in previous step):

5. For final step, chose a hostname for your machine along with some additional settings:

Upon droplet creation, connection using SSH, as root user can be established (droplet_ip_address will be displayed in control panel, after droplet is created):

1 | $ ssh root@droplet_ip_address

Create a new user and grant administrative privileges

Using a system as a root user is not a good practice. The recommendation is creation of a regular user for daily use:

1 | $ adduser ag04

Command execution will trigger a set of questions, like account password, full name, work phone, etc. Most questions can be left at default values, except account password.

The next step is to set up sudo privileges for the newly created users, which will allow the user to execute administrative tasks as the root user. 

Add user ag04 to sudo group:

1 | $ usermod -aG sudo ag04

The above command will modify the default user settings, including the sudo group in the list of groups a user already belongs to. With append  (-a) argument, the user is added to the supplementary group. Append argument is to use only with groups (-G) option.

Setting up a basic firewall

UFW (Uncomplicated Firewall) is a firewall configuration tool that is specially designed to simplify the process of configuring firewall and it comes with Ubuntu servers. It is used to make sure that only connections to certain services are allowed on the server. It allows or blocks incoming and outgoing connections to and from the server.

OpenSSH, the service which allows the connection to the server, has a profile registered within UFW.

To get started, check all current available profiles:

1 | $ ufw app list

Output should be similar to the following:

1 | Available applications:
2 |   OpenSSH

Allow SSH connections:

1 | $ ufw allow openSSH

Enable the firewall:

1 | $ ufw enable

If all steps were successful, the output should look like:

1 | Firewall is active and enabled on system startup

Check if SSH connections are allowed:

1 | $ ufw status

Output will be similar to the following:

1 | Status: active
2 |
3 | To               Action     From
4 | --               ------     -----
5 | OpenSSH          ALLOW      Anywhere
6 | OpenSSH (v6)     ALLOW      Anywhere (v6)

Upon setting UFW, the firewall is blocking all connections except for SSH. If any additional services are installed, firewall settings should be adjusted to allow acceptable traffic in.

Enabling external access for regular user

After a regular user is created for daily use it must be configured so it can be used for SSH connection into the account directly.

To log in successfully, the local public key must be copied to the new user’s ~/.ssh/authorized_keys file. But since SSH keys were used to create the droplet, the public key is already added in the root account’s ~/.ssh/authorized_keys file. 

The next step is to copy the directory structure to the newly created user account in the existing session.

Multiple commands can be used to perform that action. However, we will use rsync. It is the simplest way to copy all the files and the correct ownership and permissions. Copy the root user’s .ssh directory, preserve the permissions, and modify the file owners:

1 | $ rsync --archive --chown=ag04:ag04 ~/.ssh /home/ag04

Check if everything is setup correctly run (preferably in other terminal session):

1 | $ ssh ag04@droplet_ip_address

Above command should establish a connection to a server without being prompted for the remote user’s SSH password for authentication.

Adding a GitHub self-hosted runner to an organization

Self-hosted runners, when added at organization level, can be used to process jobs for multiple repositories. Self-hosted runner can be created using following steps:

1. Under organization, go to “Settings”:

2. In the left sidebar, under “Actions”, select “Runners”: 

3. Click “New Runner”:

The above selection will trigger a new window with detailed instructions on how to install GitHub Actions application for multiple platforms. In this article, we used installation steps for Linux distribution.

Installation steps for GitHub Actions application

1. Create and move inside a new directory:

1 | $ mkdir actions-runner && cd actions-runner

2. Download the installation package:

1 | $ curl -o actions-runner-linux-x64-2.286.0.tar.gz -L
2 | https://github.com/actions/runner/releases/download/v2.286.0/
3 | actions-runner-linux-x64-2.286.0.tar.gz

3. Extract the downloaded package:

1 | $ tar xzf ./actions-runner-linux-x64-2.286.0.tar.gz

Extracted file contains:

1 | actions-runner-linux-x64-2.286.0.tar.gz bin config.sh env.sh
2 | externals run.sh

4. Create the runner using the configuration script:

1 | $ ./config.sh –url https://github.com/ag04 --token token_value

Running the configuration script will trigger “Self-hosted runner registration” section in which runner group, runner name, tags and working directory are configured. It will look similar to the following (for testing purpose all default values for runner creation are selected):

5. Start the runner using script:

1 | $ ./run.sh

If everything is setup correctly, runner will start listening for jobs:

1 | √ Connected to GitHub
2 |
3 | Current runner version: '2.286.0'
4 | 2022-01-17 13:41:05Z: Listening for Jobs

All above settings and runner’s status can be inspected in GitHub’s “Actions” section:

Configuring the application to run as a service

In order to simplify the application running, there is an option to configure the GitHub Actions application to run as a service so it’s automatically started when the machine starts.

In the compressed file that was extracted in the previous step, there is a script called svc.sh which can be used to install a managed application as a service.

The script has multiple options, which can be checked by starting the script without any argument (must be started with sudo):

Sudo

1 | $ sudo ./svc.sh

Generated output will display all available arguments:

1 | Usage:
2 | ./svc.sh [install, start, stop, status, uninstall]
3 | Commands:
4 |    install [user]: Install runner service as Root or specified user.
5 |    start: Manually start the runner service.
6 |    stop: Manually stop the runner service.
7 |    status: Display status of runner service.
8 |    uninstall: Uninstall runner service.

Install the self-hosted application as a service using install argument:

1 | $ sudo ./svc.sh install

Upon installation service will be inactive and it can be started using start argument:

1 | $ sudo ./svc.sh start

Service should be up and running now, which can be checked using status argument:

1 | $ sudo ./svc.sh status

The command output will contain information about the service’s state and logs:

1  | /etc/systemd/system/actions.runner.ag04.github-runner.service
2  | ● actions.runner.ag04.github-runner.service - GitHub Actions Runner 
3  | (ag04.github-runner)
4  |     Loaded: loaded (/etc/systemd/system/actions.runner.ag04.github-runner.service; 
5  |             enabled; vendor preset: enabled)
6  |     Active: active (running) since Mon 2022-01-17 13:49:57 UTC; 6s ago
7  |    Main PID: 2922 (runsvc.sh)
8  |       Tasks: 23 (limit: 1136)
9  |      Memory: 34.8M
10 |     CGroup: /system.slice/actions.runner.ag04.github-runner.service
11 |             ├─2922 /bin/bash /home/ag04/actions-runner/runsvc.sh
12 |             ├─2929 ./externals/node12/bin/node ./bin/RunnerService.js
13 |             └─2942 /home/ag04/actions-runner/bin/Runner.Listener run --startuptype    
14 |                    service
15 |
16 | Jan 17 13:49:57 github-runner systemd[1]: Started GitHub Actions Runner 
17 |                 (ag04.github-runner).
18 | Jan 17 13:49:57 github-runner runsvc.sh[2929]: Starting Runner listener with startup 19 |                 type: service
20 | Jan 17 13:49:57 github-runner runsvc.sh[2929]: Started running service
21 | Jan 17 13:50:00 github-runner runsvc.sh[2929]: √ Connected to GitHub
22 | Jan 17 13:50:00 github-runner runsvc.sh[2929]: Current runner version: '2.286.0'
23 | Jan 17 13:50:00 github-runner runsvc.sh[2929]: 2022-01-17 13:50:00Z: Listening for Job

To test if the runner is connected to GitHub and that everything is working correctly, minor code changes are required and that is where labels come in handy.

Labels are used to organize self-hosted runners based on their characteristics. There are two types of labels; default and custom labels. Default labels are created upon Actions application installation and they are:

• self-hosted: Applied to all self-hosted runners
• linux, windows or macOS: Applied depending on the operating system
• x64, ARM or ARM64: Applied depending on hardware architecture

On a project example, it would mean that GitHub Actions will trigger the build, based on build.yml file in .github/workflows directory.

Label update is done in the YAML workflow file (e.g. build.yml). Default value for runs-on parameter is ubuntu-latest, which needs to be replaced with self-hosted, so that job section look like:

1 | jobs:
2 |   validation:
3 |     runs-on: self-hosted

The GitHub repository contains at least one workflow, defined as a separate YAML file in the .github/workflows directory. Each workflow is triggered by one or more events, which can be some internal GitHub event like a push request, a scheduled event like a cron job, etc.

A workflow consists of one or more jobs, which are basically a set of commands that will be executed once the workflow is triggered. When workflow is triggered, all of its jobs run in parallel, by default, but they can also be configured to run sequentially if needed. Each job runs on a specific runner marked with a different label.

Using custom labels with self-hosted runners

Using a custom label to a runner allows us to configure a job to only execute on runners with that label. Unlike default labels which are fixed and cannot be removed or changed, custom labels can be added, assigned etc., but cannot be manually deleted (any unused labels will be automatically deleted within 24 hours).

Custom labels can be created with configuration script (part of GitHub Actions installation package) on self-hosted runner:

1 | $ sudo ./config.sh --labels test-runner

The label is created if it does not already exist.

Custom labels allow sending jobs to particular types of self-hosted runners, based on how they’re labeled. A self-hosted runner that matches all the assigned labels will then be eligible to run the job.

For example, multiple labels (both default and custom) can be combined for one job:

1 | jobs:
2 |   validation:
3 |     runs-on: [self-hosted, test-runner]

• self-hosted: Run this job on a self-hosted runner.
• test-runner: Custom label assigned to self-hosted runner with specific hardware configuration.

Install Docker Engine on runner machine

Docker is a platform that allows building, testing and deploying applications. Docker packages software into standardized units called containers. Containers allow applications to run in resource-isolated processes. They’re similar to virtual machines, but containers are more portable, resource-friendly, and dependent on the host operating system.

Example of GitHub Actions usage is to build and publish Docker images. In order to successfully execute that task, the machine on which runner is installed must have Docker Engine installed.

The Docker installation package is available on the official Ubuntu repository. But in this demo, we will install it from the official Docker repository.

A good practice when installing a new package on a system is to check for available updates and if necessary update the whole system.

Check if there are any new updates available:

1 | $ sudo apt update

Update all available packages:

1 | $ sudo apt upgrade

Shortcut to check and update all packages using combination of above mentioned commands:

1 | $ sudo apt update && sudo apt upgrade -y

Install a few prerequisites packages which lets apt use packages over HTTPS:

1 | $ sudo apt install apt-transport-https ca-certificates curl
2 |   software-properties-common

Add the GPG key for the official Docker repository to the system:

1 | $ curl -fsSL https://download.docker.com/linux/ubuntu/gpg | 
2 |   sudo apt-key add -

Add the Docker repository to APT sources:

1 | $ sudo add-apt-repository "deb [arch=amd64] 
2 |   https://download.docker.com/linux/ubuntu focal stable"

Install Docker Engine:

1 | $ sudo apt install docker-ce

Check if Docker is successfully installed:

1 | $ systemctl status docker

If installation process was successful, output will be similar to:

1  | ● docker.service - Docker Application Container Engine
2  |      Loaded: loaded (/lib/systemd/system/docker.service; enabled; 
3  |              vendor preset: enabled)
4  |      Active: active (running) since Thu 2022-01-27 10:51:55 UTC; 31s ago
5  | TriggeredBy: ● docker.socket
6  |        Docs: https://docs.docker.com
7  |    Main PID: 88078 (dockerd)
8  |       Tasks: 8
9  |      Memory: 29.9M
10 |      CGroup: /system.slice/docker.service
11 |              └─88078 /usr/bin/dockerd -H fd:// 
12 |                       --containerd=/run/containerd/containerd.sock
13 |
14 | Jan 27 10:51:54 github-runner dockerd[88078]: 
15 |                 time="2022-01-27T10:51:54.787623149Z" level=warning msg="Y>
16 | Jan 27 10:51:55 github-runner systemd[1]: Started Docker Application 
17 |                 Container Engine.
18 | Jan 27 10:51:55 github-runner dockerd[88078]: 19 |   
                     time="2022-01-27T10:51:55.654941680Z" level=info msg="API >

Above command output can be interpreted as that Docker service is successfully installed and enabled to start on boot (as machine starts).

Execute the Docker commands without sudo

By default, the docker command can only be run by the root user or by a user in the docker group, which is automatically created during Docker’s installation process. Any attempt to run the docker command without prefixing it with sudo or without being in the docker group, will result with following error:

1 | Got permission denied while trying to connect to the Docker 
2 | daemon socket at unix:///var/run/docker.sock: Get 
3 | "http://%2Fvar%2Frun%2Fdocker.sock/v1.24/containers/json": 
4 | dial unix /var/run/docker.sock: connect: permission denied

To avoid using sudo with every docker-related command, add username to the docker group (e.g. username set up at the beginning of the article):

1 | $ sudo usermod -aG docker ag04

To apply the new group membership, log out off the server and back in, or use the command:

1 | $ su - ag04

Check if username is added to the docker group:

1 | $ groups

The output of above command should look like:

1 | ag04 sudo docker

Another thing that needs to check for successfully executing a job on a newly created self-hosted runner is docker.sock permissions.

By default docker.sock file has the following permissions:

1 | srw-r---- 1 root docker 0 Jan 27 10:51 docker.sock

Default permissions will not allow successful job execution, but result in a similar error as for username:

1 | Got permission denied while trying to connect to the Docker  
2 | daemon socket at unix:///var/run/docker.sock: Post 
3 | "http://%2Fvar%2Frun%2Fdocker.sock/v1.24/build?buildargs=
4 | %7B%7D&cachefrom=%5B%5D&cgroupparent=&cpuperiod=0&cpuquota
5 | =0&cpusetcpus=&cpusetmems=&cpushares=0&dockerfile=Dockerfile&
6 | labels=%7B%7D&memory=0&memswap=0&networkmode=default&rm=1&
7 | shmsize=0&t=4746e9%3Ac00f274114c741e6ba192ef78629f7d1&target=
8 | &ulimits=null&version=1": dial unix /var/run/docker.sock: 
9 | connect: permission denied

A quick solution to this problem is to change the permissions of docker.sock file (add read and write permission for other users):

1 | $ sudo chmod 666 /var/run/docker.sock

The permissions, after the command is executed:

1 | srw-rw-rw- 1 root docker 0 Jan 27 10:51 docker.sock

Conclusion

Using self-hosted runners on an organizational level offers more control over hardware, operating system, software, etc. However, it also bears some other responsibilities like updating software and OS along with security issues. Self-hosted runners are flexible in terms of supported architectures (x64, ARM64, ARM32)  and operating systems, multiple Linux (RHEL, CentOS, Ubuntu, Fedora, Mint, etc.) and Windows distributions (Windows 7, 8, Windows Server, etc.) along with macOS distributions.

With self-hosted runners, there is no need for a clean instance for every job execution.

When configuring self-hosted runners, it’s important to keep in mind what type of project will be running. Based on that, select type of machine for GitHub Actions application. If it’s a simple Java project basic machine (e.g. 2 CPU/2GB RAM) will perform just fine. But if we are talking about complex NodeJS-based or Angular/React project, than a machine with better configuration will be required for job to perform smoothly.

Another important thing to keep in mind is that one self-hosted runner can only run one job at a time. When no available runners are idle, the subsequent jobs will be in queue until available runners are idle.

In this article, we demonstrated the basic installation and usage scenario. By that, it is safe to say that many fields can be updated in terms of security, optimization, auto-scaling, etc.

Check out – Email notifications and GitHub webhooks with Argo CD

References

• Self-hosted GitHub Runners 
• DigitalOcean How-Tos
• Uncomplicated Firewall
• Docker Installation