✍️ Step-by-Step Jenkins Setup for iOS in 2023 | Ultimate Guide

I spent over 3 weeks setting it up the first time. Don't make the same mistakes I did!

It’s 2023 and you and your team decided to move away from the current CI provider you have, towards a self-hosted Jenkins CI. Great! In this article we won't cover the pros/cons of using one CI system against another, but rather be laser focused in showing how to set up a fully working Jenkins CI environment for iOS.

Although you can probably get an environment up and running in a couple hours by discovering Jenkins on your own, and even be able to run an iOS build on it, there are lots of small issues that compound over time and make such naive implementation quite unsustainable.

I did a lot of experimenting, and struggled for a few weeks to get everything right, specially after getting multiple cryptic errors alongside issues that would happen “every once in awhile”. After surprisingly not finding a comprehensive guide on “Jenkins best practices”, I decided to put this one together to share knowledge I learned the hard way. 🥲

Preconditions

I won’t be covering the steps needed to get yourself a macOS machine. This article assumes you have a bare metal machine running macOS (yes, macOS is required), and that you have already installed Jenkins in your machine. If you haven’t installed Jenkins yet, here’s the official guide (this step is straightforward): https://www.jenkins.io/doc/book/installing/macos

Furthermore, in this article we’re going to be using Homebrew, rbenv, xcodes, and Bundler. I won’t go into details as to why I recommend each one of these (maybe in a future post 😉), but feel free to reach out to me if you’re curious!

Installing dependencies

First and foremost, we need to install Homebrew. You should follow the instructions in this link — it’s quite straightforward.

Updating your ~/.zshrc file

These settings should get you up and running more easily, and features a security measure for fastlane. Append this to your ~/.zshrc file:

# Initialize rbenv if it's already installed
export PATH=$PATH:/usr/local/bin:$HOME/.rbenv/bin:$HOME/.rbenv/shims
if which rbenv > /dev/null; then
  eval "$(rbenv init -)"
fi

# Set these according to your project's needs/configuration
export XCODE_VERSION="14.3"
export BUNDLER_VERSION="2.2.32"
export RUBY_VERSION="3.1.2"

# For security measures, ask fastlane to not store your fastlane password.
# Even though your CI workflows and scripts probably doesn't even use App Store Connect User+Password anymore, if you happen to run fastlane manually in the CI machine, you don't want it to accidentally store your own Apple ID password in its Keychain.
export FASTLANE_DONT_STORE_PASSWORD="1"

# Required by fastlane (to prevent issues with unicode)
export LC_ALL=en_US.UTF-8
export LANG=en_US.UTF-8
export LANGUAGE=en_US.UTF-8

Note that depending on which cloud provider you’re using (e.g. AWS, Azure, etc), your existing ~/.zshrc file might already have some content in it, thus, simply append the snippet above at the end of the file, to avoid issues.

After updating your ~/.zshrc file, source it to apply the changes (or kill your SSH session and start a new one):

source ~/.zshrc

Then, you can copy and paste this snippet in your Terminal to get your dependencies set up more easily:

echo "Installing rbenv and the right ruby version that your project uses"
brew install rbenv ruby-build
rbenv install $RUBY_VERSION
rbenv global $RUBY_VERSION

echo "Speeding up gem installs"
echo "gem: --no-document" >> ~/.gemrc

echo "Initializing rbenv (will run the initialization code that we just saved in the ~/.zshrc file)"
source ~/.zshrc

echo "Installing bundler"
gem install rubygems-update
gem update --system
gem install bundler -v $BUNDLER_VERSION

echo "Installing xcodes"
brew install xcodesorg/made/xcodes

echo "Installing the Xcode version your team uses"
echo "Note that this one is gonna take a long while (maybe 10-20 minutes). Take a break, and once it's, done you're gonna need to enter the sudo password so the installation completes"
brew install aria2
xcodes install $XCODE_VERSION --experimental-unxip --select --update

Configuring your git credentials

For this step, I wrote a dedicated article. Pause reading this article and follow the instructions on this other article to get your git credentials configured in your Jenkins machine:

Authenticating with GitHub on Jenkins CI using a GitHub App

Configuring Jobs

There is a race condition in how Jenkins picks up which branches to build when using different strategies to “discover” branches (i.e. “Exclude branches that are also filed as PRs” VS “Only branches that are also filed as PRs”). This causes problems like PRs being stuck in a forever “pending” state, thus blocking them from being merged. For this reason, we are going to need 2 Job pipelines:

• One will build all branches that aren’t intended to be filed as PRs, e.g. main, master, develop, staging, production (depends on how you call them).

• Another to build all the other branches, that may become PRs.

Took me a really long time to figure this out, as about 5-10% of the times the PRs would get stuck and become unable to get merged because they fell in some weird state. This was the only fully working solution I found to this problem.

Configuring the Job that will build all branches that won’t become pull requests

Go to https://<your_jenkins_domain.com>/view/all/newJob to create a new job. Choose Multibranch Pipeline from the options:

I like appending the project type (in this case, Multibranch Pipeline) to the pipeline name, so I know which structure it uses, just by the name. Thus, in this case I’d name it something like protected-branch-multibranch-pipeline 😊

When setting things up for this Job, these are the key settings you need to follow:

• GitHub Credentials: select the GitHub App Credentials you added in the “Configuring your git credentials” section above.

• Add a new behavior “Discover branches” and select “All branches”.

  • Add a “Filter by name (with regular expression)” filter under this section, with the text “master” or “(master|staging|production)”

• Periodically if not otherwise run: ☑️

  • Interval: 1 minute.

  • Honestly this one comes from a time where Jenkins wasn’t always picking up the builds it needed to run. But it was in the very beginning of the process, before I even use GitHub App, so I’m not sure whether this one can be turned off. Possibly.

All the remaining settings not mentioned above are up to you to decide how to fill (e.g. are specific to your project, or tastes).

Before continuing, a quick announcement!

I’m excited to launch my 1st #indiedev SaaS! Solving a pain point I’ve had for years: notifying your team when apps are ready for testing or passed review 👀

Customers are loving it! You can check it out here: https://statused.com

Configuring the Job that will build pull requests

Visit https://<your_jenkins_domain.com>/view/all/newJob again to create a new job, and choose Multibranch Pipeline from the options. Again, I’ll recommend naming it something like pull-requests-multibranch-pipeline, so you can easily identify it in the future.

Follow the same steps as above, except the behavior you’re going to add is “Discover pull requests from origin”, selecting the strategy as “The current pull request revision“:

Configuring your Jenkinsfile

You didn’t think I’d skip the most critical part, did you?

In the steps above, when configuring a new Job, you had to select the path of your Jenkinsfile. The way of setting up Jenkins explained in this article has to be matched with a particular way to configure your Jenkinsfile and your CI scripts, so I’ll cover them below.

Here’s a template you can use for the main Jenkinsfile, which can be used to run both protected-branch-multibranch-pipeline and pull-requests-multibranch-pipeline:

pipeline {
    agent any
    options {
        ansiColor('xterm') // Adds color to logs, enable via https://github.com/jenkinsci/ansicolor-plugin
        timeout(time: 8, unit: 'HOURS') // Set the timeout limit for builds
        disableConcurrentBuilds(abortPrevious: true) // Cancel the previous build upon pushing newer commits in the same branch
    }
    environment {
        // Set up all your secrets (aka credentials) here, e.g. API keys for fastlane, danger, etc.
        APP_STORE_CONNECT_API_KEY_ISSUER_ID = credentials('APP_STORE_CONNECT_API_KEY_ISSUER_ID')
        APP_STORE_CONNECT_API_KEY_KEY = credentials('APP_STORE_CONNECT_API_KEY_KEY')
        APP_STORE_CONNECT_API_KEY_KEY_ID = credentials('APP_STORE_CONNECT_API_KEY_KEY_ID')
        DANGER_GITHUB_API_TOKEN = credentials('DANGER_GITHUB_API_TOKEN')
        MATCH_PASSWORD = credentials('MATCH_PASSWORD')
        // …etc
    }
    stages {
        stage("1. Set Up") {
            steps {
                withCredentials([usernamePassword(credentialsId: '<team_name>_github_app', usernameVariable: 'GITHUB_APP_USERNAME_TOKEN', passwordVariable: 'GITHUB_APP_PASSWORD_TOKEN')]) {
                    sh '''
                    source ~/.zshrc # Needs to be run to set up the right PATH env var, initialize rbenv, and everything else we configured previously in the ~/.zshrc file

                    # Refs.: https://git-scm.com/docs/gitcredentials#_custom_helpers and https://stackoverflow.com/q/61146986/4075379
                    git config credential.username ${GITHUB_APP_USERNAME_TOKEN}
                    git config credential.helper "!echo password=${GITHUB_APP_PASSWORD_TOKEN}; echo"

                    # From now on you can add your scripts here, e.g. make, bundle install, pod install, xcodebuild build and test, etc.
                    make
                    '''
                }
            }
        }
        stage("2. Static Code Analysis") {
            steps {
                sh '''
                source ~/.zshrc # Yes, unfortunately you need to run this every time you declare a new "sh" shell script in your Jenkinsfile.
                bundle exec rake danger
                '''
            }
        }
        stage("3. Build & Distribute") {
            steps {
                withCredentials([usernamePassword(credentialsId: '<team_name>_github_app', usernameVariable: 'GITHUB_APP_USERNAME_TOKEN', passwordVariable: 'GITHUB_APP_PASSWORD_TOKEN')]) {
                    sh '''
                    source ~/.zshrc
                    bundle exec fastlane archive_and_distribute # This action needs access to GITHUB_APP_USERNAME_TOKEN and GITHUB_APP_PASSWORD_TOKEN env vars directly
                    '''
                }
            }
        }
    }
    post {
        // Always clean your workspace after you finish using it, otherwise after a few builds you will end up with a full disk and your machine will likely die.
        always {
            cleanWs(cleanWhenAborted: true, cleanWhenFailure: true, cleanWhenNotBuilt: true, cleanWhenSuccess: true, cleanWhenUnstable: true, deleteDirs: true)
        }
    }
}

Explaining the “withCredentials” function

The way the “withCredentials” function works is as follows:

// 1st parameter is the name of your GitHub App credentials ID, as you registered in Jenkins.
// 2nd and 3rd parameters are variable names you're declaring now, so you can name them whatever, but you will need to reference them later, so make them relatable.
withCredentials([usernamePassword(
    credentialsId: '<team_name>_github_app',
    usernameVariable: 'GITHUB_APP_USERNAME_TOKEN',
    passwordVariable: 'GITHUB_APP_PASSWORD_TOKEN'
)])

The input is the Credentials ID that you registered earlier in your Jenkins credentials. That gives access to this Jenkins plugin to be able to generate ephemeral tokens (username and password) that can then be used to make HTTPS requests to GitHub. It’s critically important for you to understand that this is a HTTPS auth, and not SSH — so if you have git commands relying on SSH down the pipeline, you should identify that you’re running under a CI environment and switch them to use HTTPS instead. A common example for this, in an iOS environment, is the Git URL used by fastlane match.

The first time we’re generating such ephemeral credentials, we’re exposing them to git config credential.helper, so that any git operation (that uses HTTPS) after that moment (even in other Stages) will be able to run without asking for username and password again. In the example above, you can observe this on Stage 2, where we’re invoking danger (which posts comments to GitHub, so it needs credentials), and we didn’t need to generate new credentials for it. Different from the Stage 3, where we need access to the credentials env vars directly, and the env vars don’t persist from one Stage to another, so we re-generate the credentials, thus, setting values to the env vars again.

If you’re wondering, here’s how you could configure the fastlane match action in your Fastfile:

git_url = is_ci? ? "https://#{ENV["GITHUB_APP_USERNAME_TOKEN"]}:#{ENV["GITHUB_APP_PASSWORD_TOKEN"]}@github.com/myorg/myrepo.git" : "git@github.com:myorg/myrepo.git"

match(git_url: gir_url)

Note 1: you must have the GitHub Branch Source plugin on version 2.7.1 or above to use these APIs. This feature was introduced and announced in 2020.

Note 2: as the announcement states, the API token you get will only be valid for one hour. So don’t get it at the start of the pipeline and assume it will be valid all the way through.

Final touches

There are two remaining things to do before you call it a day for your Jenkins setup journey.

Builds are getting stuck forever

This might happen when your pipeline tries to access git within the pipeline (e.g. when running pod install, or fetching SPMs, etc). It could be caused because your machine is asking for the Keychain password authentication, but that is not surfaced on the Jenkins logs. It’s unfortunate, but the only solution I found to be fully working in this case is to log on the machine with access to the user interface (e.g. VNC, not SSH), and clicking in “Always Allow" in the pop up that shows up asking for permission to access “login.keychain”. Enter the root password of the macOS machine when prompted.

You only need to do this once, then never again.

Xcode git credentials clash with GitHub App’s credentials

There’s an issue that causes builds to fail with error:

stderr: remote: Invalid username or password

This would happen in about 5-10% of the builds, breaking them. Turns out that Xcode’s git credentials may clash with the ones we’re setting and then it breaks. To fix this, simply follow my answer on this Stack Overflow question:

Jenkins Github Authentication fails sometimes?

You will need to repeat those steps every time a new Xcode version is installed.

Conclusion

In this (admittedly) long guide, you learned a highly opinionated way to set up your Jenkins machine, tailored for an iOS environment. If you’re working with other environments, the majority of the tips shown in this article will help you get on the right track as well, such as for Android, React, Flutter, Node, etc. What will change is probably just your dependencies, and the specific examples I gave when setting up your Jenkinsfile.

When I initially set up my first Jenkins machine, I got my first build in a matter of hours, but it was far from an environment that would fit the team’s needs. GitHub Checks are definitely a must have for example (something that is not straightforward to set up), and the environment should be 100% stable, with no flakiness, otherwise the CI system won’t be trusted by your team. In other words, I got 80% of the work done very quickly, but the remaining 20% to get a polished CI pipeline took me literally weeks. I hope that by following this guide you won’t have to worry about the environment, and will be able to focus on what matters: building the perfect pipeline for your project, and your team 🤗

In my next blog post, I’ll cover what you can do to preserve the hard work you did to set up your Jenkins machine. To put it simply: how to save a backup of your CI! Stay tuned.