Testing multiple local Swift dependencies efficiently

Learn how to reduce the test time of a project that makes heavy use of multiple local Swift Packages to promote modularization.

For some time I struggled to reduce the test time when having multiple local dependencies, such as Development Pods or local Swift Packages. In a project with, say, 10 local dependencies, it can easily take 1 hour for your CI to run the unit tests for all of them, even if they only have 1 test inside each dependency.

For the purposes of this article, I considered a project with local Swift Package dependencies, and not other types of local dependencies (like static libraries and frameworks, or Development Pods).

Why does this happen?

What I figured is that xcodebuild's iOS Simulator startup time is too slow, and this causes the tests to take significantly longer. For each dependency you have, it will launch a new iOS Simulator process, and this process takes anywhere from 4 to 10 minutes, dependending on the machine it's run - even when launching a "headless" simulator.

The solution: Test Plans

Xcode Test Plans, which debuted in Xcode 11, serve the purpose of bundling tests together. You may want to create different sets of tests such as "Nightly", "UI", "Performance", etc, or you may simply bundle them up all together in the same Test Plan.

What I realized is that when using Test Plans, I obtained test runs up to 93% faster than by running individual Swift Package tests (xcodebuild test <swift package>) for iOS targets (more on this later).

Show me the code!

Setting up your Test Plan

Test Plans can be created via Xcode by creating a new file (cmd + N) and choosing "Test Plan" - yes, that simple. However, if you're using local Swift Packages, there's a trick here: your .xcodeproj or .xcworkspace must have the Swift Packages targets you want to test in its folder hierarchy otherwise they won't show up to be added in your Test Plan.

Once you have a Test Plan file, open it an click the + at the bottom left of the window and add all the targets you'd like to test.

Next, you may want to convert your one of your schemes of your main project (one that consumes those dependencies) to use Test Plans, or not. The only difference is that if you convert it, you can simply run xcodebuild test -scheme <your_scheme>, and if not, then you'll have to specify the name of your Test Plan explicitly, like xcodebuild test -testPlan <test_plan_name>.

To convert your scheme, open its Edit window on Xcode, and select Convert to use Test Plans…:

Collecting code coverage

Every setup is different, but I like using Rake tasks to run tests and collect code coverage. You may be using fastlane, in which case the APIs used will be different but the concepts remain the same. Also, I'll be using Code Climate in my example as it's the code coverage service I usually use for Swift projects.

First, we have to kick off the code coverage program that Code Climate distributes:

# Path to your Code Climate test reporter binary. You may want to fetch this from remote and do a checksum check before running it.
test_reporter_bin = "path/to/your/cc-test-reporter"

# Invoke the before-build function required by Code Climate, before collecting your code coverage
sh("#{test_reporter_bin} before-build")

Then we build the Test

# This becomes your new Derived Data path, stored in your project directory (you can gitignore this new folder)
build_path = ".build"

xcodebuild = []
xcodebuild << "arch -x86_64" # Only required if your project only runs on Rosetta 2
xcodebuild << "xcodebuild test"
xcodebuild << "-workspace your-project.xcworkspace"
xcodebuild << "-scheme 'Your Scheme'"
xcodebuild << "-destination 'platform=iOS Simulator,name=iPhone 14 Pro'"
xcodebuild << "-enableCodeCoverage=YES"
xcodebuild << "-derivedDataPath '#{build_path}/your-project'"
xcodebuild << "-clonedSourcePackagesDirPath '#{build_path}/SourcePackages'"
xcodebuild << "-quiet" # Optionally silents all xcodebuild output
sh(xcodebuild.join(" "))

Since xcodebuild will collect code coverage for all the targets that it goes through, including 3rd party, we have to filter out which targets we're interested. Let's declare an array for this purpose, that will be used later:

# You may want to collect this array programmatically somehow via scripting, for instance by crawling your directory hierarchy and reading which packages have test targets. For simplicity sake here we're just declaring them statically. 
targets_to_keep = %w(
  foo
  bar
  baz
)

Then we must interpret the .xcresult file produced by xcodebuild, and convert it to a format that Code Climate can interpret:

# This is a special path where your xcresult (test results) will be stored, within the Derived Data path specified by you earlier.
most_recent_xcresult = Dir.glob("#{build_path}/your-project/Logs/Test/*.xcresult").max_by { |f| File.mtime(f) }

# Specify where we're going to store the code coverage reports. Feel free to gitignore these directories or clean them up after collection
xcodebuild_reports_directory = "coverage/xcodebuild_only_reports"
xccov_report = File.absolute_path("#{xcodebuild_reports_directory}/your-project-xccov-report.json")
output_file = File.expand_path("coverage/your-project-codeclimate.json")
FileUtils.mkdir_p(xcodebuild_reports_directory)

# Use `xccov` (a built-in tool from Xcode) to convert the .xcresult into a json file that uses the xccov format
sh("xcrun xccov view --report --json '#{most_recent_xcresult}' > #{xccov_report}")

# Invoke this function that will clean up the code coverage report from targets we're not interested in collecting data for
remove_all_targets_except(xccov_report, targets_to_keep)

# Finally, format the json report (that uses the xccov format) into another json format specific to Code Climate 
sh("#{test_reporter_bin} format-coverage '#{xccov_report}' --input-type xccov --output '#{output_file}'")

Note that this snippet makes use of a function you must declare in your Rakefile called remove_all_targets_except. More about it here: https://gist.github.com/rogerluan/aaba6694a45ed5381e7e6b2259abd9f7

Finally, submit the code coverage report to Code Climate:

sh("#{test_reporter_bin} upload-coverage --input '#{output_file}'")

Monorepo setup

If you repository has multiple projects in it, this means you'll probably have more than 1 TestPlans (one for each project). If that's your case, this means you'll still have to merge the code coverage reports of your multiple projects before uploading them to Code Climate, by using the sum-coverage --parts <number_of_parts> function of their code coverage reporter tool. Again, this is only for Code Climate, so your mileage may vary.

Conclusion

With the setup I described in this article, I was able to reduce build times from ~61mins to ~4mins (when running locally), and from ~90mins to ~20mins (when running on Jenkins). That's an improvement of 77-93%!

One caveat with Swift Packages specifically is that, if your package is capable of running on macOS target (e.g. has no iOS dependency like UIKit), then your fastest alternative will certainly be using swift test, because it usually takes under 1 second to run whatever test suite you may have (of course, as long as they're not time-dependent). Yes, really.

I hope this setup also helps you and your team to reduce build times significantly. And if you know other tricks to reduce test time, drop a note on my twitter @rogerluan_!