How we speed up testing and building process of our Ember app at Brandnew? ~2.5x faster!

The scale

With that introduction you can imagine the scale of application we work on. Some more technical insights are:

  • project started 2 years ago
  • we pushed about 3 000 commits
  • let’s say 1.5 front-end developer was involved at the same time ;)
  • ~39k lines of application code (~46k with comments and empty lines) across ~1.3k files
  • 1 364 test cases (with ESLint)
  • 270+ components
  • 94 dependencies (that’s a bit scary)

Shipping fast is important

The ability to ship fast your application is in fact crucial. It should make you feel more comfortable with making changes (new feature, bug fix, refactor — you name it!).

  • Running Tests — 10:26
  • Making whole build — 12:04
  • With deployment — 18:20

So every minute we gain in building process can gain us about 2.5 minutes in the whole process.

Assumption worth to test.

Run tests in parallel

In our current CircleCI plan we can use 3 containers at the same time but this project uses only 1. Thanks to awesome ember-exam addon we can split the tests.

$ ember exam --split=NUMBER_OF_PARTITIONS --parallel
$ ember exam --split=3 --partition=1 --parallel
# circle.ymltest:
override:
- npm run-script test -- --split=$CIRCLE_NODE_TOTAL --partition=`expr $CIRCLE_NODE_INDEX + 1`:
parallel: true
  • Running Tests — 3:19 (~3.1x faster)
  • Making whole build — 5:30 (~2.2x faster)
  • With deployment — 11:10 (~1.6x faster)
Timings per container
  1. Make decision which test should be run next at runtime and put it into available container. As far as I understand, this is not possible with ember-exam .
  2. Keep historical execution times for all tests and try to decide how to split them to make partitions more balanced.

Store tests execution time

CircleCI do all the magic for us. We just need to feed it with the right data. That seems to be easy. We will change a bit our circle.yml file - set reporter to xunit and silent the output:

# circle.ymltest:
override:
- npm run-script test -- --split=$CIRCLE_NODE_TOTAL --partition=`expr $CIRCLE_NODE_INDEX + 1` --silent -r "xunit":
parallel: true
# testem.js/* eslint-env node */
let options = {
test_page: 'tests/index.html?hidepassed',
disable_watching: true,
launch_in_ci: [
'Chrome'
],
launch_in_dev: [
'Chrome',
'Firefox',
'Safari'
],
framework: 'qunit'
};

if (process.env.CIRCLE_TEST_REPORTS) {
options.report_file = process.env.CIRCLE_TEST_REPORTS + '/junit/test-results.xml'
}

module.exports = options;
Test Summary in CircleCI

Update your machine

Currently we run our tests on Ubuntu 12.04. CircleCI gives us an opportunity to use Ubuntu 14.04.

  • Running Tests — 2:50 (~3.7x faster)
  • Making whole build — 5:00 (~2.4x faster)
  • With deployment — 10:40 (~1.7x faster)

Update browser used for running tests

I know many people use PhantomJS and could argue why we use Chrome but that’s not the case, right? Let’s check what we can get by upgrading Chrome to the newest version in our builds (version 59 at the time of writing).

# circle.ymldependencies:
cache_directories:
- '~/downloads'
pre:
# download the latest Google Chrome if enabled by environmental variable $USE_LATEST_CHROME
- if [[ $USE_LATEST_CHROME == true ]]; then if test -f "$HOME/downloads/use_chrome_stable_version.sh"; then sh $HOME/downloads/use_chrome_stable_version.sh; else curl -o $HOME/downloads/use_chrome_stable_version.sh --create-dirs https://raw.githubusercontent.com/azachar/circleci-google-chrome/master/use_chrome_stable_version.sh && bash $HOME/downloads/use_chrome_stable_version.sh; fi; fi;
  • Running Tests — 2:40 (~3.9x faster)
  • Making whole build — 5:00 (~2.4x faster) — the same
  • With deployment — 10:40 (~1.7x faster) — the same

Let’s stop for a second

The results are pretty cool. The config has been already used for some time so let’s leave it for another sprint or two and end this post.

What’s next?

Things we would like to try:

  • Incremental builds — like the ones you use on your dev. machine (cached per branch) — could be beneficial for feature branches.
  • Yarn instead of Bower and NPM — all dependencies are cached so it takes only 40 seconds to set everything up. I do not expect much progress here if yarn is truly faster.
  • Reduce number of Acceptance Tests with faster Integration Tests if possible — this is kind of work in progress 🛠. Maybe separate article with comparison would be helpful?

--

--

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
Tomek Niezurawski

Tomek Niezurawski

30 Followers

I connect humans and machines. Usually write about interfaces, digital products, and UX on tomekdev.com. Founder of checknlearn.com. A bit nerdy 🤓