How to Keep Your Automated Testing Tools Fast
by Martin Führlinger, Backend Engineer
In the backend team we usually try to automate things. Therefore we have tons of tests to verify correctness of our code in our gems and services. Automated tests are executed much faster and with much higher coverage than any tester can do manually in a similar time. Over the years a lot of functionality has been added, and as a result, a lot of tests have been added. This led to our test suites becoming slower over time. For example, we have a service where around 5000 tests take about 8 minutes. Another service takes about 15 minutes for around 3000 tests. So why is service A so fast and service B so slow?
In this blog post I will show some bad examples of how to write a test, and how to improve automated testing tools to make tests faster. The Runtastic backend team usually uses `rspec` in combination with `factory_bot` and jruby.
The Test File Example
The following code shows a small part of a real example of a test file we had in our test suite. It creates some users and tries to find them with the UserSearch use case.
describe Users::UseCase::UserSearch::ByAnyEmail do
describe "#run!" do
subject { Users::UseCase::UserSearch::ByAnyEmail.new(search_criteria).run! }
let(:current_user_id) { nil }
let(:search_criteria) { double(query: query, size: size, number: number, current_user_id: current_user_id) }
let(:default_photo_url) { "#{Rails.configuration.services.runtastic_web.public_route}/assets/user/default_avatar_male.jpg" }
def expected_search_result_for(user)
UserSearchResultWrapper.new(user.attributes.merge("avatar_url" => default_photo_url))
end
shared_examples "find users by email" do
it { expect(subject.class).to eq UserSearchResult }
it { expect(subject.users).to return_searched_users expected_users }
it { expect(subject.more_data_available).to eq more_data? }
end
let!(:s_m_user) { FactoryBot.create :user, email: "s.m@mail.com" }
let(:runner_gmail_user) { FactoryBot.create :user, google_email: "runner@gmail.at" }
let!(:su_12_user) { FactoryBot.create :user, email: "su+12@gmx.at" }
let(:su_12_google_user) { FactoryBot.create :user, google_email: "su+12@gmx.at" }
let!(:user_same_mail) do
FactoryBot.create :user, email: "user@rt.com", google_email: "user@rt.com”
end
let!(:combined_user) do
FactoryBot.create :user, email: "user1@rt.at", google_email: "user1@google.at"
end
let!(:johnny_gmail_user) { FactoryBot.create :user, google_email: "johnny@gmail.com" }
let!(:jane_user) { FactoryBot.create :user, email: "jane@email.at" }
let!(:zorro) { FactoryBot.create :user, email: "zorro@example.com" }
before do
FactoryBot.create(:user, google_email: "jane@email.at").tap do |u|
u.update_attribute(:deleted_at, 1.day.ago)
end
runner_gmail_user
su_12_google_user
end
context "the query is '123'" do
it_behaves_like "find users by email" do
let(:size) { 4 }
let(:number) { 1 }
let(:query) { [123] }
let(:expected_users) { [] }
let(:more_data?) { false }
end
end
context "the query contains invalid emails" do
it_behaves_like "find users by email" do
let(:query) do
["s.m@mail.com", "su+12gmx.at", "", "'", "johnny@gmail.com"]
end
let(:size) { 50 }
let(:number) { 1 }
let(:expected_users) do
[
expected_search_result_for(s_m_user),
expected_search_result_for(johnny_gmail_user)
]
end
let(:more_data?) { false }
end
end
end
end
So let’s analyze the test: It has a subject, which indicates what to test. In this case: Run the use case and return the result. It defines a shared example which contains the actual tests. These shared examples help because the tests are grouped together and they can be reused. This way it is possible to just set up the tests with different parameters and call the example by using it_behaves_like. The test above contains some user objects created with let and a before block, which is called before each test. The it-block contains two contexts to describe the setup and calls the shared example once per context. So basically this test runs 6 tests (3 tests in the shared_example, called twice). Running them locally on my laptop results in this:
Users::UseCase::UserSearch::ByAnyEmail
#run!
the query is '123'
behaves like find users by email
should return searched users
should eq false
should eq UserSearchResult
the query contains invalid emails
behaves like find users by email
should eq UserSearchResult
should eq false
should return searched users #<UserSearchResultWrapper:0x7e9d3832 @avatar_url="http://localhost.runtastic.com:3002/assets/user/def....jpg", @country_id=nil, @gender="M", @id=51, @last_name="Doe-51", @first_name="John", @guid="ab
c51"> and #<UserSearchResultWrapper:0x33c8a528 @avatar_url="http://localhost.runtastic.com:3002/assets/user/def....jpg", @country_id=nil, @gender="M", @id=55, @last_name="Doe-55", @first_name="John", @guid="abc55">
Finished in 34.78 seconds (files took 20.66 seconds to load)
6 examples, 0 failures
So about 35 seconds for 6 tests.
Let vs Let!
As you can see, we are using let! and let. The difference between these two methods is, that let! always executes, and let only executes if the reference is used. In the above example:
let!(:s_m_user) let(:runner_gmail_user)
“s_m_user” is created always, “runner_gmail_user” is created only if used. So the above let! usages are creating 7 users for the tests.
Before Block
The before block is also executed every time before the test. If nothing is passed to the before method, it defaults to :each. The above before block creates a user, and references 2 other users, which then immediately are created, too. So we are creating 10 users for each test.
rspec-it-chains
As every it is a single test, the shared example contains 3 single tests. Every test gets a clean state, so the users are created again for each test. Having multiple it blocks one after another, referring to the same subject, somehow looks like a chain.
Test setup
What do the tests actually do? The first one passes a page size of 4 with a query “123” to the search use case and expects, as no user has 123 in the email attribute, no users to be found.
context "the query is '123'" do
it_behaves_like "find users by email" do
let(:size) { 4 }
let(:number) { 1 }
let(:query) { [123] }
let(:expected_users) { [] }
let(:more_data?) { false }
end
End
So we are creating 3 times (3 it blocks) 10 users but expect no user to be found.
The second context passes some of the emails, and some invalid ones into the search, and expect 2 users to be found.
context "the query contains invalid emails" do
it_behaves_like "find users by email" do
let(:query) do
["s.m@mail.com", "su+12gmx.at", "", "'", "johnny@gmail.com"]
end
let(:size) { 50 }
let(:number) { 1 }
let(:expected_users) do
[
wrap(s_m_user),
wrap(johnny_gmail_user)
]
end
let(:more_data?) { false }
end
end
So we are creating 3 times 10 users to be able to find 2 of them in one test and get the right flag in another test. Having a closer look at the shared_example:
it { expect(subject.class).to eq UserSearchResult }
it { expect(subject.users).to return_searched_users expected_users }
it { expect(subject.more_data_available).to eq more_data? }
you can see that the first one is not even expecting anything user-related to be returned. It just expects the use-case to return a specific class. The second one actually tests if the result contains the users we want to find. The third it block checks if the more_data_available flag is set properly.
Overall, we have 6 tests, needing 35 seconds to run, creating 10 users for each test (60 users entirely) and calling the subject 6 times, and we basically only expect to find 2 users once.
Obviously, this can be improved.
Improvement
First of all, let’s get rid of the it chain, combine it within one it block.
shared_examples "find users by email" do
it "returns user info" do
expect(subject.class).to eq UserSearchResult
expect(subject.users).to return_searched_users expected_users
expect(subject.more_data_available).to eq more_data?
end
end
Combining it blocks makes sense if they usually test a similar thing (as above). For example, doing a request and expecting some response body and status 200 does not need to be two separate tests. Combining two it blocks which test something different, however, does not make sense, such as tests for the response code of a request and if that request stored the data correctly in the database.
This results in the tests finishing within ~ 15 seconds, only 2 examples.
The next step is to not create the users if they are not needed. Therefore let’s switch to let instead of let!. Also remove the before block as it is, and only create some proper amount of users necessary for the test. The tests look like this in end:
describe Users::UseCase::UserSearch::ByAnyEmail do
describe "#run!" do
subject { Users::UseCase::UserSearch::ByAnyEmail.new(search_criteria).run! }
let(:current_user_id) { nil }
let(:search_criteria) { double(query: query, size: size, number: number, current_user_id: current_user_id) }
let(:default_photo_url) { "#{Rails.configuration.services.runtastic_web.public_route}/assets/user/default_avatar_male.jpg" }
def expected_search_result_for(user)
UserSearchResultWrapper.new(user.attributes.merge("avatar_url" => default_photo_url))
end
shared_examples "find users by email" do
it "return user info" do
expect(subject.class).to eq UserSearchResult
expect(subject.users).to return_searched_users expected_users
expect(subject.more_data_available).to eq more_data?
end
end
let(:s_m_user) { FactoryBot.create :user, email: "s.m@mail.com" }
let(:runner_gmail_user) { FactoryBot.create :user, google_email: "runner@gmail.at" }
let(:su_12_user) { FactoryBot.create :user, email: "su+12@gmx.at" }
let(:su_12_google_user) { FactoryBot.create :user, google_email: "su+12@gmx.at" }
let(:user_same_mail) do
FactoryBot.create :user, email: "user@rt.com", google_email: "user@rt.com"
end
let(:combined_user) do
FactoryBot.create :user, email: "user1@rt.at", google_email: "user1@google.at"
end
let(:johnny_gmail_user) { FactoryBot.create :user, google_email: "johnny@gmail.com" }
let(:jane_user) { FactoryBot.create :user, email: "jane@email.at", fb_proxied_email: "jane@fb.at" }
let(:zorro) { FactoryBot.create :user, email: "zorro@example.com" }
let(:deleted_user) do
FactoryBot.create(:user, google_email: "jane@email.at").tap do |u|
u.update_attribute(:deleted_at, 1.day.ago)
end
end
context "the query is '123'" do
before do
s_m_user
end
it_behaves_like "find users by email" do
let(:size) { 4 }
let(:number) { 1 }
let(:query) { [123] }
let(:expected_users) { [] }
let(:more_data?) { false }
end
end
context "the query contains invalid emails" do
before do
s_m_user
su_12_user
johnny_gmail_user
end
it_behaves_like "find users by email" do
let(:query) do
["s.m@mail.com", "su+12gmx.at", "", "'", "johnny@gmail.com"]
end
let(:size) { 50 }
let(:number) { 1 }
let(:expected_users) do
[
expected_search_result_for(s_m_user),
expected_search_result_for(johnny_gmail_user)
]
end
let(:more_data?) { false }
end
end
end
end
And result in
Users::UseCase::UserSearch::ByAnyEmail
#run!
the query is '123'
behaves like find users by email
return user info
the query contains invalid emails
behaves like find users by email
return user info
Finished in 8.16 seconds (files took 22.34 seconds to load)
2 examples, 0 failures
As you can see, I do create users, even if I don’t expect them to be in the result, to prove the correctness of the use case. But I don’t create 10 per test, only 1 and 3. Some of the above users are not created (or used) at all now, but as the original test file contains more tests, which in the end need them again for other contexts, I kept them in the example too.
So now we only create 4 users, instead of 60. By just adapting the code a bit, we have the same test coverage with only 2 tests instead of 6, and only needing 8 instead of 35 seconds, which is 77% less time.
FactoryBot: create vs build vs attribute_for
As you can see above, we are using FactoryBot heavily to create objects during the tests.
let(:user) { FactoryBot.create(:user) }
This creates a new user object as soon as `user` is referenced in the tests. The drawback of this line is that it really creates the user in the database, which is pretty often not necessary. The better approach, if applicable, would be to only build the object without storing it:
let(:user) { FactoryBot.build(:user) }
Obviously this does not work if you need the object in the database, as for the test example above, but that highly depends on the test.
Another less known feature of FactoryBot is to create only the attributes for an object, represented as hash.
let(:user_attrs) { FactoryBot.attributes_for(:user) }This would create a hash containing the attributes for a user. It does not even create a User object, which is even faster than build.
A possible simple test would be:
describe User do
50.times do
subject { FactoryBot.create(:user) }
it { expect(subject.has_first_login_dialog_completed).to eq(false) }
end
end
As the has_first_login_dialog_completed method only needs some attributes set on a user, no matter if it is stored in a database, a build would be much faster than a create, running the test 100 times to also use the effect of the just-in-time compiler of the used jruby interpreter. This way the real difference between create and build is more visible. So switching from .create to .build saves about 45% of the execution time.
Finished in 1 minute 1.61 seconds (files took 23.4 seconds to load) 100 examples, 0 failures
Finished in 34.87 seconds (files took 21.69 seconds to load) 100 examples, 0 failures
Summary
So simple improvements in the tests can lead to a nice performance boost running them.
- Avoid it-chains if the tests correlate to each other
- Avoid let! in favor of let, and create the objects inside before blocks when necessary
- Avoid before blocks creating a lot of stuff which may not be necessary for all tests
- Use FactoryBot.build instead of .create if applicable.
Keep an eye on your test-suite and don’t hesitate to remove duplicate tests, maybe already obsolete tests. As (in our case) the tests are running before every merge and on every commit, try to keep your test suite fast.
***
