How many times have you heard this before? It will usually be followed by rerunning the test suite a few more times to see if the flaky tests keep flaking. If it does not; move on and forget about the flaky test till the next time you try to deploy.

Flaky tests are a big problem. A test suite is only useful when it’s fast and reliable. The problem with flaky tests is that they slow down the test suite quite a bit (causing re-runs of the build at every attempt to merge a PR sometimes), and when they fail, you won’t know immediately if the test flaked as usual, or if there’s a real bug that requires your attention.

There are a lot of ways to write bad, non-deterministic tests. This post is about a subtle one that’s relatively straightforward to fix: hardcoding IDs.

Why are hardcoded IDs bad

RSpec.describe "User adding a specific book to their collection" do
  let(:user) { create(:user) }
  let(:first_book) { create(:book, id: 1) }

  it "adds the first book to the user's collection" do
    # This almost guarantees to introduce a test order dependent error
    expect(user.books.first.id).to eq(1)
  end
end

PostgreSQL often uses sequences to decide on the ID new records will get. They start at one and keep increasing.

Most test frameworks like to rollback a database transaction after the test runs, however, the rollback does not roll back sequences. This feature of PostgreSQL (and many other DBs) means that we need to account for different starting conditions every time we run our test suite.

If we do not take this into account, it’s very likely to run into this error with a hardcoded ID:

ActiveRecord::RecordNotUnique:
  PG::UniqueViolation: ERROR: duplicate key value violates unique constraint "books_pkey"
  DETAIL: Key (id)=(1) already exists.

What to do if you need specific IDs in your tests

Don’t!

If for whatever reason your application code has a hardcoded ID, don’t use that as an excuse to do the same mistake in your tests.

You might see something like this in your codebase and continue along with writing a flaky test.

const favoriteBook = user.favoriteBook === null ?
  {id: "23"} :
  {id: user.favoriteBook.id}

Instead of doing that, stop for a bit and think about why the ID is being used here. Is there a way you can identify this object better? Do you usually identify a book by its ID? Or can the identity of a book be represented better by a combination of its name and its author and the actual contents of the book?

Once you’ve decided to get rid of the hardcoded ID from the application, it might make more sense to first finish writing your test with the hardcoded ID and make sure it passes in isolation. Use this new temporarily flaky test to change the application to replace the hardcoded ID with something less brittle, and finally remove the hardcoded ID from your test as well.

const favoriteBook = user.favoriteBook === null ?
  {name: "Ulysses", author: "James Joyce"} :
  {name: user.favoriteBook.name, author: user.favoriteBook.author}

How to replace hardcoded IDs in tests

RSpec.describe "User ordering a copy of their favorite book" do
  let(:user) { create(:user) }
  let(:ulysses) { create(:book, id: "23") }

  it "adds the default favorite book to the user's order" do
    expect(user.order.favorite_book.id).to eq("23")
  end
end

There are two main reasons why this test is brittle. First, if the ID of “Ulysses” in our actual database ever changes, this test will become obsolete.

The second, and more likely the problem, is that this test depends on the order of the tests being run in the suite, and what books get created in other tests. If everything aligns, this test might pass. But if a different book gets created with the ID “23” earlier, this test will fail.

So how do we fix this? It’s easy!

Instead of making an assertion on the hardcoded ID value, make an assertion that the user’s order has the expected default book object.

RSpec.describe "User ordering a copy of their favorite book" do
  let(:user) { create(:user) }
  let(:ulysses) { create(:book, name: "Ulysses", author: "James Joyce") }
	
  it "adds the default favorite book to the user's order" do
    expect(user.order.favorite_book).to eq(ulysses)
  end
end

I hope this convinces you to never hardcode IDs in your tests again and move towards a more robust, less brittle test suite.