Testing Guide - Gumroad

Testing Stack

Gumroad uses a comprehensive testing stack for quality assurance:

RSpec 3.12 - Test framework
Capybara 3.38 - Integration testing
Selenium WebDriver - Browser automation
Factory Bot - Test data generation
Faker - Realistic fake data
VCR - HTTP interaction recording
WebMock - HTTP request stubbing

Running Tests

Setup Test Environment

Before running tests, ensure the test database is set up:

Start Docker services

make local

Setup test database

RAILS_ENV=test bin/rails db:setup

Export JavaScript constants

RAILS_ENV=test bin/rails js:export

The js:export command generates JavaScript constants for the test environment. Without this, tests may try to navigate to the development domain and encounter 502 errors.

Running Test Suite

bin/rspec

bin/rspec spec/requests/dashboard_spec.rb

bin/rspec spec/requests/dashboard_spec.rb:75

for i in {1..10}; do
  echo "Run number $i\n-"
  bin/rspec spec/requests/product_creation_spec.rb:28
done

macOS Fork Error

If you encounter fork-related errors on macOS:

objc[11912]: +[__NSCFConstantString initialize] may have been in progress in another thread when fork() was called.

Solution: Disable Spring before running tests:

export DISABLE_SPRING=1
bin/rspec spec/requests/balance_pages_spec.rb

Test Organization

Tests are organized by type:

spec/
├── business/          # Business logic tests
├── controllers/       # Controller tests
├── factories/         # Factory Bot definitions
├── helpers/           # Helper method tests
├── jobs/             # Background job tests
├── models/           # Model tests
├── policies/         # Pundit policy tests
├── presenters/       # Presenter tests
├── requests/         # Integration/E2E tests
├── services/         # Service object tests
└── sidekiq/          # Sidekiq worker tests

Writing Tests

Test Naming Conventions

Don’t use “should” in test descriptions. Be descriptive about the behavior being tested.

it "displays the user's balance" do
  expect(page).to have_text("Balance $100")
end

it "redirects to login when not authenticated" do
  visit dashboard_path
  expect(current_path).to eq(login_path)
end

it "should display the balance" do
  # Don't start with "should"
end

it "works" do
  # Too vague
end

Using Factories

Factory Bot provides test data generation:

# Define factories in spec/factories/
FactoryBot.define do
  factory :user do
    email { "seller@example.com" }
    username { "testuser" }
    password { "password" }
  end
  
  factory :named_seller, parent: :user do
    name { Faker::Name.name }
  end
end

# Use in tests
let(:seller) { create(:named_seller) }
let(:product) { create(:product, user: seller) }

Use @example.com for emails and example.com, example.org, example.net for custom domains in tests.

Integration Testing with Capybara

Capybara enables end-to-end testing through browser automation.

Writing Capybara Tests

RSpec.describe "Dashboard", js: true, type: :system do
  let(:seller) { create(:named_seller) }
  
  before do
    login_as seller
  end
  
  it "displays correct values and headings for stats" do
    visit dashboard_path
    
    within "main" do
      expect(page).to have_text("Balance $100", normalize_ws: true)
      expect(page).to have_text("Last 7 days $50", normalize_ws: true)
      expect(page).to have_text("Last 28 days $150", normalize_ws: true)
    end
  end
end

Element Selection Best Practices

Prefer semantic selectors over class names:

Text content (preferred)

click_on "Submit"
expect(page).to have_text("Success")

Input labels

fill_in "Email", with: "user@example.com"

Input placeholders

fill_in placeholder: "Search products", with: "eBook"

ARIA attributes

find("[aria-label='Close dialog']").click

Class names (last resort)

find(".product-card").click
# Only when above options don't work

Interaction Methods

# Preferred
click_on "Text"

# When text method doesn't work
find_and_click "selector", text: "Text"

# Preferred
fill_in "Email", with: "user@example.com"
fill_in "locator", with: "value"

# Alternative
find("locator").fill_in(with: "value")

# Avoid
# x.native.send_keys("...")  # Don't use this

Preventing Flaky Specs

Use have_selector for waiting

# Good - Waits automatically
expect(page).to have_selector(".notification")

# Bad - No waiting
find(".notification")

Never use sleep

# Bad
sleep 2

# Good - Wait for AJAX
wait_for_ajax

# Good - Implicit waiting
expect(page).to have_selector(".loaded")

Test multiple times locally

for i in {1..10}; do
  echo "Run number $i\n-"
  bin/rspec spec/requests/product_creation_spec.rb:28
done

Capybara’s have_selector is smart and handles waiting automatically. Always prefer it over find when checking for element presence.

Browser Setup

Chrome and ChromeDriver

Integration specs use Google Chrome:

wget -q -O - https://dl-ssl.google.com/linux/linux_signing_key.pub | sudo apt-key add -
sudo sh -c 'echo "deb [arch=amd64] http://dl.google.com/linux/chrome/deb/ stable main" >> /etc/apt/sources.list.d/google.list'
sudo apt-get update
sudo apt-get install google-chrome-stable

# Install XQuartz for display
brew install XQuartz

HTTP Mocking

VCR for Recording HTTP Interactions

VCR records HTTP requests and replays them in tests:

RSpec.describe PayPalService do
  it "processes payment" do
    VCR.use_cassette("paypal/successful_payment") do
      result = PayPalService.process_payment(amount: 100)
      expect(result).to be_success
    end
  end
end

Scope VCR cassettes to specific test files. Sharing cassettes across tests causes collisions where tests read incorrect cached responses.

WebMock for Stubbing

WebMock allows stubbing HTTP requests without recording:

stub_request(:get, "https://api.example.com/user")
  .to_return(status: 200, body: { name: "Test User" }.to_json)

Testing Background Jobs

Sidekiq Test Helpers

RSpec.describe ProcessOrderJob do
  it "enqueues the job" do
    expect {
      ProcessOrderJob.perform_async(order.id)
    }.to change { ProcessOrderJob.jobs.size }.by(1)
  end
  
  it "processes the order" do
    ProcessOrderJob.new.perform(order.id)
    expect(order.reload.status).to eq("processed")
  end
end

Avoid to_not have_enqueued_sidekiq_job or not_to have_enqueued_sidekiq_job because they’re prone to false positives. Make assertions on SidekiqWorkerName.jobs.size instead.

Testing Guidelines

General Principles

Descriptive Names

Write test names that explain the behavior being tested, not implementation details.

Group Related Tests

Use describe and context blocks to organize related test cases.

Keep Tests Independent

Each test should be able to run in isolation without depending on other tests.

Test the Fix

Tests must fail when the fix is reverted. If a test passes without the application code change, it’s invalid.

API Endpoint Testing

For API endpoints, test:

Response status codes
Response format (JSON structure)
Response content accuracy
Error cases and edge cases

RSpec.describe "Products API" do
  describe "GET /api/v2/products" do
    it "returns successful response" do
      get "/api/v2/products"
      expect(response).to have_http_status(:ok)
    end
    
    it "returns products array" do
      create(:product)
      get "/api/v2/products"
      expect(json_response["products"]).to be_an(Array)
    end
  end
end

Continuous Integration

Buildkite

Tests run automatically on Buildkite for every commit.

Reproducing CI Failures

To reproduce Buildkite test failures locally, ensure:

Same Ruby and Node versions
Fresh database setup
Elasticsearch reindexed
JavaScript exports updated

Troubleshooting

502 error navigating to checkout

Problem: Tests get 502 errors when navigating to checkout.Solution: Run the JavaScript export command:

RAILS_ENV=test bin/rails js:export

The bin/dev script automatically generates JS constants for development, so tests need this command to use the test domain.

Chromedriver connection timeout

Problem: Integration spec times out with:

Selenium::WebDriver::Error::WebDriverError:
  unable to obtain stable chromedriver connection in 60 seconds

Solution: Ensure Chrome and ChromeDriver are properly installed and versions match.

Rack timeout

Problem: Integration spec times out:

Rack::Timeout::RequestTimeoutError: Request ran for longer than 60000ms

Solution: Create .env.test.local with:

DISABLE_RACK_TIMEOUT="1"

Then restart Spring:

bin/spring stop

Visual Testing

Gumroad uses QA Wolf for visual regression testing.

Enabled by default on main branch
Not enabled by default on feature branches
Catches visual regressions before production

Test Data Best Practices

Use factories over fixtures

Factory Bot provides more flexibility than fixtures and makes tests clearer.

Use realistic data

Faker generates realistic test data that catches edge cases fixtures might miss.

Clean data between tests

RSpec’s database_cleaner ensures tests don’t interfere with each other.

Use let and let! strategically

let is lazy-loaded, let! is eager. Choose based on when data is needed.

Next Steps

Contributing

Learn testing requirements for pull requests

Architecture

Understand the system architecture

Authentication

Test authentication flows

Deployment

Deploy after tests pass

​Testing Stack

​Running Tests

​Setup Test Environment

​Running Test Suite

​macOS Fork Error

​Test Organization

​Writing Tests

​Test Naming Conventions

​Using Factories

​Integration Testing with Capybara

​Writing Capybara Tests

​Element Selection Best Practices

​Interaction Methods

​Preventing Flaky Specs

​Browser Setup

​Chrome and ChromeDriver

​HTTP Mocking

​VCR for Recording HTTP Interactions

​WebMock for Stubbing

​Testing Background Jobs

​Sidekiq Test Helpers

​Testing Guidelines

​General Principles

Descriptive Names

Group Related Tests

Keep Tests Independent

Test the Fix

​API Endpoint Testing

​Continuous Integration

​Buildkite

​Reproducing CI Failures

​Troubleshooting

​Visual Testing

​Test Data Best Practices

​Next Steps

Contributing

Architecture

Authentication

Deployment

Testing Stack

Running Tests

Setup Test Environment

Running Test Suite

macOS Fork Error

Test Organization

Writing Tests

Test Naming Conventions

Using Factories

Integration Testing with Capybara

Writing Capybara Tests

Element Selection Best Practices

Interaction Methods

Preventing Flaky Specs

Browser Setup

Chrome and ChromeDriver

HTTP Mocking

VCR for Recording HTTP Interactions

WebMock for Stubbing

Testing Background Jobs

Sidekiq Test Helpers

Testing Guidelines

General Principles

API Endpoint Testing

Continuous Integration

Buildkite

Reproducing CI Failures

Troubleshooting

Visual Testing

Test Data Best Practices

Next Steps