Testing a Rails upgrade in CI without hiding failures

A Rails upgrade in CI needs a passing current-version job and a visible result for the target version. Keep main shipping, run both dependency graphs, reject new deprecations and reproduce failures before changing the framework version.

An experimental job may fail without failing the workflow. That is useful while removing upgrade blockers, but the green workflow badge cannot approve the final switch. Remove the target job’s failure allowance and require its check before merging that switch.

Give each Rails version a visible result

The generated application accompanying this article keeps Ruby 3.4.10 and Rails defaults at 8.0 while testing Rails 8.0.5.1 and 8.1.3.1 against PostgreSQL 15. Separate lockfiles preserve each dependency graph. These are fixture choices, not a claim about Allerin production infrastructure.

next_rails 1.7.0 supplies the switch. Its --init command creates Gemfile.next and a next? helper for the Gemfile. The application helper is NextRails.next?. The recorded setup uses:

bundle exec next_rails --init
bundle exec next bundle install
BUNDLE_GEMFILE=Gemfile.next bin/rails test

Use dependencies that support both frameworks, merge compatible repairs into main, and keep the framework switch small. The fixture’s first form-request test caught a dependency failure before that switch. The current-version bundle resolved JSON 3.0.0, then raised ArgumentError: unknown keyword: quirks_mode. Constraining the shared JSON dependency to the compatible 2.x line preserved the application assertion. JSON 3.0.0, released on 7 September 2026, tightened option validation; that change and the exact Rails encoder explain this recorded path. With JSON 2.21.2, both serial suites passed seven tests and 18 assertions. JSON change, Rails 8.0 encoder.

name: Rails upgrade
on: [push, pull_request]
permissions:
  contents: read
concurrency:
  group: ${{ github.workflow }}-${{ github.ref }}
  cancel-in-progress: true
jobs:
  test:
    name: Rails (${{ matrix.gemfile }})
    runs-on: ubuntu-24.04
    timeout-minutes: 15
    continue-on-error: ${{ matrix.experimental }}
    strategy:
      fail-fast: false
      matrix:
        include:
          - gemfile: Gemfile
            experimental: false
            timezone: UTC
          - gemfile: Gemfile.next
            experimental: true
            timezone: America/New_York
    services:
      postgres:
        image: postgres:15.17
        env:
          POSTGRES_PASSWORD: fixture-only
        ports: ['5432:5432']
        options: >-
          --health-cmd pg_isready --health-interval 10s
          --health-timeout 5s --health-retries 5
    env:
      BUNDLE_GEMFILE: ${{ matrix.gemfile }}
      DATABASE_URL: postgres://postgres:fixture-only@localhost:5432/allerin_ci67_test
      RAILS_ENV: test
      CI: '1'
      SECRET_KEY_BASE: public-ci-fixture-only-not-a-deployment-secret-public-ci-fixture-only
      RUBYOPT: '-W:deprecated'
      PARALLEL_WORKERS: '2'
      TZ: ${{ matrix.timezone }}
    steps:
      - uses: actions/checkout@v4
      - uses: ruby/setup-ruby@v1
        with:
          ruby-version: '3.4.10'
          bundler: '2.5.22'
          bundler-cache: true
      - run: bin/rails db:prepare
      - name: Run every check
        run: |
          if [ "$BUNDLE_GEMFILE" = Gemfile.next ]; then
            bin/ci
          else
            bundle exec ruby script/ci_legacy.rb
          fi
      - uses: actions/upload-artifact@v4
        if: always()
        with:
          name: rails-${{ matrix.gemfile }}
          path: |
            log/test.log
            tmp/screenshots/
          retention-days: 7

This workflow deliberately permits the target job to fail while the upgrade is unfinished. Inspect that job’s steps and retained output separately. fail-fast: false lets both cells report. The concurrency group cancels superseded runs, and the timeout limits a stuck job. These controls bound waste; they do not promise that two cells cost exactly twice as much. Workflow semantics.

Rails 7.2 introduced generated GitHub CI, Brakeman and the RuboCop omakase configuration in 2024. The released 8.1.3.1 workflow still lacks the 15-minute timeout added to main by PR #58595 on 28 August 2026. The example adopts that limit explicitly. 7.2 release, timeout change.

Run the same checks locally

Rails 8.1 introduced bin/ci and config/ci.rb in 2025. Its runner executes all steps, reports their results and exits unsuccessfully if any failed. It does not stop after the first failure. Here is the complete fixture definition. Runner source.

CI.run do
  step "Style", "bundle exec rubocop"
  step "Security scan", "bundle exec brakeman --no-pager"
  step "Gem audit", "bundle exec bundler-audit check --update"
  step "JavaScript audit", "bin/importmap audit"
  step "Autoload", "bin/rails zeitwerk:check"
  step "Schema drift", "sh script/check_schema.sh"
  step "Unit and integration tests", "bin/rails test"
  step "System tests", "bin/rails test:system"
end

The current Rails 8.0 cell cannot load the 8.1 runner. Its equivalent plain script retains the same checks and final failure status:

require "rbconfig"
$stdout.sync = true
ENV["CI"] = "true"
ENV["PATH"] = "#{File.dirname(RbConfig.ruby)}#{File::PATH_SEPARATOR}#{ENV.fetch('PATH')}"
steps = [
  [ "Style", "bundle exec rubocop" ],
  [ "Security scan", "bundle exec brakeman --no-pager" ],
  [ "Gem audit", "bundle exec bundler-audit check --update" ],
  [ "JavaScript audit", "bin/importmap audit" ],
  [ "Autoload", "bin/rails zeitwerk:check" ],
  [ "Schema drift", "sh script/check_schema.sh" ],
  [ "Unit and integration tests", "bin/rails test" ],
  [ "System tests", "bin/rails test:system" ]
]
results = steps.map do |label, command|
  puts "\n#{label}"
  success = system(command)
  [ label, success ]
end
abort "Failed steps: #{results.reject(&:last).map(&:first).join(', ')}" unless results.all?(&:last)

The lockfiles identify Brakeman 8.0.6, bundler-audit 0.9.3, rubocop-rails-omakase 1.1.0 and importmap-rails 2.2.3. The advisory database is refreshed by the audit command. bin/importmap audit applies to this importmap application; a different JavaScript package system needs its own check. Passing scans describe their findings, not a security certification.

Both Linux cells completed all eight pipeline steps. Each ran seven tests and 18 assertions in two processes, plus one headless Chrome test with two assertions. The validation wrapper also checked deliberate failures and required both cells to pass. A local macOS forked database connection crashed before an assertion result; local serial runs and Linux process runs are recorded separately. Recorded results.

A 2007 Rails core discussion about CruiseControl.rb asked for the breaking changeset and useful failure details. Preserve that information in today’s artifacts too. A link to an empty report is still unhelpful. Original discussion.

Reject new warnings without hiding known ones

Rails 6.1 added disallowed-deprecation controls in 2020, before the current time-based maintenance policy. config.active_support.disallowed_deprecation_warnings selects warnings for config.active_support.disallowed_deprecation. It is a denylist. A growing list can prevent repaired warnings returning, but it cannot catch every new warning. Disallowed warnings.

The fixture uses a narrow known-warning allowlist and raises for everything else:

module CiDeprecationPolicy
  KNOWN = [ "CI fixture old API will be removed" ].freeze
  HANDLER = lambda do |message, _callstack|
    normalized = message.sub(/\ADEPRECATION WARNING: /, "").sub(/ \(called from .*\z/m, "")
    if KNOWN.include?(normalized)
      warn "Known deprecation: #{normalized}"
    else
      raise ActiveSupport::DeprecationException, message
    end
  end
end

# config/environments/test.rb, Rails 7.1 and later
require_relative "../../lib/ci_deprecation_policy"
Rails.application.configure do
  config.active_support.deprecation = CiDeprecationPolicy::HANDLER
end

For the separately exercised Active Support 6.1.7.10 probe, load the same policy and use the older setter:

ActiveSupport::Deprecation.behavior = CiDeprecationPolicy::HANDLER

Rails 7.1’s 2023 per-library deprecators changed the object being configured. Rails.application.deprecators manages registered deprecators, including later additions. The separate 6.1 probe uses the older singleton setter. Both probes permit the known warning and reject the unknown one with ActiveSupport::DeprecationException. Collection.

For a large existing backlog, deprecation_toolkit 2.4.0 can maintain a reviewed per-test baseline. Its record mode passed, the unchanged baseline passed, and controlled additions and removals failed with DeprecationIntroduced and DeprecationRemoved. Remove obsolete exceptions and review every addition so the backlog shrinks. Recording belongs in a deliberate baseline change, never an unconditional CI step. Toolkit instructions.

Ruby 3.4’s 2024 release also changed Hash#inspect spacing. Compare values when formatting is irrelevant; keep explicit output assertions where presentation is the feature. The Ruby upgrade reproduction records that failure separately. Ruby release.

RUBYOPT="-W:deprecated" also reveals Ruby warnings, including Ruby 3.4’s chilled-string warnings. It does not itself turn those warnings into build failures. A Rails deprecation handler does not automatically govern Ruby’s warning channel.

Check what the environment can conceal

Rails 7.2 changed tests to respect an explicitly configured Active Job adapter consistently. Without configuration, the test adapter remains the fallback. An application using Solid Queue must still decide whether a test checks enqueue intent or a real worker. 7.2 upgrade guide.

In both tested versions, asserting that perform_later had already changed a record failed. Asserting enqueue intent passed; wrapping the enqueue in perform_enqueued_jobs passed when the assertion needed the performed result. An explicitly configured inline adapter also made assert_enqueued_with reject the setup because that helper requires the test adapter.

When checking error pages, inspect Rails 7.1’s 2023 show_exceptions values and your application’s chosen setting. The generated :rescuable setting renders configured HTTP errors while other exceptions still raise. Upgrade guide.

Strict loading, introduced in Rails 6.1 in 2020, can expose unintended association loads; adopt it deliberately where the suite can assert the expected queries. 6.1 release.

Use the production database engine and major version in CI. SQLite is not a substitute for testing PostgreSQL types, constraints or query behavior. Keep RAILS_ENV=test explicit for the autoload and migration checks:

set -eu
export RAILS_ENV=test
bin/rails zeitwerk:check
git ls-files --error-unmatch db/schema.rb db/schema.next.rb >/dev/null
bin/rails db:migrate
git diff --exit-code -- db/schema.rb db/schema.next.rb

The shared schema baseline failed the next-version drift check. Rails 8.1 changed its version header and column order. The repair keeps reviewed schema.rb and schema.next.rb baselines, selected through database.yml’s schema_dump setting. Both checks then passed. The gate still rejects an intentional edit. 8.1 schema change.

The pipeline does not establish safe lock duration on production tables. Review migration operations separately; strong_migrations 2.8.0 is an optional check, not an executed part of this fixture. Gem source.

Rails 7.0’s 2021 generator introduced config.eager_load = ENV["CI"].present?. Upgrading gems does not rewrite an older application’s test.rb. zeitwerk:check, available with Rails 6.0’s 2019 autoloader, provides a named check rather than relying on which test happens to load a constant. 6.0 task, Test environment.

Reproduce a flaky failure before quarantining it

Rails 4.0 moved from Test::Unit to MiniTest in 2013; 4.1 adopted Minitest::Test in 2014. Rails 5.0 made random ordering the ActiveSupport test default in 2016. A seed helps reproduce a run; it does not repair shared state. 4.0 source, 4.1 source, upgrade guide.

bin/rails test checks/order_dependence_test.rb --seed 3911
bundle exec minitest_bisect --seed 3911 -Itest checks/order_dependence_test.rb
bin/rails test checks/order_dependence_fixed_test.rb --seed 3911

The recorded failure expected UTC and received Hawaii. minitest-bisect 1.8.0 reduced four tests to the two that leaked and observed the application’s zone. Restoring the zone after each test made the same seed pass four assertions. Process TZ and Rails’ application time zone are separate settings. Bisect instructions.

Rails 6.0 added parallel tests in 2019. Rails 7.0 added the default threshold of 50 tests; the normal condition is more than 50, not 50 or more. Setting PARALLEL_WORKERS can override that threshold. Treat CPU count as a starting budget and account for database connections and memory. The fixture forces two workers on Linux. 7.0 threshold, Threshold source.

travel_to arrived in Rails 4.1 in 2014; freeze_time in Rails 5.2 in 2018. System tests and shared transactional test connections arrived in 5.1. Quarantine a remaining flaky test only with an owner and linked issue. Blanket retries hide evidence. 4.1 release, Time helpers, system-test release.

The unordered-select shuffle in PR #58548 was on main at the 7 September check, disabled by default. It does not cover every SQL path and is not a released setting in the tested frameworks. No Rails 8.2 delivery date is assumed. Shuffle discussion.

What to do this week

1. Establish passing current-version checks, then add the target lockfile and visible experimental job.
2. Turn a demonstrated failure into a check, review the warning baseline and merge compatible repairs on main.
3. Match the database engine and major, exercise browser and job behavior, and retain seeds, logs and screenshots.
4. Remove the target job’s failure allowance and require its check before the framework switch. Rehearse deployment separately; CI does not prove production migration safety.

FastRuby’s September article demonstrates the same principle for website audits, with content, link and metadata checks. Its scope is distinct from the upgrade experiments here. Audit checks.

Methods checked 7 September 2026. Codex researched tagged sources and executed a generated application and deliberate failures. These are bounded reproductions, not a client case study or byline-owner production account. Download the application, lockfiles, workflow and verification script. The samples are not published as a separate public repository. Retest the recorded dependencies against current patches before an application upgrade.

For the application’s upgrade sequence, use Allerin’s Rails upgrade guide.