Sitelet https://github.com/errbit/errbit/pull/3092
Skip to content

Port Errbit from MongoDB to SQLite - #3092

Open
mikelolasagasti wants to merge 42 commits into
errbit:mainfrom
mikelolasagasti:port-to-sqlite
Open

mikelolasagasti wants to merge 42 commits into
errbit:mainfrom
mikelolasagasti:port-to-sqlite

Conversation

@mikelolasagasti

@mikelolasagasti mikelolasagasti commented Aug 20, 2026 •

Copy link
Copy Markdown
Contributor

This PR completes the migration of Errbit's application runtime from MongoDB/Mongoid to ActiveRecord backed by SQLite.

The goal is to make SQLite the runtime database while preserving MongoDB and Mongoid temporarily as an in-process migration path. This is intentionally a transition release rather than the final MongoDB removal.

Background

This PR builds on lessons from the earlier partial SQL migration PR and the cph and Undev forks. Those implementations informed the relational model design, BSON ID preservation, dependency-ordered imports, callback suppression, batching, cached problem state, and resumable migrations.

The recommended release sequence is:

  1. Errbit 0.11: prepare and back up MongoDB

    • Continue operating Errbit on the existing MongoDB runtime.
    • Take and validate a MongoDB backup before upgrading.
    • Schedule a migration window and stop Errbit, or otherwise prevent new notices from being written while migration is running.
    • Keep the backup until the SQLite deployment has been validated.
  2. Errbit 0.12: deploy the transition release

    • Deploy the version containing this PR with MongoDB/Mongoid migration support still enabled.
    • Configure persistent, writable SQLite storage.
    • For Docker bind mounts, /rails/storage must be writable by UID/GID 1000.
  3. Errbit 0.12: migrate MongoDB data

    • Run db:migrate and errbit:sqlite:configure in the 0.12 environment before starting the data migration.
    • Run errbit:migrate:all against the existing MongoDB database.
    • The migration runs in dependency order, preserves original BSON IDs and timestamps, suppresses import side effects, and is safe to rerun after a partial failure.
    • The command fails if rows cannot be imported.
    • The command runs the post-migration verification pass before recording the completed cutover.
  4. Errbit 0.12: verify the cutover

    • Run errbit:migrate:verify explicitly after errbit:migrate:all.
    • Confirm that row counts, relationship mappings, cached problem state, and representative application flows are correct.
    • Confirm that the cutover marker is present in persistent storage: test -f storage/.mongo_to_sql_migrated for a host bind mount, or test -f /rails/storage/.mongo_to_sql_migrated inside the container.
  5. Errbit 0.12: operate on SQLite

    • Start Errbit normally.
    • Controllers, API endpoints, notice ingestion, jobs, mailers, policies, maintenance tasks, and seeds now use the ActiveRecord models.
    • Bootstrap refuses to seed or serve an empty SQL database if MongoDB data is detected without a verified cutover.
    • Keep MongoDB available as migration-only support and retain the backup during the validation period.
  6. Future release: remove MongoDB migration support

    • Remove Mongoid, MongoDB dependencies, Mongoid-specific test dependencies, and legacy root models.
    • Remove MongoDB configuration, environment variables, Devise Mongoid loading, CI coverage, and deployment services.
    • Keep upgrade documentation for users migrating from 0.12.

What Changed

  • Added ActiveRecord models for users, apps, watchers, site configuration, integrations, problems, errs, notices, backtraces, and comments.
  • Added the SQLite schema and persistent BSON ID mappings.
  • Ported the runtime from Mongoid models to Errbit::* ActiveRecord models.
  • Added idempotent MongoDB-to-SQL migration tasks with batching and explicit dependency order.
  • Added migration side-effect suppression for mail, notifications, denormalization, and recaching callbacks.
  • Added post-migration verification and fail-closed migration behavior.
  • Added SQL runtime coverage for controllers, requests, system flows, models, policies, jobs, mailers, and migration behavior.
  • Added SQLite WAL configuration and deployment documentation for persistence, permissions, and safe backups.

Real-World Migration Verification

The migration was tested against a restored real MongoDB dump in an isolated Podman environment. The archive was created with MongoDB 6.0.26 and was restored using the matching mongo:6.0.26 image. The original errbit database namespace was preserved.

The source dump contained:

  • 9 users
  • 3 apps
  • 1 site configuration
  • 0 watchers
  • 2 issue trackers
  • 2 notification services
  • 3 notice fingerprinters
  • 17,890 backtraces
  • 368 problems
  • 368 errs
  • 5,143 notices
  • 0 comments

In total, 23,782 MongoDB documents were restored and migrated.

Results:

  • errbit:migrate:all: passed with 0 failures
  • errbit:migrate:verify: 9,203 checks, 0 failures
  • Repeated verification: passed
  • Post-migration bootstrap: passed
  • SQLite WAL smoke check: passed
  • Controller/request suite: 216 examples, 0 failures
  • System suite: 11 examples, 0 failures
  • Focused migration/model/runtime suite: 39 examples, 0 failures
  • RuboCop and StandardRB checks: passed

Migration fixtures and edge-case coverage were created from production-data shapes after anonymization, using synthetic values.

Operational Notes

  • SQLite WAL improves reader/writer concurrency but SQLite still permits only one writer. SQLite is intended for small, low-write, single-process deployments; higher-write deployments should use a server database.
  • /rails/storage must persist production.sqlite3, -wal, -shm, and .mongo_to_sql_migrated.
  • If bootstrap is bypassed or deployment is manual, run db:migrate followed by errbit:sqlite:configure before serving traffic.
  • Do not copy the SQLite database file while Errbit is running. Use SQLite's backup mechanism for consistent backups, for example: docker compose exec errbit sqlite3 /rails/storage/production.sqlite3 ".backup '/rails/storage/production-backup.sqlite3'". This procedure is also documented in docs/quickstart.md.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Ports Errbit’s runtime persistence from Mongoid/MongoDB to ActiveRecord/SQLite while retaining MongoDB for transitional migration support.

Changes:

  • Adds relational models, schema, policies, jobs, and runtime integrations.
  • Ports controllers, views, mailers, tasks, factories, and tests to SQL models.
  • Adds SQLite deployment, persistence, WAL, backup, and upgrade guidance.

Reviewed changes

Copilot reviewed 172 out of 175 changed files in this pull request and generated 5 comments.

Show a summary per file
File Description
spec/system/watch_unwatch_app_spec.rb Uses SQL factories
spec/system/sign_in_with_google_spec.rb Uses SQL users
spec/system/sign_in_with_github_spec.rb Uses SQL users
spec/system/sign_in_and_sign_out_spec.rb Uses SQL users
spec/support/mongoid.rb Makes Mongo cleanup optional
spec/support/macros.rb Uses SQL users
spec/support/active_record.rb Adds SQL test cleanup
spec/requests/users_controller_spec.rb Ports user request tests
spec/requests/unlink_googles_controller_spec.rb Ports Google unlink tests
spec/requests/unlink_githubs_controller_spec.rb Ports GitHub unlink tests
spec/requests/sql_html_runtime_spec.rb Adds SQL runtime coverage
spec/requests/notices_controller_spec.rb Ports notice requests
spec/requests/health_controller_spec.rb Ports health requests
spec/policies/errbit/user_policy_spec.rb Tests SQL user policy
spec/policies/errbit/app_policy_spec.rb Tests SQL app policy
spec/models/user_spec.rb Tests SQL seeding
spec/models/problem_spec.rb Removes migrated unmerge test
spec/models/errbit/watcher_spec.rb Tests SQL watchers
spec/models/errbit/user_spec.rb Tests SQL users
spec/models/errbit/problem_spec.rb Tests SQL problems
spec/models/errbit/notice_spec.rb Tests SQL notices
spec/models/errbit/backtrace_spec.rb Tests SQL backtraces
spec/models/errbit/app_spec.rb Tests SQL apps
spec/mailers/errbit/mailer_spec.rb Tests SQL mailers
spec/lib/airbrake_api/v3/notice_parser_spec.rb Uses SQL apps
spec/jobs/errbit/destroy_problems_by_id_job_spec.rb Tests SQL deletion job
spec/jobs/errbit/destroy_problems_by_app_job_spec.rb Tests app deletion job
spec/jobs/destroy_problems_by_id_job_spec.rb Removes Mongo job tests
spec/jobs/destroy_problems_by_app_job_spec.rb Removes Mongo job tests
spec/interactors/user_destroy_spec.rb Removes Mongo interactor tests
spec/interactors/resolved_problem_clearer_spec.rb Removes Mongo clearer tests
spec/interactors/problem_recacher_spec.rb Removes Mongo recacher tests
spec/interactors/problem_merge_spec.rb Removes Mongo merge tests
spec/interactors/problem_destroy_spec.rb Removes Mongo deletion tests
spec/interactors/outdated_problem_clearer_spec.rb Removes Mongo clearer tests
spec/interactors/notice_refingerprinter_spec.rb Removes Mongo refingerprinting tests
spec/interactors/errbit/user_destroy_spec.rb Tests SQL user deletion
spec/interactors/errbit/problem_merge_spec.rb Tests SQL merging
spec/interactors/errbit/problem_destroy_spec.rb Tests SQL deletion
spec/factories/errbit/watcher_factory.rb Adds watcher factory
spec/factories/errbit/user_factory.rb Adds user factory
spec/factories/errbit/site_config_factory.rb Adds configuration factory
spec/factories/errbit/problem_factory.rb Adds problem factory
spec/factories/errbit/notification_service_factory.rb Adds service factories
spec/factories/errbit/notice_fingerprinter_factory.rb Adds fingerprinter factory
spec/factories/errbit/notice_factory.rb Adds notice factory
spec/factories/errbit/issue_tracker_factory.rb Adds tracker factory
spec/factories/errbit/err_factory.rb Adds error factory
spec/factories/errbit/comment_factory.rb Adds comment factory
spec/factories/errbit/backtrace_factory.rb Adds backtrace factory
spec/factories/errbit/app_factory.rb Adds app factory
spec/controllers/watchers_controller_spec.rb Ports watcher tests
spec/controllers/users/omniauth_callbacks_controller_spec.rb Ports OAuth tests
spec/controllers/site_config_controller_spec.rb Ports configuration tests
spec/controllers/problems_controller_spec.rb Ports problem tests
spec/controllers/notices_controller_spec.rb Ports notice tests
spec/controllers/devise/sessions_controller_spec.rb Ports session tests
spec/controllers/comments_controller_spec.rb Ports comment tests
spec/controllers/api/v3/notices_controller_spec.rb Ports v3 API tests
spec/controllers/api/v1/problems_controller_spec.rb Ports problem API tests
spec/controllers/api/v1/notices_controller_spec.rb Ports notice API tests
spec/controllers/api/v1/comments_controller_spec.rb Ports comment API tests
README.md Documents SQLite runtime
lib/tasks/errbit/demo.rake Ports demo generation
lib/tasks/errbit/database.rake Ports maintenance tasks
lib/tasks/errbit/bootstrap.rake Adds SQL bootstrap guard
Gemfile.lock Locks SQLite dependency
Gemfile Adds SQLite adapter
docs/upgrading.md Documents data migration
docs/quickstart.md Documents persistent SQLite
docs/deployment/heroku.md Warns about ephemeral storage
docs/deployment/dokku.md Documents persistent storage
docs/deployment/docker.md Ports Docker deployment
docs/configuration.md Marks Mongo migration-only
docker-compose.yml Adds SQLite deployment service
db/seeds.rb Seeds SQL users
db/migrate/20260818120000_create_errbit_sql_schema.rb Creates relational schema
config/routes.rb Configures SQL Devise model
config/mongo.rb Retains migration fallback
config/initializers/devise.rb Loads ActiveRecord adapter
config/database.yml Configures SQLite databases
config/application.rb Enables ActiveRecord
bin/docker-entrypoint Automates database preparation
app/views/problems/show.ics.erb Uses SQL ordering
app/views/problems/index.html.erb Uses SQL user model
app/views/problems/_table.html.erb Uses SQL decorator
app/views/errbit/mailer/err_notification.text.erb Adds text error email
app/views/errbit/mailer/err_notification.html.erb Adds HTML error email
app/views/errbit/mailer/comment_notification.text.erb Adds text comment email
app/views/errbit/mailer/comment_notification.html.erb Adds HTML comment email
app/views/apps/show.html.erb Uses SQL user model
app/views/apps/_service_notification_fields.html.erb Uses SQL service hierarchy
app/policies/errbit/user_policy.rb Adds SQL user policy
app/policies/errbit/application_policy.rb Adds SQL policy base
app/policies/errbit/app_policy.rb Adds SQL app policy
app/policies/application_policy.rb Generalizes policy documentation
app/models/errbit/watcher.rb Adds relational watchers
app/models/errbit/user.rb Adds relational users
app/models/errbit/site_config.rb Adds relational configuration
app/models/errbit/problem.rb Adds relational problems
app/models/errbit/notification_services/webhook_service.rb Ports webhook integration
app/models/errbit/notification_services/slack_service.rb Ports Slack integration
app/models/errbit/notification_services/pushover_service.rb Ports Pushover integration
app/models/errbit/notification_services/hubot_service.rb Ports Hubot integration
app/models/errbit/notification_services/hoiio_service.rb Ports Hoiio integration
app/models/errbit/notification_services/campfire_service.rb Ports Campfire integration
app/models/errbit/notification_service.rb Adds SQL service base
app/models/errbit/notice.rb Adds relational notices
app/models/errbit/notice_fingerprinter.rb Adds SQL fingerprinting
app/models/errbit/issue.rb Ports issue operations
app/models/errbit/issue_tracker.rb Adds relational trackers
app/models/errbit/error_report.rb Ports notice ingestion
app/models/errbit/err.rb Adds relational error groups
app/models/errbit/comment.rb Adds relational comments
app/models/errbit/backtrace.rb Adds relational backtraces
app/models/errbit/application_record.rb Adds SQL model base
app/models/errbit/app.rb Adds relational apps
app/models/errbit.rb Adds SQL namespace support
app/mailers/errbit/mailer.rb Adds SQL-backed mailer
app/mailers/errbit/application_mailer.rb Adds mailer base
app/lib/airbrake_api/v3/notice_parser.rb Uses SQL error reports
app/jobs/errbit/destroy_problems_by_id_job.rb Adds SQL deletion job
app/jobs/errbit/destroy_problems_by_app_job.rb Adds SQL app cleanup job
app/jobs/errbit/application_job.rb Adds namespaced job base
app/jobs/destroy_problems_by_id_job.rb Removes Mongo deletion job
app/jobs/destroy_problems_by_app_job.rb Removes Mongo app job
app/interactors/user_destroy.rb Removes Mongo user deletion
app/interactors/resolved_problem_clearer.rb Removes Mongo resolved clearer
app/interactors/problem_recacher.rb Removes Mongo recacher
app/interactors/problem_merge.rb Removes Mongo merger
app/interactors/problem_destroy.rb Removes Mongo deletion
app/interactors/outdated_problem_clearer.rb Removes Mongo outdated clearer
app/interactors/notice_refingerprinter.rb Removes Mongo refingerprinter
app/interactors/errbit/user_destroy.rb Adds SQL user deletion
app/interactors/errbit/resolved_problem_clearer.rb Adds SQL resolved clearer
app/interactors/errbit/problem_recacher.rb Adds SQL recacher
app/interactors/errbit/problem_merge.rb Adds SQL problem merger
app/interactors/errbit/problem_destroy.rb Adds SQL problem deletion
app/interactors/errbit/outdated_problem_clearer.rb Adds SQL outdated clearer
app/interactors/errbit/notice_refingerprinter.rb Adds SQL refingerprinter
app/helpers/apps_helper.rb Uses relational app ordering
app/helpers/application_helper.rb Uses SQL decorators
app/decorators/errbit/problem_decorator.rb Adds problem decorator
app/decorators/errbit/notice_decorator.rb Adds notice decorator
app/decorators/errbit/issue_tracker_type_decorator.rb Adds tracker-type decorator
app/decorators/errbit/issue_tracker_field_decorator.rb Adds tracker-field decorator
app/decorators/errbit/issue_tracker_decorator.rb Adds tracker decorator
app/decorators/errbit/backtrace_line_decorator.rb Adds backtrace-line decorator
app/decorators/errbit/backtrace_decorator.rb Adds backtrace decorator
app/decorators/errbit/app_decorator.rb Adds app decorator
app/controllers/watchers_controller.rb Ports watcher actions
app/controllers/users/omniauth_callbacks_controller.rb Ports OAuth persistence
app/controllers/users_controller.rb Ports user CRUD
app/controllers/unlink_googles_controller.rb Ports Google unlinking
app/controllers/unlink_githubs_controller.rb Ports GitHub unlinking
app/controllers/site_config_controller.rb Ports configuration updates
app/controllers/problems_controller.rb Ports problem management
app/controllers/notices_controller.rb Ports notice ingestion
app/controllers/health_controller.rb Uses SQL API keys
app/controllers/concerns/problems_searcher.rb Removes Mongo search concern
app/controllers/comments_controller.rb Ports comment management
app/controllers/apps_controller.rb Ports app CRUD
app/controllers/application_controller.rb Uses SQL token authentication
app/controllers/api/v3/notices_controller.rb Ports v3 ingestion response
app/controllers/api/v1/stats_controller.rb Ports statistics queries
app/controllers/api/v1/problems_controller.rb Ports problem API serialization
app/controllers/api/v1/notices_controller.rb Ports notice API serialization
app/controllers/api/v1/comments_controller.rb Ports comment API serialization
.gitignore Ignores SQLite artifacts
.github/workflows/rubocop.yml Updates lint target
.env.default Removes Mongo runtime default

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment thread docker-compose.yml Outdated
build: .
environment:
RAILS_ENV: production
SECRET_KEY_BASE: change-this-secret
Comment thread config/application.rb
require "active_model/railtie"
require "active_job/railtie"
# require "active_record/railtie"
require "active_record/railtie"
Comment thread app/models/errbit/problem.rb Outdated
Comment on lines +171 to +174
messages: increment_hash_counter(problem.messages, Digest::MD5.hexdigest(notice.message.to_s), notice.message),
hosts: increment_hash_counter(problem.hosts, Digest::MD5.hexdigest(notice.host.to_s), notice.host),
user_agents: increment_hash_counter(problem.user_agents, Digest::MD5.hexdigest(notice.user_agent_string.to_s), notice.user_agent_string)
)
Comment thread docs/quickstart.md
SQLite instead:

```shell
docker compose exec errbit sqlite3 /rails/storage/production.sqlite3 ".backup '/rails/storage/production-backup.sqlite3'"
Comment on lines +189 to +207
def uncache_notice(notice)
last_notice = notices.reorder(created_at: :desc).first
return unless last_notice

attrs = {
environment: last_notice.environment_name,
error_class: last_notice.error_class,
last_notice_at: last_notice.created_at,
message: last_notice.message,
where: last_notice.where,
notices_count: (notices_count.to_i > 1) ? notices_count - 1 : 0
}

CACHED_NOTICE_ATTRIBUTES.each do |attr, source|
attrs[attr] = decrement_hash_counter(send(attr), Digest::MD5.hexdigest(notice.send(source).to_s))
end

update!(attrs)
end
Add sqlite3 gem, enable ActiveRecord railtie, and configure
database.yml for SQLite. Add Errbit::ApplicationRecord as the base
for all SQL models. Update Devise to load the ActiveRecord ORM and
point routes to Errbit::User.
Create a single migration defining all 13 tables for Errbit's data
model. Every table includes a unique `bson_id` column to preserve
MongoDB identity for migration mapping. JSON columns store flexible
payloads (request, notifier, server_environment, cached attributes).
Text columns for unpredictable strings (message, body).
Create `Errbit::*` ActiveRecord models mirroring the Mongoid models.
Models use `Errbit::` table prefix and preserve all Mongoid
associations, validations, and scopes. Includes STI support for
notification services and issue trackers.
Add FactoryBot factories for every SQL model and model specs
covering validations, associations, and scope behavior. Configure
RSpec to load the SQL schema before tests and truncate SQL tables
between examples while keeping Mongoid support for migration specs.
Add `errbit:migrate:*` rake tasks that copy data from MongoDB to
SQLite in dependency order. Each task is idempotent by `bson_id`
and uses batched cursor iteration for memory efficiency. Timestamps
are preserved via `record_timestamps = false`.
Create `Errbit::ErrorReport` that processes incoming notices using
SQL models instead of Mongoid. Uses `find_or_create_by!` for
backtraces with unique fingerprint index. Skips notification
delivery and problem recaching when `Errbit.migrating?` is true.
Port all problem-related interactors to use SQL models:
- ProblemDestroy: uncache notices, destroy problem and errs
- ProblemMerge: merge multiple problems into one
- ProblemRecacher: rebuild cached attributes from notices
- NoticeRefingerprinter: reprocess notices with new fingerprint
- ResolvedProblemClearer / OutdatedProblemClearer: cleanup jobs
Create `Errbit::*` decorators wrapping SQL models for the view layer.
BacktraceLineDecorator handles the most complex view logic with
regex matching for in-app detection, gem detection, and path
normalization. All decorators delegate to the wrapped model.
Port async problem deletion jobs to SQL. `DestroyProblemsByIdJob`
accepts an array of problem IDs and destroys them sequentially.
`DestroyProblemsByAppJob` finds all problems for an app and
destroys them. Both use the SQL `Errbit::ProblemDestroy` interactor.
Add Pundit policies for SQL models: `Errbit::ApplicationPolicy`
(base with `permitted_attributes` abstract method),
`Errbit::AppPolicy` (admin-only CRUD), `Errbit::UserPolicy`
(admin-only index/destroy, self-edit). Update `ApplicationPolicy`
doc comment to reference SQL models.
Create `Errbit::Mailer` with `err_notification` and
`comment_notification` actions. Templates use SQL decorators for
consistent rendering. Copies the original mailer's headers and
notification logic but targets SQL models.
Wire the notifier endpoints (Hoptoad v2 and API v3) to use SQL
models. Update `Errbit::ErrorReport` reference, health controller
to use SQL user, and API v3 notice parser.
Wire API v1 controllers to SQL models: comments, notices, problems,
and stats. Update application controller to use SQL user. Update
routes and specs. The backtrace factory reference in specs was
updated to use the SQL factory.
Wire user-facing controllers to SQL models: registration, profile
editing, deletion, GitHub/Google OAuth callbacks, and unlink
controllers. Update request specs to use SQL factories.
Wire the main problem flows: problems controller (search, filter,
sort, resolve, unresolve, merge, delete), apps controller (problem
listing), comments controller, watchers controller. Update helpers
and view partials. Add request specs for the full problem flows.
Wire site config controller to SQL model. Update specs and
application helper to use SQL models.
Remove the `ProblemsSearcher` concern that provided `selected_problems`
and `err_ids` helpers. These methods are now defined directly in
`AppsController` and `ProblemsController` since the concern was
designed for Mongoid's `Problem.find` behavior. Update apps/show
view to reference the new helper.
Update the app notification service form partial to work with SQL
models. The form handles STI for notification services (Slack,
Webhook, etc.) via `type` attribute. Update request specs.
Remove all legacy Mongoid interactors (problem_destroy, problem_merge,
problem_recacher, notice_refingerprinter, resolved_problem_clearer,
outdated_problem_clearer, user_destroy) and their specs. Remove
legacy jobs (destroy_problems_by_app_job, destroy_problems_by_id_job)
and their specs now that callers use the namespaced SQL implementations.
Update database.rake to reference SQL models.
Update the demo rake task to use SQL factories (`:errbit_app`,
`:errbit_problem`, etc.) and `Errbit::ErrorReport`. This allows
the demo task to work with the SQL runtime.
Update `db/seeds.rb` to use `Errbit::User` instead of `User` for
admin user creation. Update bootstrap rake task to reference SQL
models.
Add `errbit:migrate:verify` task that compares SQL row counts
against MongoDB source data. Verifies embedded relations (watchers,
issue_trackers, notification_services, notice_fingerprinters),
problem state preservation (resolved, notices_count, cached
attributes), and relationship mappings (problem->app, err->problem,
notice->app/err/backtrace, comment->problem/user).
Add `Errbit.migrating?` flag and `with_import_mode` helper to
suppress callbacks during migration import. Models check this flag
to skip notification delivery, problem recaching, app denormalization,
and notice-destroy callback effects. This prevents unwanted side
effects when importing large datasets.
Add `ERRBIT_MIGRATE_STRICT` environment variable that causes the
migration to raise an error when any task records failed rows.
This enables CI operators to fail fast with non-zero exit when
migration data is incomplete.
Add request specs that exercise the full HTML runtime against SQL
models: apps index/show/create/edit/copy/delete, problems index/show,
notice redirect flows. These specs prove that all SQL models are
wired correctly and the view layer renders without errors.
Explicitly set `use_active_record = false` in RSpec config to
prevent accidental SQL queries in Mongoid-only specs. This makes
it clear which specs exercise the SQL runtime vs the Mongo runtime.
Update `bin/docker-entrypoint` to run `errbit:bootstrap` (which
runs `db:migrate` and `db:seed`) on container boot. This ensures
the SQL database is ready before the Rails server starts. Update
bootstrap task to include `prepare_mongo_migration` for Mongo
index creation before migration.
Add three migration tests based on real production data shapes. These
cases prevent import failures caused by nulls, empty values, unusually
large payloads, and values that do not map cleanly to SQL:
- Notices with OGNL/attack payloads in messages (from real dump)
- Large hash distributions in cached problem attributes (1140+ counts)
- Problems with notices_count of 0
Add nine tests covering edge cases found in anonymized production data.
These cases protect imports against missing values, empty values, STI
base records, and unusually large or multiline payloads:
- Base NotificationService (no subclass fields) STI fallback
- Problems with resolved_at timestamps (17 resolved in dump)
- Notices with null environment-name in server_environment
- Backtraces with empty lines
- Large user agent distributions (113 entries max in dump)
- Multi-line error messages (300 chars with newlines)
- Apps with empty bitbucket_repo
- Base NotificationService with notify_at_notices [0]
- server_environment with null app-version
Wrap migration writes in bounded transactions and cache SQL ID mappings
per task to reduce query volume during large imports. Keep failures
visible, reset mappings when dependent data changes, and record the
verified cutover marker only after the full migration and verification
pass complete.
Cover migration failures caused by invalid relationships and unique SQL
collisions. Verify that failed imports can resume safely and that
application startup refuses unverified MongoDB data while allowing a
verified migration to proceed.
Refuse to seed or serve an empty SQL database when legacy MongoDB data is
present without a completed migration and verification marker. Run the
cutover guard during bootstrap so accidental data loss requires an
explicit, verified migration first.
Document persistent storage requirements, SQLite WAL limitations, safe
backup procedures, and the MongoDB-to-SQL upgrade path. Add the SQLite
configuration task and ignore the verified cutover marker in deployments.
Document the persistent storage directory, required container ownership,
SQLite WAL sidecar files, and manual configuration when bootstrap is
bypassed. Ignore local MongoDB dumps so migration artifacts are not added
to the repository.
Update controller, request, and factory references from Mongoid models to
the namespaced ActiveRecord models. Adapt expectations for SQL relations,
private controller helpers, pagination, persistence, and error handling
while preserving the existing controller behavior coverage.
Remove the implicit MongoDB URL from the default environment and only
run the MongoDB cutover guard when a MongoDB connection is explicitly
configured. Keep Mongoid and the Devise Mongoid adapter available for
the bridge-release migration path.

Update installation, Docker, deployment, and upgrade documentation, and
add fresh-install regression coverage.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants