Port Errbit from MongoDB to SQLite - #3092
Open
mikelolasagasti wants to merge 42 commits into
Open
mikelolasagasti wants to merge 42 commits into
mikelolasagasti wants to merge 42 commits into
Conversation
mikelolasagasti
marked this pull request as ready for review
August 20, 2026 11:11
mikelolasagasti
force-pushed
the
port-to-sqlite
branch
from
August 21, 2026 11:11
e74273d to
72d43a0
Compare
Contributor
There was a problem hiding this comment.
Pull request overview
Ports Errbit’s runtime persistence from Mongoid/MongoDB to ActiveRecord/SQLite while retaining MongoDB for transitional migration support.
Changes:
- Adds relational models, schema, policies, jobs, and runtime integrations.
- Ports controllers, views, mailers, tasks, factories, and tests to SQL models.
- Adds SQLite deployment, persistence, WAL, backup, and upgrade guidance.
Reviewed changes
Copilot reviewed 172 out of 175 changed files in this pull request and generated 5 comments.
Show a summary per file
| File | Description |
|---|---|
spec/system/watch_unwatch_app_spec.rb |
Uses SQL factories |
spec/system/sign_in_with_google_spec.rb |
Uses SQL users |
spec/system/sign_in_with_github_spec.rb |
Uses SQL users |
spec/system/sign_in_and_sign_out_spec.rb |
Uses SQL users |
spec/support/mongoid.rb |
Makes Mongo cleanup optional |
spec/support/macros.rb |
Uses SQL users |
spec/support/active_record.rb |
Adds SQL test cleanup |
spec/requests/users_controller_spec.rb |
Ports user request tests |
spec/requests/unlink_googles_controller_spec.rb |
Ports Google unlink tests |
spec/requests/unlink_githubs_controller_spec.rb |
Ports GitHub unlink tests |
spec/requests/sql_html_runtime_spec.rb |
Adds SQL runtime coverage |
spec/requests/notices_controller_spec.rb |
Ports notice requests |
spec/requests/health_controller_spec.rb |
Ports health requests |
spec/policies/errbit/user_policy_spec.rb |
Tests SQL user policy |
spec/policies/errbit/app_policy_spec.rb |
Tests SQL app policy |
spec/models/user_spec.rb |
Tests SQL seeding |
spec/models/problem_spec.rb |
Removes migrated unmerge test |
spec/models/errbit/watcher_spec.rb |
Tests SQL watchers |
spec/models/errbit/user_spec.rb |
Tests SQL users |
spec/models/errbit/problem_spec.rb |
Tests SQL problems |
spec/models/errbit/notice_spec.rb |
Tests SQL notices |
spec/models/errbit/backtrace_spec.rb |
Tests SQL backtraces |
spec/models/errbit/app_spec.rb |
Tests SQL apps |
spec/mailers/errbit/mailer_spec.rb |
Tests SQL mailers |
spec/lib/airbrake_api/v3/notice_parser_spec.rb |
Uses SQL apps |
spec/jobs/errbit/destroy_problems_by_id_job_spec.rb |
Tests SQL deletion job |
spec/jobs/errbit/destroy_problems_by_app_job_spec.rb |
Tests app deletion job |
spec/jobs/destroy_problems_by_id_job_spec.rb |
Removes Mongo job tests |
spec/jobs/destroy_problems_by_app_job_spec.rb |
Removes Mongo job tests |
spec/interactors/user_destroy_spec.rb |
Removes Mongo interactor tests |
spec/interactors/resolved_problem_clearer_spec.rb |
Removes Mongo clearer tests |
spec/interactors/problem_recacher_spec.rb |
Removes Mongo recacher tests |
spec/interactors/problem_merge_spec.rb |
Removes Mongo merge tests |
spec/interactors/problem_destroy_spec.rb |
Removes Mongo deletion tests |
spec/interactors/outdated_problem_clearer_spec.rb |
Removes Mongo clearer tests |
spec/interactors/notice_refingerprinter_spec.rb |
Removes Mongo refingerprinting tests |
spec/interactors/errbit/user_destroy_spec.rb |
Tests SQL user deletion |
spec/interactors/errbit/problem_merge_spec.rb |
Tests SQL merging |
spec/interactors/errbit/problem_destroy_spec.rb |
Tests SQL deletion |
spec/factories/errbit/watcher_factory.rb |
Adds watcher factory |
spec/factories/errbit/user_factory.rb |
Adds user factory |
spec/factories/errbit/site_config_factory.rb |
Adds configuration factory |
spec/factories/errbit/problem_factory.rb |
Adds problem factory |
spec/factories/errbit/notification_service_factory.rb |
Adds service factories |
spec/factories/errbit/notice_fingerprinter_factory.rb |
Adds fingerprinter factory |
spec/factories/errbit/notice_factory.rb |
Adds notice factory |
spec/factories/errbit/issue_tracker_factory.rb |
Adds tracker factory |
spec/factories/errbit/err_factory.rb |
Adds error factory |
spec/factories/errbit/comment_factory.rb |
Adds comment factory |
spec/factories/errbit/backtrace_factory.rb |
Adds backtrace factory |
spec/factories/errbit/app_factory.rb |
Adds app factory |
spec/controllers/watchers_controller_spec.rb |
Ports watcher tests |
spec/controllers/users/omniauth_callbacks_controller_spec.rb |
Ports OAuth tests |
spec/controllers/site_config_controller_spec.rb |
Ports configuration tests |
spec/controllers/problems_controller_spec.rb |
Ports problem tests |
spec/controllers/notices_controller_spec.rb |
Ports notice tests |
spec/controllers/devise/sessions_controller_spec.rb |
Ports session tests |
spec/controllers/comments_controller_spec.rb |
Ports comment tests |
spec/controllers/api/v3/notices_controller_spec.rb |
Ports v3 API tests |
spec/controllers/api/v1/problems_controller_spec.rb |
Ports problem API tests |
spec/controllers/api/v1/notices_controller_spec.rb |
Ports notice API tests |
spec/controllers/api/v1/comments_controller_spec.rb |
Ports comment API tests |
README.md |
Documents SQLite runtime |
lib/tasks/errbit/demo.rake |
Ports demo generation |
lib/tasks/errbit/database.rake |
Ports maintenance tasks |
lib/tasks/errbit/bootstrap.rake |
Adds SQL bootstrap guard |
Gemfile.lock |
Locks SQLite dependency |
Gemfile |
Adds SQLite adapter |
docs/upgrading.md |
Documents data migration |
docs/quickstart.md |
Documents persistent SQLite |
docs/deployment/heroku.md |
Warns about ephemeral storage |
docs/deployment/dokku.md |
Documents persistent storage |
docs/deployment/docker.md |
Ports Docker deployment |
docs/configuration.md |
Marks Mongo migration-only |
docker-compose.yml |
Adds SQLite deployment service |
db/seeds.rb |
Seeds SQL users |
db/migrate/20260818120000_create_errbit_sql_schema.rb |
Creates relational schema |
config/routes.rb |
Configures SQL Devise model |
config/mongo.rb |
Retains migration fallback |
config/initializers/devise.rb |
Loads ActiveRecord adapter |
config/database.yml |
Configures SQLite databases |
config/application.rb |
Enables ActiveRecord |
bin/docker-entrypoint |
Automates database preparation |
app/views/problems/show.ics.erb |
Uses SQL ordering |
app/views/problems/index.html.erb |
Uses SQL user model |
app/views/problems/_table.html.erb |
Uses SQL decorator |
app/views/errbit/mailer/err_notification.text.erb |
Adds text error email |
app/views/errbit/mailer/err_notification.html.erb |
Adds HTML error email |
app/views/errbit/mailer/comment_notification.text.erb |
Adds text comment email |
app/views/errbit/mailer/comment_notification.html.erb |
Adds HTML comment email |
app/views/apps/show.html.erb |
Uses SQL user model |
app/views/apps/_service_notification_fields.html.erb |
Uses SQL service hierarchy |
app/policies/errbit/user_policy.rb |
Adds SQL user policy |
app/policies/errbit/application_policy.rb |
Adds SQL policy base |
app/policies/errbit/app_policy.rb |
Adds SQL app policy |
app/policies/application_policy.rb |
Generalizes policy documentation |
app/models/errbit/watcher.rb |
Adds relational watchers |
app/models/errbit/user.rb |
Adds relational users |
app/models/errbit/site_config.rb |
Adds relational configuration |
app/models/errbit/problem.rb |
Adds relational problems |
app/models/errbit/notification_services/webhook_service.rb |
Ports webhook integration |
app/models/errbit/notification_services/slack_service.rb |
Ports Slack integration |
app/models/errbit/notification_services/pushover_service.rb |
Ports Pushover integration |
app/models/errbit/notification_services/hubot_service.rb |
Ports Hubot integration |
app/models/errbit/notification_services/hoiio_service.rb |
Ports Hoiio integration |
app/models/errbit/notification_services/campfire_service.rb |
Ports Campfire integration |
app/models/errbit/notification_service.rb |
Adds SQL service base |
app/models/errbit/notice.rb |
Adds relational notices |
app/models/errbit/notice_fingerprinter.rb |
Adds SQL fingerprinting |
app/models/errbit/issue.rb |
Ports issue operations |
app/models/errbit/issue_tracker.rb |
Adds relational trackers |
app/models/errbit/error_report.rb |
Ports notice ingestion |
app/models/errbit/err.rb |
Adds relational error groups |
app/models/errbit/comment.rb |
Adds relational comments |
app/models/errbit/backtrace.rb |
Adds relational backtraces |
app/models/errbit/application_record.rb |
Adds SQL model base |
app/models/errbit/app.rb |
Adds relational apps |
app/models/errbit.rb |
Adds SQL namespace support |
app/mailers/errbit/mailer.rb |
Adds SQL-backed mailer |
app/mailers/errbit/application_mailer.rb |
Adds mailer base |
app/lib/airbrake_api/v3/notice_parser.rb |
Uses SQL error reports |
app/jobs/errbit/destroy_problems_by_id_job.rb |
Adds SQL deletion job |
app/jobs/errbit/destroy_problems_by_app_job.rb |
Adds SQL app cleanup job |
app/jobs/errbit/application_job.rb |
Adds namespaced job base |
app/jobs/destroy_problems_by_id_job.rb |
Removes Mongo deletion job |
app/jobs/destroy_problems_by_app_job.rb |
Removes Mongo app job |
app/interactors/user_destroy.rb |
Removes Mongo user deletion |
app/interactors/resolved_problem_clearer.rb |
Removes Mongo resolved clearer |
app/interactors/problem_recacher.rb |
Removes Mongo recacher |
app/interactors/problem_merge.rb |
Removes Mongo merger |
app/interactors/problem_destroy.rb |
Removes Mongo deletion |
app/interactors/outdated_problem_clearer.rb |
Removes Mongo outdated clearer |
app/interactors/notice_refingerprinter.rb |
Removes Mongo refingerprinter |
app/interactors/errbit/user_destroy.rb |
Adds SQL user deletion |
app/interactors/errbit/resolved_problem_clearer.rb |
Adds SQL resolved clearer |
app/interactors/errbit/problem_recacher.rb |
Adds SQL recacher |
app/interactors/errbit/problem_merge.rb |
Adds SQL problem merger |
app/interactors/errbit/problem_destroy.rb |
Adds SQL problem deletion |
app/interactors/errbit/outdated_problem_clearer.rb |
Adds SQL outdated clearer |
app/interactors/errbit/notice_refingerprinter.rb |
Adds SQL refingerprinter |
app/helpers/apps_helper.rb |
Uses relational app ordering |
app/helpers/application_helper.rb |
Uses SQL decorators |
app/decorators/errbit/problem_decorator.rb |
Adds problem decorator |
app/decorators/errbit/notice_decorator.rb |
Adds notice decorator |
app/decorators/errbit/issue_tracker_type_decorator.rb |
Adds tracker-type decorator |
app/decorators/errbit/issue_tracker_field_decorator.rb |
Adds tracker-field decorator |
app/decorators/errbit/issue_tracker_decorator.rb |
Adds tracker decorator |
app/decorators/errbit/backtrace_line_decorator.rb |
Adds backtrace-line decorator |
app/decorators/errbit/backtrace_decorator.rb |
Adds backtrace decorator |
app/decorators/errbit/app_decorator.rb |
Adds app decorator |
app/controllers/watchers_controller.rb |
Ports watcher actions |
app/controllers/users/omniauth_callbacks_controller.rb |
Ports OAuth persistence |
app/controllers/users_controller.rb |
Ports user CRUD |
app/controllers/unlink_googles_controller.rb |
Ports Google unlinking |
app/controllers/unlink_githubs_controller.rb |
Ports GitHub unlinking |
app/controllers/site_config_controller.rb |
Ports configuration updates |
app/controllers/problems_controller.rb |
Ports problem management |
app/controllers/notices_controller.rb |
Ports notice ingestion |
app/controllers/health_controller.rb |
Uses SQL API keys |
app/controllers/concerns/problems_searcher.rb |
Removes Mongo search concern |
app/controllers/comments_controller.rb |
Ports comment management |
app/controllers/apps_controller.rb |
Ports app CRUD |
app/controllers/application_controller.rb |
Uses SQL token authentication |
app/controllers/api/v3/notices_controller.rb |
Ports v3 ingestion response |
app/controllers/api/v1/stats_controller.rb |
Ports statistics queries |
app/controllers/api/v1/problems_controller.rb |
Ports problem API serialization |
app/controllers/api/v1/notices_controller.rb |
Ports notice API serialization |
app/controllers/api/v1/comments_controller.rb |
Ports comment API serialization |
.gitignore |
Ignores SQLite artifacts |
.github/workflows/rubocop.yml |
Updates lint target |
.env.default |
Removes Mongo runtime default |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
| build: . | ||
| environment: | ||
| RAILS_ENV: production | ||
| SECRET_KEY_BASE: change-this-secret |
| require "active_model/railtie" | ||
| require "active_job/railtie" | ||
| # require "active_record/railtie" | ||
| require "active_record/railtie" |
Comment on lines
+171
to
+174
| messages: increment_hash_counter(problem.messages, Digest::MD5.hexdigest(notice.message.to_s), notice.message), | ||
| hosts: increment_hash_counter(problem.hosts, Digest::MD5.hexdigest(notice.host.to_s), notice.host), | ||
| user_agents: increment_hash_counter(problem.user_agents, Digest::MD5.hexdigest(notice.user_agent_string.to_s), notice.user_agent_string) | ||
| ) |
| SQLite instead: | ||
|
|
||
| ```shell | ||
| docker compose exec errbit sqlite3 /rails/storage/production.sqlite3 ".backup '/rails/storage/production-backup.sqlite3'" |
Comment on lines
+189
to
+207
| def uncache_notice(notice) | ||
| last_notice = notices.reorder(created_at: :desc).first | ||
| return unless last_notice | ||
|
|
||
| attrs = { | ||
| environment: last_notice.environment_name, | ||
| error_class: last_notice.error_class, | ||
| last_notice_at: last_notice.created_at, | ||
| message: last_notice.message, | ||
| where: last_notice.where, | ||
| notices_count: (notices_count.to_i > 1) ? notices_count - 1 : 0 | ||
| } | ||
|
|
||
| CACHED_NOTICE_ATTRIBUTES.each do |attr, source| | ||
| attrs[attr] = decrement_hash_counter(send(attr), Digest::MD5.hexdigest(notice.send(source).to_s)) | ||
| end | ||
|
|
||
| update!(attrs) | ||
| end |
Add sqlite3 gem, enable ActiveRecord railtie, and configure database.yml for SQLite. Add Errbit::ApplicationRecord as the base for all SQL models. Update Devise to load the ActiveRecord ORM and point routes to Errbit::User.
Create a single migration defining all 13 tables for Errbit's data model. Every table includes a unique `bson_id` column to preserve MongoDB identity for migration mapping. JSON columns store flexible payloads (request, notifier, server_environment, cached attributes). Text columns for unpredictable strings (message, body).
Create `Errbit::*` ActiveRecord models mirroring the Mongoid models. Models use `Errbit::` table prefix and preserve all Mongoid associations, validations, and scopes. Includes STI support for notification services and issue trackers.
Add FactoryBot factories for every SQL model and model specs covering validations, associations, and scope behavior. Configure RSpec to load the SQL schema before tests and truncate SQL tables between examples while keeping Mongoid support for migration specs.
Add `errbit:migrate:*` rake tasks that copy data from MongoDB to SQLite in dependency order. Each task is idempotent by `bson_id` and uses batched cursor iteration for memory efficiency. Timestamps are preserved via `record_timestamps = false`.
Create `Errbit::ErrorReport` that processes incoming notices using SQL models instead of Mongoid. Uses `find_or_create_by!` for backtraces with unique fingerprint index. Skips notification delivery and problem recaching when `Errbit.migrating?` is true.
Port all problem-related interactors to use SQL models: - ProblemDestroy: uncache notices, destroy problem and errs - ProblemMerge: merge multiple problems into one - ProblemRecacher: rebuild cached attributes from notices - NoticeRefingerprinter: reprocess notices with new fingerprint - ResolvedProblemClearer / OutdatedProblemClearer: cleanup jobs
Create `Errbit::*` decorators wrapping SQL models for the view layer. BacktraceLineDecorator handles the most complex view logic with regex matching for in-app detection, gem detection, and path normalization. All decorators delegate to the wrapped model.
Port async problem deletion jobs to SQL. `DestroyProblemsByIdJob` accepts an array of problem IDs and destroys them sequentially. `DestroyProblemsByAppJob` finds all problems for an app and destroys them. Both use the SQL `Errbit::ProblemDestroy` interactor.
Add Pundit policies for SQL models: `Errbit::ApplicationPolicy` (base with `permitted_attributes` abstract method), `Errbit::AppPolicy` (admin-only CRUD), `Errbit::UserPolicy` (admin-only index/destroy, self-edit). Update `ApplicationPolicy` doc comment to reference SQL models.
Create `Errbit::Mailer` with `err_notification` and `comment_notification` actions. Templates use SQL decorators for consistent rendering. Copies the original mailer's headers and notification logic but targets SQL models.
Wire the notifier endpoints (Hoptoad v2 and API v3) to use SQL models. Update `Errbit::ErrorReport` reference, health controller to use SQL user, and API v3 notice parser.
Wire API v1 controllers to SQL models: comments, notices, problems, and stats. Update application controller to use SQL user. Update routes and specs. The backtrace factory reference in specs was updated to use the SQL factory.
Wire user-facing controllers to SQL models: registration, profile editing, deletion, GitHub/Google OAuth callbacks, and unlink controllers. Update request specs to use SQL factories.
Wire the main problem flows: problems controller (search, filter, sort, resolve, unresolve, merge, delete), apps controller (problem listing), comments controller, watchers controller. Update helpers and view partials. Add request specs for the full problem flows.
Wire site config controller to SQL model. Update specs and application helper to use SQL models.
Remove the `ProblemsSearcher` concern that provided `selected_problems` and `err_ids` helpers. These methods are now defined directly in `AppsController` and `ProblemsController` since the concern was designed for Mongoid's `Problem.find` behavior. Update apps/show view to reference the new helper.
Update the app notification service form partial to work with SQL models. The form handles STI for notification services (Slack, Webhook, etc.) via `type` attribute. Update request specs.
Remove all legacy Mongoid interactors (problem_destroy, problem_merge, problem_recacher, notice_refingerprinter, resolved_problem_clearer, outdated_problem_clearer, user_destroy) and their specs. Remove legacy jobs (destroy_problems_by_app_job, destroy_problems_by_id_job) and their specs now that callers use the namespaced SQL implementations. Update database.rake to reference SQL models.
Update the demo rake task to use SQL factories (`:errbit_app`, `:errbit_problem`, etc.) and `Errbit::ErrorReport`. This allows the demo task to work with the SQL runtime.
Update `db/seeds.rb` to use `Errbit::User` instead of `User` for admin user creation. Update bootstrap rake task to reference SQL models.
Add `errbit:migrate:verify` task that compares SQL row counts against MongoDB source data. Verifies embedded relations (watchers, issue_trackers, notification_services, notice_fingerprinters), problem state preservation (resolved, notices_count, cached attributes), and relationship mappings (problem->app, err->problem, notice->app/err/backtrace, comment->problem/user).
Add `Errbit.migrating?` flag and `with_import_mode` helper to suppress callbacks during migration import. Models check this flag to skip notification delivery, problem recaching, app denormalization, and notice-destroy callback effects. This prevents unwanted side effects when importing large datasets.
Add `ERRBIT_MIGRATE_STRICT` environment variable that causes the migration to raise an error when any task records failed rows. This enables CI operators to fail fast with non-zero exit when migration data is incomplete.
Add request specs that exercise the full HTML runtime against SQL models: apps index/show/create/edit/copy/delete, problems index/show, notice redirect flows. These specs prove that all SQL models are wired correctly and the view layer renders without errors.
Explicitly set `use_active_record = false` in RSpec config to prevent accidental SQL queries in Mongoid-only specs. This makes it clear which specs exercise the SQL runtime vs the Mongo runtime.
Update `bin/docker-entrypoint` to run `errbit:bootstrap` (which runs `db:migrate` and `db:seed`) on container boot. This ensures the SQL database is ready before the Rails server starts. Update bootstrap task to include `prepare_mongo_migration` for Mongo index creation before migration.
Add three migration tests based on real production data shapes. These cases prevent import failures caused by nulls, empty values, unusually large payloads, and values that do not map cleanly to SQL: - Notices with OGNL/attack payloads in messages (from real dump) - Large hash distributions in cached problem attributes (1140+ counts) - Problems with notices_count of 0
Add nine tests covering edge cases found in anonymized production data. These cases protect imports against missing values, empty values, STI base records, and unusually large or multiline payloads: - Base NotificationService (no subclass fields) STI fallback - Problems with resolved_at timestamps (17 resolved in dump) - Notices with null environment-name in server_environment - Backtraces with empty lines - Large user agent distributions (113 entries max in dump) - Multi-line error messages (300 chars with newlines) - Apps with empty bitbucket_repo - Base NotificationService with notify_at_notices [0] - server_environment with null app-version
Wrap migration writes in bounded transactions and cache SQL ID mappings per task to reduce query volume during large imports. Keep failures visible, reset mappings when dependent data changes, and record the verified cutover marker only after the full migration and verification pass complete.
Cover migration failures caused by invalid relationships and unique SQL collisions. Verify that failed imports can resume safely and that application startup refuses unverified MongoDB data while allowing a verified migration to proceed.
Refuse to seed or serve an empty SQL database when legacy MongoDB data is present without a completed migration and verification marker. Run the cutover guard during bootstrap so accidental data loss requires an explicit, verified migration first.
Document persistent storage requirements, SQLite WAL limitations, safe backup procedures, and the MongoDB-to-SQL upgrade path. Add the SQLite configuration task and ignore the verified cutover marker in deployments.
Document the persistent storage directory, required container ownership, SQLite WAL sidecar files, and manual configuration when bootstrap is bypassed. Ignore local MongoDB dumps so migration artifacts are not added to the repository.
Update controller, request, and factory references from Mongoid models to the namespaced ActiveRecord models. Adapt expectations for SQL relations, private controller helpers, pagination, persistence, and error handling while preserving the existing controller behavior coverage.
mikelolasagasti
force-pushed
the
port-to-sqlite
branch
from
September 3, 2026 18:47
72d43a0 to
cc466fe
Compare
Remove the implicit MongoDB URL from the default environment and only run the MongoDB cutover guard when a MongoDB connection is explicitly configured. Keep Mongoid and the Devise Mongoid adapter available for the bridge-release migration path. Update installation, Docker, deployment, and upgrade documentation, and add fresh-install regression coverage.
mikelolasagasti
force-pushed
the
port-to-sqlite
branch
from
September 3, 2026 22:26
cc466fe to
59f7bdd
Compare
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This PR completes the migration of Errbit's application runtime from MongoDB/Mongoid to ActiveRecord backed by SQLite.
The goal is to make SQLite the runtime database while preserving MongoDB and Mongoid temporarily as an in-process migration path. This is intentionally a transition release rather than the final MongoDB removal.
Background
This PR builds on lessons from the earlier partial SQL migration PR and the cph and Undev forks. Those implementations informed the relational model design, BSON ID preservation, dependency-ordered imports, callback suppression, batching, cached problem state, and resumable migrations.
The recommended release sequence is:
Errbit 0.11: prepare and back up MongoDB
Errbit 0.12: deploy the transition release
/rails/storagemust be writable by UID/GID1000.Errbit 0.12: migrate MongoDB data
db:migrateanderrbit:sqlite:configurein the 0.12 environment before starting the data migration.errbit:migrate:allagainst the existing MongoDB database.Errbit 0.12: verify the cutover
errbit:migrate:verifyexplicitly aftererrbit:migrate:all.test -f storage/.mongo_to_sql_migratedfor a host bind mount, ortest -f /rails/storage/.mongo_to_sql_migratedinside the container.Errbit 0.12: operate on SQLite
Future release: remove MongoDB migration support
What Changed
Errbit::*ActiveRecord models.Real-World Migration Verification
The migration was tested against a restored real MongoDB dump in an isolated Podman environment. The archive was created with MongoDB 6.0.26 and was restored using the matching
mongo:6.0.26image. The originalerrbitdatabase namespace was preserved.The source dump contained:
In total, 23,782 MongoDB documents were restored and migrated.
Results:
errbit:migrate:all: passed with 0 failureserrbit:migrate:verify: 9,203 checks, 0 failuresMigration fixtures and edge-case coverage were created from production-data shapes after anonymization, using synthetic values.
Operational Notes
/rails/storagemust persistproduction.sqlite3,-wal,-shm, and.mongo_to_sql_migrated.db:migratefollowed byerrbit:sqlite:configurebefore serving traffic.docker compose exec errbit sqlite3 /rails/storage/production.sqlite3 ".backup '/rails/storage/production-backup.sqlite3'". This procedure is also documented indocs/quickstart.md.