Upstream lobsters' own RSpec suite, run against a Roundhouse emit of the same checkout. Most failures name a Rails construct not yet modeled; some are answers that came out wrong, and a few are this harness's own gaps. That makes their tests the worklist generator — so the ranked causes below, not the pass count, are what this page is for.
4.0.0, the version their .ruby-version pins.
Current lobsters, not a snapshot. This tracks whatever upstream HEAD is and
records its SHA; it deliberately does not pin. A benchmark must freeze its input or its numbers stop
being comparable — but this answers a present-tense question, and pinning it would only manufacture
a second fossil beside the ruby-bench capture the
benchmark page measures.
Each run stands alone. There is no history here and no comparison to previous runs: numbers move when their code moves as well as when ours does, so a delta would attribute upstream's changes to us.
25 of 104 spec files in their suite are run here — spec/models. Request, feature, mailbox and routing specs are not run. Model specs exercise the compiler surface most densely; the request tier is next, and the capybara/mailbox tier is ledgered rather than attempted.
Occurrences churn; causes are the unit of work, so each carries a stable slug you can still refer to after another forty upstream commits. Owner says who can act: most of this is Roundhouse's own modeling debt.
Counts are how many examples are currently stopped by each cause, not how many will pass when it lands: an example is attributed to the first exception it raised, so clearing one cause can reveal another behind it.
| Blocked | Cause | Owner | |
|---|---|---|---|
| 15 | Symbol used where an Integer index is expectedsymbol-into-integer | roundhouse | |
Undiagnosed. Shape suggests a Hash emitted as an Array (or a symbol-keyed read lowered to an index) — needs a reduction before it can be ranked honestly.
5 of 15 examples
| |||
| 13 | Ran, answered wrongexpectation-mismatch | roundhouse | |
No exception — the emitted code executed and produced a different answer than Rails does. Each needs individual diagnosis; they cluster by message no better than by coin flip, so this bucket is a count, not a cause. Keep it visible: it is where correctness bugs hide once the crash-shaped failures clear.
5 of 13 examples
| |||
| 2 | errors is an Array, not an ActiveModel::Errorserrors-object-api | roundhouse | |
The runtime errors collection is Array[String]; attribute identity is lost at push, so errors[:attr] and errors.messages have nothing to answer with.
2 examples
| |||
| 1 | Nested constant not emittednested-constant-missing | roundhouse | |
Constants nested inside a model class (Hat::Mail, Notification::Notifiable) are dropped rather than emitted alongside the class.
1 example
| |||
| 206 | Not yet triagedunclassified | untriaged | |
No rule in bench/lobsters/spec-causes.json matches these. Triage, then add a rule.
10 of 206 examples
| |||
Rules matching nothing this run — either cleared, or the failure's message changed and is now counted as untriaged: harness-time-helpers, relation-pick, model-class-first, update-attribute, apppath-concat, kernel-bigdecimal, nil-arith-coercion, dirty-saved-change-alias, has-attribute-predicate, scope-not-on-relation, binary-column-text-escape, searchparser-parslet-dsl, keystore-upsert.
| File | Examples | Pass | Fail |
|---|---|---|---|
spec/models/story_spec.rb | 81 | 9 | 72 |
spec/models/search_spec.rb | 33 | 0 | 33 |
spec/models/user_spec.rb | 22 | 0 | 22 |
spec/models/domain_spec.rb | 20 | 1 | 19 |
spec/models/comment_spec.rb | 24 | 8 | 16 |
spec/models/notification_spec.rb | 16 | 0 | 16 |
spec/models/link_spec.rb | 14 | 0 | 14 |
spec/models/vote_spec.rb | 13 | 0 | 13 |
spec/models/category_spec.rb | 6 | 2 | 4 |
spec/models/invitation_request_spec.rb | 6 | 2 | 4 |
spec/models/message_spec.rb | 4 | 0 | 4 |
spec/models/mod_note_spec.rb | 4 | 0 | 4 |
spec/models/invitation_spec.rb | 4 | 1 | 3 |
spec/models/moderation_spec.rb | 3 | 0 | 3 |
spec/models/tag_spec.rb | 10 | 7 | 3 |
spec/models/story_image_spec.rb | 2 | 0 | 2 |
spec/models/comment_stat_spec.rb | 1 | 0 | 1 |
spec/models/hat_spec.rb | 5 | 4 | 1 |
spec/models/mod_mail_spec.rb | 1 | 0 | 1 |
spec/models/story_text_spec.rb | 1 | 0 | 1 |
spec/models/username_spec.rb | 1 | 0 | 1 |
spec/models/hat_request_spec.rb | 3 | 3 | 0 |
spec/models/keystore_spec.rb | 4 | 4 | 0 |
spec/models/mastodon_app_spec.rb | 3 | 3 | 0 |
spec/models/search_parser_spec.rb | 78 | 78 | 0 |
The shim that boots the emit in place of Rails does these things on the emit's behalf. Listed because a pass rate achieved by propping up the subject is not a pass rate about the subject.
| Compensation | Kind | Why |
|---|---|---|
require "net/http" in the shim | emit gap | app/models/sponge.rb references Net:: constants but the emit never requires net/http. |
Numeric duration builders in the shim | by design | Specs and factories are untranspiled plain Ruby; the emit path rewrites n.days at transpile time, so the shared runtime never reopens Numeric. |
A spec file or hook that failed to run at all. These are not counted as failures above, so the denominator understates by however many examples they would have contributed.
An error occurred in an `after(:context)` hook. Failure/Error: @comments.each(&:destroy!) NoMethodError: undefined method 'each' for nil # /home/rubys/lobsters/spec/models/search_spec.rb:60:in 'block (2 levels) in <top (required)>'
From the transpile that produced the tree under test: 812 files, 0 errors, 2,867 warnings. Warnings are a modeling-debt ledger rather than a failure — the Ruby target emits through its gaps — so they are not comparable to the spec failures above.
| Diagnostic | Count |
|---|---|
gradual_untyped | 1,244 |
unresolved_type | 766 |
lower_residue | 486 |
unsupported | 240 |
send_dispatch_failed | 88 |
ivar_unresolved | 18 |
missing_preload | 12 |
incompatible_binop | 11 |
blank_unlowered | 2 |