Lobsters conformance

Upstream lobsters' own RSpec suite, run against a Roundhouse emit of the same checkout. Most failures name a Rails construct not yet modeled; some are answers that came out wrong, and a few are this harness's own gaps. That makes their tests the worklist generator — so the ranked causes below, not the pass count, are what this page is for.

lobsters 9e849fd42b22 · roundhouse 727ebe6c0027 · 2026-07-31T11:58:55Z

48/354 examples pass25 of 104 spec filesemit 742 files, 0 errors

48 passing · 306 failing · 1 error(s) outside examples

What this measures. Not whether Roundhouse compiles lobsters — the Ruby target is gap-tolerant and answers that with a zero whether or not the output behaves. Their suite asserts behavior example by example, so a failure here names a construct that was emitted but is wrong or missing. The emit is produced fresh from the same checkout the specs come from, and runs on Ruby 4.0.0, the version their .ruby-version pins.

Current lobsters, not a snapshot. This tracks whatever upstream HEAD is and records its SHA; it deliberately does not pin. A benchmark must freeze its input or its numbers stop being comparable — but this answers a present-tense question, and pinning it would only manufacture a second fossil beside the ruby-bench capture the benchmark page measures.

Each run stands alone. There is no history here and no comparison to previous runs: numbers move when their code moves as well as when ours does, so a delta would attribute upstream's changes to us.

Scope

25 of 104 spec files in their suite are run here — spec/models. Request, feature, mailbox and routing specs are not run. Model specs exercise the compiler surface most densely; the request tier is next, and the capybara/mailbox tier is ledgered rather than attempted.

Todos, by failures blocked

Occurrences churn; causes are the unit of work, so each carries a stable slug you can still refer to after another forty upstream commits. Owner says who can act: most of this is Roundhouse's own modeling debt.

Counts are how many examples are currently stopped by each cause, not how many will pass when it lands: an example is attributed to the first exception it raised, so clearing one cause can reveal another behind it.

Blocked		Cause	Owner
133		a model scope does not chain off a Relation `scope-not-on-relation`	roundhouse
	Scopes lower to class methods on the model, so Model.scope resolves but relation.scope does not — lobsters reaches them through associations (user.comments.active, where :active is scope :active on Comment). Matches the whole family rather than one scope name, since which scope surfaces first is an accident of call order. Sibling of the residual noted in the Relation[T] work: scalar class methods deliberately do not forward. 5 of 133 examples Comment should get a short id Comment links to users during and after creation Comment subtracts karma if mod intervenes Comment hat can be one of the user's hats Comment speed limit is not enforced as a regular validation
78		SearchParser emits an empty class — parslet's class-body DSL is dropped `searchparser-parslet-dsl`	roundhouse
	parslet `rule(...)` calls in a class body are silently dropped though main.rb requires the gem, so SearchParser emits as a 2-line empty class. Needs verbatim class-body passthrough for non-AR classes on the ruby family, and a warning either way. 5 of 78 examples SearchParser parses SearchParser stopwords parses a stopword SearchParser stopwords parses a stopword followed by whitespace SearchParser stopwords doesn't a stopword prefixing a term SearchParser stopwords doesn't parse a random short non-stopword
24		Symbol used where an Integer index is expected `symbol-into-integer`	roundhouse
	Undiagnosed. Shape suggests a Hash emitted as an Array (or a symbol-keyed read lowered to an index) — needs a reduction before it can be ranked honestly. 5 of 24 examples Comment hat can't be worn if user doesn't have that hat InvitationRequest has a limit on the email field InvitationRequest has a limit on the memo field InvitationRequest has a limit on the name field InvitationRequest has a limit on the ip_address field
21		Ran, answered wrong `expectation-mismatch`	roundhouse
	No exception — the emitted code executed and produced a different answer than Rails does. Each needs individual diagnosis; they cluster by message no better than by coin flip, so this bucket is a count, not a cause. Keep it visible: it is where correctness bugs hide once the crash-shaped failures clear. 5 of 21 examples Category validations does not allow a category with a name too long to be saved Category logs modification in moderation log logs on create Comment extracts links from markdown Comment.accessible_to_user when user is a moderator Comment.accessible_to_user when user does not a moderator
11		nil reaching integer arithmetic `nil-arith-coercion`	roundhouse
	A nullable read flows into arithmetic without narrowing. Sibling of the nullable-column arc that closed across all targets on 07-25 — this is the arithmetic seam that arc did not cover. 5 of 11 examples Story scopes hidden exclude tags are empty Story scopes hidden excluded tags are provided Story scopes newest exclude tags are emtpy returns two stories Story scopes newest exclude tags are emtpy first story in a list is last created Story scopes newest exclude tags are emtpy last story in a list is first created
6		Kernel#BigDecimal not available on models `kernel-bigdecimal`	roundhouse
	Kernel-level conversion function called from model bodies; the runtime exposes no BigDecimal. 5 of 6 examples Comment hat logs hat use on new comment Comment hat logs hat use on edited comment that adds it Comment hat doesn't relog hat use on edit that doesn't change hat Comment confidence is low for flagged comments Comment confidence it is high for upvoted comments
5		Rails::AppPath#+ unimplemented `apppath-concat`	roundhouse
	Rails.root is modeled as an AppPath value class that supports join but not +. 5 examples Story fetching titles can fetch PDF titles properly Story fetching titles can fetch its title properly Story fetching titles does not follow rel=canonical when this is to the main page Story fetching titles does not assign canonical url when the response is non-200 Story fetching titles assigns canonical when url when it resolves 200
5		Model.first unimplemented as a class method `model-class-first`	roundhouse
	Relation#first exists; the class-level shorthand does not forward to it. Related to the scalar-class-method forwarding gap. 5 examples Category logs modification in moderation log logs on update Tag validations allows a valid tag to be created Tag logs modification in moderation log logs on create Tag logs modification in moderation log logs on update User unbans a user
4		errors is an Array, not an ActiveModel::Errors `errors-object-api`	roundhouse
	The runtime errors collection is Array[String]; attribute identity is lost at push, so errors[:attr] and errors.messages have nothing to answer with. 4 examples ModNote validates the length of note ModNote validates the presence of note Moderation validates the length of action Moderation validates the length of reason
3		update_attribute unimplemented `update-attribute`	roundhouse
	Single-attribute write that skips validation. 3 examples User doesn't allow changing usernames in < 1 year of last change User allows changing username with no creation/changing in < 1 year User doesn't allow changing to a username someone else stopped using in the last 5 years
2		Nested constant not emitted `nested-constant-missing`	roundhouse
	Constants nested inside a model class (Hat::Mail, Notification::Notifiable) are dropped rather than emitted alongside the class. 2 examples Hat santizes email addresses Notification should_display? messages should display all messages
1		travel_to unavailable to specs `harness-time-helpers`	this harness
	ActiveSupport::Testing::TimeHelpers is a spec-side helper stock Rails mixes into example groups. Ours to provide in the shim, not compiler debt. 1 example User allows totp codes from the surrounding time steps
13		Not yet triaged `unclassified`	untriaged
	No rule in bench/lobsters/spec-causes.json matches these. Triage, then add a rule. 10 of 13 examples Domain origin takes a selector and replacement to generate an origin identifier `NoMethodError: undefined method 'find_each' for an instance of Array` Domain origin creates a bare-domain origin for bare and trailing slash URLs `NoMethodError: undefined method 'find_each' for an instance of Array` Domain ban should have moderation moderation should be created `SQLite3::SQLException: no such column: domain:` Domain ban should have moderation has correct moderator_user_id `SQLite3::SQLException: no such column: domain:` Domain ban should have moderation has correct action `SQLite3::SQLException: no such column: domain:` Domain ban should have moderation has correct reason `SQLite3::SQLException: no such column: domain:` Domain unban should have moderation moderation should be created `SQLite3::SQLException: no such column: domain:` Domain unban should have moderation has correct moderator_user_id `SQLite3::SQLException: no such column: domain:` Domain unban should have moderation has correct action `SQLite3::SQLException: no such column: domain:` Domain unban should have moderation has correct reason `SQLite3::SQLException: no such column: domain:`

Rules matching nothing this run — either cleared, or the failure's message changed and is now counted as untriaged: keystore-upsert, has-attribute-predicate, dirty-saved-change-alias, relation-pick.

By spec file

File	Examples	Pass	Fail
`spec/models/search_parser_spec.rb`	78	0	78
`spec/models/story_spec.rb`	76	5	71
`spec/models/search_spec.rb`	33	0	33
`spec/models/comment_spec.rb`	24	3	21
`spec/models/user_spec.rb`	22	3	19
`spec/models/notification_spec.rb`	16	0	16
`spec/models/link_spec.rb`	14	0	14
`spec/models/domain_spec.rb`	20	7	13
`spec/models/vote_spec.rb`	13	0	13
`spec/models/invitation_request_spec.rb`	6	2	4
`spec/models/mod_note_spec.rb`	4	0	4
`spec/models/category_spec.rb`	6	3	3
`spec/models/invitation_spec.rb`	4	1	3
`spec/models/moderation_spec.rb`	3	0	3
`spec/models/tag_spec.rb`	10	7	3
`spec/models/message_spec.rb`	4	2	2
`spec/models/story_image_spec.rb`	2	0	2
`spec/models/comment_stat_spec.rb`	1	0	1
`spec/models/hat_spec.rb`	5	4	1
`spec/models/mod_mail_spec.rb`	1	0	1
`spec/models/story_text_spec.rb`	1	0	1
`spec/models/hat_request_spec.rb`	3	3	0
`spec/models/keystore_spec.rb`	4	4	0
`spec/models/mastodon_app_spec.rb`	3	3	0
`spec/models/username_spec.rb`	1	1	0

What the harness compensates for

The shim that boots the emit in place of Rails does these things on the emit's behalf. Listed because a pass rate achieved by propping up the subject is not a pass rate about the subject.

Compensation	Kind	Why
`require "net/http" in the shim`	emit gap	app/models/sponge.rb references Net:: constants but the emit never requires net/http.
`Numeric duration builders in the shim`	by design	Specs and factories are untranspiled plain Ruby; the emit path rewrites n.days at transpile time, so the shared runtime never reopens Numeric.

Errors outside examples

A spec file or hook that failed to run at all. These are not counted as failures above, so the denominator understates by however many examples they would have contributed.

An error occurred in an `after(:context)` hook.
Failure/Error: @comments.each(&:destroy!)

NoMethodError:
  undefined method 'each' for nil
# /home/rubys/lobsters/spec/models/search_spec.rb:60:in 'block (2 levels) in <top (required)>'

Emit diagnostics

From the transpile that produced the tree under test: 742 files, 0 errors, 2,364 warnings. Warnings are a modeling-debt ledger rather than a failure — the Ruby target emits through its gaps — so they are not comparable to the spec failures above.

Diagnostic	Count
`gradual_untyped`	1,314
`unresolved_type`	689
`unsupported`	168
`lower_residue`	80
`send_dispatch_failed`	63
`ivar_unresolved`	29
`missing_preload`	12
`incompatible_binop`	9

Artifact: summary.json — every number on this page comes from it.