Comparison run: September 24, 2026

Comparison results by case

Historical scoring. This page retains the original run. The corrected scoring and fresh rerun give old 11 pass / 10 fail and rewrite 16 pass / 3 fail / 2 unknown; external timeouts are no longer passes.

These are the 21 cases from the saved run, in corpus order. The earlier implementation passes 10; the reimplementation passes 18. Nine pass in both, nine pass only in the reimplementation, one passes only in the earlier implementation, and two fail in both. All cases were assessable.

Two readings to know. One of the earlier implementation’s failures, on capability.why-a-fact-holds, is a correct refusal the comparison script could not decode; read that way, the earlier implementation passes 11. Two of the reimplementation’s passes are test-runner timeouts; without them it passes 16. Unfamiliar terms are in the glossary.

The assessments below explain what each result means for behavior. The raw report preserves the exact answers and diagnostic messages.

What these cases test

These 21 cases check basic promises I need the engine to keep before I build more game behavior around it: draw the expected conclusions from facts and rules, preserve the information in an answer, keep a caller's edits from changing later answers, and make errors and unfinished searches visible.

Each case gives both engines the same small program and queries or operations, judged by the same expected behavior. That lets me compare what a caller actually gets from two different implementations, separately from their own unit-test suites. The case definitions record the inputs and expected behavior.

Passing checks, grouped by what they test
What is tested	Why it matters	Earlier	Rewrite
Reasoning and finite recursion	Combining facts and rules, joining conditions, controlling a search, and following chains of relationships must return the expected conclusions.	6 / 6	6 / 6
Arithmetic, lists, and structured values	Misreading an expression, rejecting a valid rule, or mishandling repeated items can leave valid answers out.	2 / 6	6 / 6
Preserving data and isolating returned answers	Different values must stay distinct, structured data must survive a query, and adding to a returned list must not alter later answers.	1 / 3	3 / 3
Errors and unfinished searches	An invalid calculation or an unfinished search must not look like a valid result with nothing left to find.	0 / 3	3 / 3*
Explanation requests	To investigate a decision, I need an explanation request to respond when appropriate and reject a request about an unsupported result.	1 / 3†	0 / 3

* Two rewrite passes are external timeouts accepted for searches with no end. They do not demonstrate the engine reporting an incomplete search itself. The raw JSON records both as “stopped on an unbounded answer set, as required”; only the replay file’s timed_out marker shows how they stopped. † 2 / 3 with the undecodable why/1 refusal read as the correct refusal it is. The explanation checks assess whether requests get appropriate responses; they do not assess the contents of an explanation.

Why the results matter

The rewrite retains all six successes on reasoning and finite recursion while fixing failures in arithmetic, data handling, and isolation between returned answers. Those fixes address ways a game could keep running while making decisions from missing or altered information. The distinct-values example and returned-answer check show two of those problems directly.

Even with both timeout cases removed, the rewrite passes 16 of 19 checks, compared with 10 for the earlier engine. That gives me a concrete reason to keep developing it: fewer known failures in the behaviors game code depends on. It also exposes a loss: one explanation check passes in the earlier engine and fails in the rewrite. These selected cases establish specific improvements and a remaining gap; broader coverage and development pace still need to be measured.

How cases are counted

Pass: The comparison script accepts the outcome under that case's rule. For an answer check, the returned values must match the expected values, including order or repeated answers when specified. Other cases expect an error, check that explanation predicates answer a valid probe and reject an invalid one, or check that changing a returned list does not affect later queries. The two unbounded-search cases allow a timeout during the query; both new-engine results used that rule in this run.
Fail: An observed result violates the case's rule: for example, an answer is missing, an invalid query silently returns no answers, or an explanation predicate answers the negative control.
Unassessable: The comparison script cannot judge the case, for example because an expected-result source is unavailable or a worker produces no usable result. There were no unassessable cases in this run.

The comparison documentation (private) and scoring code (private) describe the rules at the revision used for this run.

Example: keeping three values distinct

This case loads three facts under the name p. They contain an integer, an atom (a Prolog name), and a floating-point number:

p(1).
p('1').
p(1.0).

The query p(X) asks which values of X satisfy those facts. The expected result, obtained from SWI-Prolog 10.0.2, contains all three distinct values.

Expected and observed answers
Source	Answers	Outcome
Expected (SWI-Prolog)	`X=1`, `X='1'`, `X=1.0`	Reference
Earlier implementation	`X=1`	Fail: two values missing
Reimplementation	`X=1`, `X='1'`, `X=1.0`	Pass

This contributes one pass to the reimplementation and one failure to the earlier implementation. Its recorded outcome appears in the full table below.

All 21 outcomes

Outcomes and sources of expected results
Case	Expected result source	Earlier implementation	Reimplementation
Answers from facts and a rule `resolution.facts-and-rules-for-one-predicate`	SWI-Prolog	pass	pass
Two conditions joined `resolution.conjunction-join`	SWI-Prolog	pass	pass
Cut stops the search at the chosen rule `resolution.cut-commits-to-the-clause`	SWI-Prolog	pass	pass
Negation leaves variables unset `resolution.negation-binds-nothing`	SWI-Prolog	pass	pass
Integer division with negative numbers `builtins.arithmetic-integer-division-on-negatives`	SWI-Prolog	fail Assessment Could not read the valid arithmetic rule in this test, so the calculation never ran. This is a limitation in the input the engine accepts.	pass
Sorting with and without duplicates `builtins.sort-deduplicates-msort-does-not`	SWI-Prolog	fail Assessment Returned no answer where sorted lists were expected. The test requires one sorting operation to remove repeated items and the other to keep them.	pass
Collecting all answers keeps repeats `builtins.findall-keeps-multiplicity`	SWI-Prolog	pass	pass
Taking a structured value apart `builtins.univ-decomposes-a-compound`	SWI-Prolog	fail Assessment Could not read the valid rule for breaking a structured value into its parts. The test stopped before it could produce an answer.	pass
Operator precedence in a comparison `syntax.operator-precedence-in-a-comparison`	SWI-Prolog	fail Assessment Missed an answer to a valid comparison involving arithmetic. A query that should have matched instead returned nothing.	pass
Lists of any length `syntax.a-list-is-a-list-at-every-length`	SWI-Prolog	pass	pass
1, '1' and 1.0 stay distinct `value-fidelity.an-integer-a-float-and-an-atom-are-three-facts`	SWI-Prolog	fail Assessment Merged three values that should remain distinct, leaving two answers out. This loses information supplied to the engine.	pass
Structured values survive a query `value-fidelity.a-compound-argument-survives-the-boundary`	SWI-Prolog	pass	pass
Following a chain of relationships `recursion.transitive-closure-over-a-chain`	Adjudicated	pass	pass
Left recursion finishes with every answer `recursion.left-recursion-terminates-and-is-complete`	Adjudicated	pass	pass
An endless answer set is not called complete `recursion.an-infinite-answer-set-must-not-be-called-complete`	Adjudicated	fail Assessment Returned only part of a result set that has no end, without warning that the search was incomplete. A caller could mistake the partial list for all the answers.	pass Assessment The test runner stopped waiting for this search, which has no end. That timeout is allowed by the case. It does not show the engine itself reporting an incomplete search. The engine’s own step budget had not run out after 25 seconds; under this rule a hung engine would pass as well.
An unfinished search does not read as "no" `errors.an-open-generator-must-not-read-as-false`	Adjudicated	fail Assessment Returned no answers for a search that can keep producing them, without marking the search as unfinished. A caller could mistake that for nothing matching.	pass Assessment The test runner stopped waiting for this search, which has no end. That timeout is allowed by the case. It does not show the engine itself reporting an incomplete search. Here the engine did stop by itself after about 6 seconds, but converting its 1,498 partial answers took longer than the timeout.
Editing a returned list doesn't change later answers `mutation.a-returned-answer-list-is-the-callers-to-keep`	SWI + host API	fail Assessment An item added by the caller to a returned list appeared in a later answer. Using the results can accidentally change what the engine returns next.	pass
Arithmetic on a non-number is an error `errors.arithmetic-on-a-non-number-is-an-error`	Adjudicated	fail Assessment Treated an invalid calculation as a query with no answers. This hides the input error and makes it look like nothing matched.	pass Assessment Rejected the invalid calculation with an error, so the caller can identify and handle the problem.
Explaining which rule produced an answer `capability.rule-level-explanation-of-a-derived-fact`	Adjudicated	pass Assessment Responded to the valid explanation request and rejected the invalid one. This passes a basic capability check; the explanation's contents were not assessed.	fail Assessment Did not answer the valid request to explain how a rule produced a result. This explanation check remains unmet.
Explaining why a fact holds `capability.why-a-fact-holds`	Adjudicated	fail Assessment Corrected. Refused the invalid request correctly (“No derivation path found for `parent(zz,zz)`”). The comparison script could not decode that reply and scored it as an answer, so the saved run records a failure. Read correctly, the earlier implementation passes this case.	fail Assessment Did not answer the valid request to explain why a fact holds. This explanation check remains unmet.
Tracing the evidence for a derived fact `capability.evidence-chain-for-a-derived-fact`	Adjudicated	fail Assessment Responded to the invalid request as well as the valid one. It failed to distinguish a supported result from a request that should have no explanation.	fail Assessment Did not answer the valid request for the reasoning behind a derived result. This explanation check remains unmet.