Repository navigation
Fix generator, context manager, and guard reconstruction - #29
Conversation
```python
def select(values, test):
result = []
for value in values:
if not test(value) or value < 0.2:
result.append("dry")
else:
result.append("wet")
return result
```
The OR-join recognizer rejects any body back edge reaching before its
condition because that normally identifies a rotated while loop. Inside
an already open for loop, the ordinary iteration edge satisfies that
check too. Rejecting the join can lose the first operand's true path or
separate the else branch from its condition.
Exempt the nearest enclosing for loop's exact top, or its proved
EXTENDED_ARG prefix, from that rejection. Keep rejecting other backward
edges, including the edges belonging to an enclosing or nested while.
This preserves short-circuit evaluation and both outcomes of the guard,
including negated calls, truth tests, None comparisons, and long bodies.
```python
def parse_timestamp(value):
if isinstance(value, (int, float)) and not isinstance(value, bool):
try:
stamp = datetime.fromtimestamp(value, timezone.utc)
except (OverflowError, OSError, ValueError):
return
return stamp
if isinstance(value, str):
try:
parsed = datetime.fromisoformat(value.replace("Z", "+00:00"))
except ValueError:
return
return parsed
```
When inert instructions appear before RETURN_VALUE, the return-tail
matcher counts that padding as part of a calculated-value sequence.
A bare None return can then look like a duplicated value-return tail,
causing the following string-conversion branch to disappear.
Ignore NOP, NOT_TAKEN, and CACHE when classifying the tail, and require
an actual return after the None load. Keep the complete instruction
sequence for genuine value-tail matching.
Recognize both LOAD_CONST None and Python 3.15's LOAD_COMMON_CONSTANT
None through the same predicate. Use it in the function-tail pair
matcher as well, so implicit None copies still delimit a real else
suite when the general value-tail matcher rejects them. Other common
constants retain their existing treatment.
```python
def select(value, make, fallback, default):
if value:
try:
return {"value": make(value)}
except Exception:
pass
return fallback.read().result or default.result
```
When padding separates RETURN_VALUE from its protected value construction,
pycdc replaces the calculated return with a bare return. This discards both
the value and its evaluation, including calls and exceptions. Calls,
attributes, tuples, and dictionaries can all reach this boundary.
Use the existing expression stack when it contains exactly one expression
and the already bounded return-only tail contains RETURN_VALUE. Consume
that expression once instead of starting an empty region walk. Leave a
real bare return and its preceding side effects unchanged.
Apply the check to the Python 3.11+ exception-table path. The affected
padded shape also occurs in 3.12 and 3.13, so restricting the check to 3.14
leaves valid earlier inputs broken. Python 3.11 remains a compatibility
control, and the current 3.15 encoding follows the same bounded rule.
```python
def parse_timestamp(value):
if isinstance(value, (int, float)) and not isinstance(value, bool):
try:
return datetime.fromtimestamp(value, timezone.utc)
except (OverflowError, OSError, ValueError):
return
if isinstance(value, str):
try:
parsed = datetime.fromisoformat(value.replace("Z", "+00:00"))
except ValueError:
return
return parsed.astimezone(timezone.utc)
```
With padding before RETURN_VALUE, pycdc can emit an empty try and move
fromtimestamp outside its handler. Values such as infinity then raise
instead of taking the intended bare return. Scanning farther for a return
can also pull the following string branch into the wrong scope.
Recover the live expression only for an except-only region whose pending
body and else are empty. Require exactly one expression on the stack,
an unconsumed RETURN_VALUE before the handler, no incoming jump from
outside the bounded tail, and no context-manager protocol. Record the
return as consumed and skip only through that instruction.
Use the Python 3.11+ exception-table path; the padded failure occurs on
3.12 through 3.15, while 3.11 remains compatible. Returns after the try
and in its else suite keep their original exception scope, and timezone
conversion after parsing remains outside the parsing handler.
```python
def select(values):
return [0.6 + 0.4 * (1.0 if 4 <= value <= 8 < 10 else 0.3)
for value in values]
```
The chained-comparison recognizer opens a statement-level if for this
expression. That loses the ternary's values and can replace an entire
dictionary comprehension with an empty list. The same problem occurs in
dictionary keys and values, including expressions with calls before the
comparison and at its endpoints.
Inside an inline comprehension, recognize the two value-building arms,
their forward merge, and the failed comparison's cleanup. Verify that
the stack still holds the exact middle operand retained by COPY, remove
that retained value once, and schedule a ternary value merge. Ordinary
comprehension filters keep their existing path.
This applies to the Python 3.12+ inline-comprehension implementation,
including 3.15. Conditional expressions date to 2.5; dictionary
comprehensions date to 3.0 and were backported to 2.7. Those earlier
versions use a different comprehension parser and are not changed here.
```python
def select(values):
return tuple((value, discarded, _)
for value, _, discarded, discarded in values)
```
The legacy generator simulator replaces optimized unpacking targets with
underscores. Here it emits `for value, _, _, _ in values` while leaving
`discarded` in the yielded expression, so the recovered function raises
NameError. The statement parser already reconstructs those targets, but
its generator route requires LOAD_FAST_AND_CLEAR, an unrelated marker
for an inlined comprehension inside the generator.
Remove that marker requirement for Python 3.12+ generator code objects.
Reuse the existing complete For/If/Yield extraction, restore the outer
iterable at the call site, and retain the legacy fallback when the
statement result is incomplete or contains unsupported statements. Keep
the existing rejection of arbitrary statements in a generator tree.
This also lets nested calls and multiple captured closure cells use the
same parser without requiring an inner inline comprehension. Generator
expressions date to Python 2.4; this change applies to the modern route
already established for 3.12 and later.
For the 3.12 layout, retain a multiline filter hint when its conditional
jumps to the same loop's shared back edge. The statement-derived clauses
must preserve this detail rather than changing already-correct bytecode
layout in generators such as `value for value in values\n if condition`.
Expanded calls now use their original direct spelling; the structural
expectations follow that recovered form.
```python
def select(rows):
return tuple(row[1] or 0 for row in rows if row[0])
```
A generator filter can appear in the statement tree as an inverted
`if condition: continue`, with the yield following it. Rejecting that
tree sends the element through a fallback that confuses its Boolean
value with another filter. Rows that should be rejected can produce
values, and `and` can return the wrong operand.
Recognize exactly one early Continue with no else as a rejection guard
for the current generator clause, and invert its condition. Continue to
reject guards outside a clause, after a yield, or mixed with extra
statements. The yielded expression stays in the statement parser.
When reconstructing a retained Boolean operand, look past NOP,
NOT_TAKEN, and CACHE before checking its conditional jump. Inert padding
must not hide the POP that consumes the retained operand.
Generator expressions and these Boolean combinations date to Python
2.4. The affected modern route is available from 3.12; 3.14 adds
NOT_TAKEN padding, and 3.15 uses the same reconstruction. Calls and truth
hooks in a None-filtered getter expression retain their evaluation order.
```python
def select(values, offset):
return collect(value + offset for value in values)
```
The main parser can recover this generator, but the bounded expression
simulator cannot build its function or instantiate it with the iterator.
A guard or return-value reconstruction that delegates this expression to
the simulator therefore fails even though the generator body is supported.
Model MAKE_FUNCTION only for a code object named <genexpr>. For closure
attributes, require a function and a tuple whose length matches its free
variables. Keep the iterator expression, recognize the generator's call
with zero explicit arguments, and extract its clauses from a complete
statement-parser result. Substitute the actual outer iterable before
returning the generator expression to the surrounding call.
Ordinary function creation, malformed closures, and incomplete generator
bodies remain rejected by this restricted simulator. GET_ITER also
requires a value on the stack.
Generator expressions date to Python 2.4. This implementation uses the
MAKE_FUNCTION/SET_FUNCTION_ATTRIBUTE protocol introduced in Python 3.13,
which is the earliest applicable encoding; it also handles 3.14 and 3.15.
```python
def select(rows):
return tuple(isinstance(row, dict)
and (change := row.get("change")) is not None
and change >= 0
for row in rows)
```
The generator statement tree contains a nonlocal declaration for change.
Rejecting that declaration discards the otherwise reconstructed Boolean
expression and falls back to a parser that loses its short-circuiting.
A non-dictionary row can then reach row.get(), raising an exception that
the original expression avoids. The outer cell must also retain the last
assigned value after the generator is consumed.
Pass the code object's actual free-variable names to the generator-body
extractor. Omit a nonlocal declaration only when every name belongs to
those enclosing cells; the assignment expression still retains its store.
Preserve the existing rejection of globals and unrelated declarations.
Apply the same scope check to ordinary generator reconstruction and to
the bounded expression simulator.
Assignment expressions start in Python 3.8. This repair applies to the
modern generator route from 3.12 onward; the simulator uses the same
check on its supported construction encodings.
```python
async def select(manager, work, record):
async with manager:
try:
await work("a")
finally:
record("cleanup")
await work("b")
return "tail"
```
The exception table splits this source region around await suspension
instructions and out-of-line CLEANUP_THROW trampolines. Treating those
trampolines as source handlers can empty the async-with body, move the
try/finally outside it, and place later awaited work inside the finally
suite. That changes cleanup order and exception suppression.
For the Python 3.14+ fragmented layout, remove suspension-cleanup targets
and entries that start at their trampolines from the source-region view.
Merge equal-handler, equal-depth, equal-lasti fragments only across the
bounded suspension protocol. Retain the original table for interpreter
cleanup processing. Recognize awaited exit sequences and intervening
protected fragments when rebuilding a context's region.
Keep cleanup-target filtering tied to that modern reconstruction path;
applying it independently to 3.12 or 3.13 changes otherwise-correct context
boundaries. Adjacent async contexts and an enclosing source handler keep
their own scope, including when __aexit__ suppresses a body exception.
Async functions and async context managers date to Python 3.5; this fix
addresses the fragmented exception-table encoding used in 3.14 and 3.15.
async def select(manager, work, record, flag):
async with manager:
if flag:
return "early"
try:
return await work("a")
finally:
record("cleanup")
The parser consumes the context-exit slot for the early-return branch,
then lacks that slot when reconstructing the remaining branch. It can
emit a call to None instead of retaining the awaited return and cleanup.
Keep exit accounting for branch-local copies while their context is
still live. Bound the finally collector at a proven context-return tail
and retain the carried value through that tail's actual return endpoint.
The protected-tail changes apply to Python 3.14 and later.
Python 3.15 also encodes None with LOAD_COMMON_CONSTANT. Use the shared
constant predicate in the return-tail matcher so both synchronous and
asynchronous context bodies keep their returned values. Remove the
corresponding known limitation.
Include native fixture checks and executable examples covering early,
computed and attribute returns, entry/body/cleanup/exit errors, actual
async suspension, and exception suppression. Earlier modern exit layouts
remain covered as controls.
def select(lock, rows, parse):
with lock:
result = []
for row in rows:
try:
result.append(parse(row))
except ValueError:
continue
return {"rows": result}
The loop's out-of-line exception fragments can keep the context open
beyond its computed-return tail. The existing late-return path expects
an already closed context statement, so it can leak exit protocol calls
or move the return outside the context. Moving that return also changes
the result when the context suppresses an exception.
When the exact exit/return protocol is proved and the top block is the
still-live context, fold the carried value into that block before
simulating cleanup slots. Preserve result evaluation inside the context
and exit before the actual function return.
Cover synchronous and asynchronous contexts returning a variable, a call
or a dictionary after handler loops. The exception-table regressions
occur on Python 3.12 and later; earlier layouts remain controls. Behavior
examples include rejected rows, cancellation, entry and exit failures,
and exception suppression.
async def select(lock, pending, release):
async with lock:
record("body")
if pending:
return False
if release:
return False
record("outside")
return True
Python 3.14 and later can place an early-return exit just past the
context's last protected fragment. The parser consumes its exit counter
too soon and exposes the later normal exit as an invalid call to None.
Recognize a bounded, otherwise pure exit/return protocol owned by an
open conditional arm, and keep the normal-exit slot available. Include
Python 3.15's LOAD_COMMON_CONSTANT encoding and extra null SEND slot.
Retain the existing 3.14 boundary; earlier superficially similar failures
use a different context-exit path.
When the exact three-None async exit intentionally lacks its modeled
callable, restore only the clean state from before that call. Existing
incomplete diagnostics remain set; other underflows stay diagnosed.
Cover constant, bare and variable early returns, consecutive guards,
ordinary body work, and user-authored awaits. Exercise suspension,
entry/body/exit failures, cancellation and exception suppression.
async def select(contexts, value):
async with contexts[0]:
async with contexts[1]:
async with contexts[2]:
return await value()
The return value is evaluated while every context remains open, then
carried across each exit before one function return. The parser matches
only a single exit group, so nested groups leak protocol calls or lose
the returned value.
Extend the return-tail matcher to report complete exit-group counts.
Accept another exact SWAP group only after the previous call result is
discarded, and never exceed the known context-exit slots. When the inner
context has closed, place the return in that inner body while retaining
the contiguous outer contexts and their suppression-path exit slots.
The multi-group SWAP layout applies from Python 3.11. Cover synchronous
and asynchronous nesting, calls, tuple and dictionary values, awaited
values, single-context controls and returns after a synchronous inner
context. Exercise entry and exit failures, cancellation, suppression at
each nesting level, and evaluation while the correct contexts are open.
async def select(owner):
try:
for row in owner.rows:
try:
await owner.arecord(row)
except ValueError:
owner.failed(row)
owner.finish()
except RuntimeError:
owner.fallback()
The bytecode can place an inner dispatch after the enclosing mainline
but before the outer dispatch. Treating that physical order as lexical
ownership moves the inner try out of the loop, processing only the last
row, or adopts the outer handler as one of the inner clauses.
Record an intervening live For as the owner when the nested try opens.
Use that ownership to avoid a flat enclosing rebuild, nested-handler
adoption, and premature clause truncation at POP_EXCEPT fragments
protected by the outer dispatch. Keep genuine nested-chain bounds.
Cover named, multiple, sibling and same-type clauses; synchronous and
async bodies with zero, one or two awaits; post-loop work; and nonloop
and outer-finally controls. Python 3.12–3.15 examples preserve every row,
outer fault scope, cancellation and failures raised by a handler itself.
async def select(owner):
async with owner:
async with owner.query("row0") as query:
first = await query.read()
async with owner.query("row1") as query:
last = await query.read()
value = owner.build(first, last)
return value
The parser recognizes synchronous context-exit calls but not their
awaited counterparts, so work after an async context can stay inside it.
Recognize GET_AWAITABLE 2 followed by a bounded SEND loop with its own
backward edge, END_SEND and POP_TOP. User-authored awaits use operand 0
and remain visible. Enable this path for Python 3.14 and later, retaining
the earlier exception-layout handling.
Keep carried and constant returns out of normal-exit consumption,
including padding around RETURN. Check exception-table ownership before
consuming an inner exit protected by an outer context, and preserve the
original protocol in displaced handler walks that have lost their
lexical Try and With blocks. These safeguards are part of the recognizer
so it never discards a return value or an enclosing context's exit slot.
Python 3.15 adds a null SEND slot and shared None constants. Recognize
both while scanning the header and the exit: otherwise the header scan
mistakes protocol setup for the body and assigns an inner context its
outer handler. Share the ownership check across both constant-loading
entry points.
Cover sequential and nested contexts, post-context work, returns within
the outer context, user awaits, branch returns, padded return protocols,
malformed suspension loops and displaced ownership. Behavior examples
exercise actual suspension, cancellation, suppression, and failures
while entering, reading, building and exiting.
async def select(owner):
accepted = False
for candidate in owner.candidates:
try:
for value in (candidate, "keeper"):
await owner.check(value)
accepted = bool(await owner.reconsider(candidate)) or accepted
except Incompatible as error:
details = "reviewable"
if owner.can_close:
try:
changed = bool(await owner.close(candidate))
details = "closed" if changed else "terminal"
except RuntimeError as inner:
await owner.log("close-failed", str(inner))
await owner.log("incompatible", details)
except ValueError as error:
await owner.log("invalid", str(error))
continue
except (OSError, asyncio.TimeoutError) as error:
await owner.log("transient", str(error))
continue
except Exception as error:
await owner.log("unavailable", str(error))
continue
await owner.log("finished", accepted)
return accepted
A nested try inside one handler body can bound that body's private walk
before the remaining clauses. The parser treats that bound as the end
of the whole dispatch and drops later typed handlers, allowing their
exceptions to escape instead of taking the corresponding branch.
Keep the private body bound, but follow an explicit mismatch edge when
it reaches a verified typed-pattern load, CHECK_EXC_MATCH and forward
false jump. Extend only the dispatch scan to that clause's own boundary.
Reject scans crossing cleanup, a new handler, a call or an unrelated
jump, and retain a fixed lookahead limit.
Cover synchronous and asynchronous handlers, named exceptions, tuple
patterns, attribute patterns from global and local namespaces, and
single-clause/no-private-try controls. Python 3.12–3.15 examples preserve
the later clauses and post-loop work, including cancellation and errors
raised inside handler bodies.
async def select(owner):
accepted = False
for candidate in owner.candidates:
try:
for value in (candidate, "keeper"):
await owner.check(value)
accepted = bool(await owner.reconsider(candidate)) or accepted
except Incompatible as error:
details = "reviewable"
if owner.can_close:
try:
changed = bool(await owner.close(candidate))
details = "closed" if changed else "terminal"
except RuntimeError as inner:
await owner.log("close-failed", str(inner))
await owner.log("incompatible", details)
except ValueError as error:
await owner.log("invalid", str(error))
continue
except (OSError, asyncio.TimeoutError) as error:
await owner.log("transient", str(error))
break
except Exception as error:
await owner.log("unavailable", str(error))
break
await owner.log("finished", accepted)
return accepted
After POP_EXCEPT and named-exception cleanup, a for-handler break drops
the live iterator before jumping to the loop exit. The parser stops at
that discard and loses the break, continuing with later candidates and
changing both the accumulated result and post-loop observations.
Consume iterator disposal only when the following jump targets a
registered For exit. Leave arbitrary body POP_TOPs alone. Python 3.15
uses two iterator slots and shared None constants during named cleanup;
recognize those encodings without widening the accepted jump target.
Cover synchronous and asynchronous handlers, named and unnamed tuple
and generic clauses, nested handler tries and direct controls. The
Python 3.12–3.15 examples retain handler logging before the break and
post-loop work exactly once, while preserving failures and cancellation.
def select(value):
if value is None or not 0.0 <= value <= 15.0:
return False
return True
def collect(rows):
result = []
for a, b, c, d in rows:
if a is None or b is None or c is None or d is None or d < 0.01:
continue
result.append(a + b + c + d)
return result
The Python 3.14–3.15 compiler shares rejection blocks among predicate
edges and jumps over them on success. The parser reads these as nested
positive conditions, allowing missing values through. A copied middle
operand in a negated comparison chain also needs to survive each link.
Prove the shared constant return or registered loop continue, simulate
predicate expressions and merge only comparison links with a carried
middle value. Preserve independent repeated calls and wrap the original
comparison in not: reversing inequalities changes NaN and rich-comparison
behavior. Resume at the proven success continuation.
Accept constant-pool, small-integer and shared-constant return encodings.
Require iterator/value cleanup before consuming a duplicate return, and
exclude the success continuation from rejection-target matching even
when it returns the same constant. This keeps a real following return
separate from compiler-generated copies.
Cover None in every loop operand, endpoint and NaN comparisons, longer
chains, side-effecting and failing calls, rich comparisons, and equal
rejection/success values. These are existing Python constructs; the new
reconstruction route targets the 3.14+ bytecode layout.
def select(value, build):
if not value or len(value) < 2:
return build("reject", amount=0)
return build("accept", amount=len(value))
def collect_result(value, rows, collect, scale):
if value is None or not 0.0 <= value <= 20.0:
return collect(scale * item for item in rows)
return collect(rows)
The guard recognizer requires a None-identity first test and a constant
return. With a constructed rejection value, missing inputs can instead
reach the success path. Accept arbitrary first predicates and a bounded
value-building region that leaves exactly one result before RETURN_VALUE.
Reject intervening stores, branches and exception cleanup so statement
bodies remain available to the ordinary parser.
Keep keyword names, constructor evaluation and lazy generator iteration
on the original selected path. Recognize duplicated short constructed
returns only when their complete instruction runs match, while retaining
the cleanup and success-boundary checks used for constant return copies.
Support dictionary construction in the bounded simulator with exact
key/value consumption and original pair order.
The Python 3.14–3.15 examples cover truth hooks, comparison predicates,
factories, lists, dictionaries and captured generator outcomes. Controls
retain explicit assignments and logging before returns, along with
exceptions from predicates, constructors and iteration. The reconstruction
route remains gated to 3.14+; these source constructs predate that layout.
def select(value, read, invalid, label, width, make_error, make_cause):
if value is None or not 0 <= read(value) <= 10 or invalid(value):
raise make_error(f"{label!r}: {value!s:>{width}}") from make_cause(value)
return value
The shared guard recognizer accepts return and continue outcomes, but a
validation raise can still lose its rejection path. A successful chained
comparison also jumps over its failed-link cleanup before later predicates;
reading that cleanup as part of the success path loses the compound guard.
Recognize RAISE_VARARGS with one exception operand or an exception and
explicit cause. Require exactly that operand count, and keep bare reraises,
intervening statements, cleanup and branches outside this expression route.
Skip a comparison cleanup trampoline only after proving its complete shape
and its jump back to the same rejection block.
Extend bounded expression simulation with conversion, simple/spec formatting
and string assembly, retaining exact stack consumption. Use the split
formatting protocol from Python 3.13 onward; the shared-guard reconstruction
itself targets Python 3.14–3.15. F-string source syntax dates to Python 3.6,
explicit causes to 3.0, and from None suppression to 3.3.
Cover ordinary raises, keyword constructors, from None, computed causes,
!s/!r/!a conversions and dynamic format specifications. Preserve one interior
read, short-circuiting, NaN/rich comparisons, formatting and constructor
order, existing exception context and suppression metadata. Reject malformed
operand counts and incomplete formatting stacks without inventing values.
Remove redundant blank lines at the ends of the newly added fixture sources and their corpus copies. Regenerate the checked-hash pyc headers with each fixture's original Python runtime so they still identify the corresponding source exactly. All 31 regenerated fixture pairs retain byte-for-byte identical marshal payloads after the 16-byte pyc header. This changes source-file formatting and source hashes without changing the compiled examples or decompiler.
|
Verified against current
Per-module regressions: none. Across the original 528 corpus modules the verdicts are 527 identical + 1 improved (3.3.7 csv AST-PASS → PASS). Across the 349 realworld modules whose output changed, bytecode-signature comparison against the original pycs shows 0 regressions and 50 files moving to signature-identical. 1564 mutated/truncated pycs: no panics or crashes (same as main). Code review: additions are localized to Merging as-is. Thanks for keeping the commits focused and rebasing onto the current main — that made this straightforward. |
- stdlib corpus 622/622 semantic (499 signature-exact, 80.2%) - behavior matrix 754/754 on 2.6-3.15 (with and without --cure) - realworld: 6411/6411 recompile, 2502 AST-identical (was 2493), serial 5.60 s / parallel 2.17 s - CHANGELOG gains an Unreleased section describing the reconstruction fixes
- cure 0.4.2 fixes the `del` propagation found while integrating 0.4.1, so the "deletes a local name" skip is gone: all 6411 real-world modules are simplified again (-9.71% lines) with no behaviour change - TO_BOOL coercion keeps its bool when the value escapes (3.12+) - docs: refreshed corpus/behavior/roundtrip/bench numbers, v0.8.1 section with Fixes / Changes / Dependencies / Verification
I noticed my previous PRs were staged into a branch before merging, so I staged a branch directly this time, with an additional batch of fixes for undecompilable syntaxes.
I kept each commit focused on one change and independently passing the Rust tests and all three verification tools (corpus, behavior, strict round-trip), using the existing baseline: no new failures, and all new cases pass. The stack can be rearranged as needed, respecting dependencies.
Commit breakdown (22 commits)
c2c5152forloops51a94be71907208bffea1tryregion9fe8e9428fbed621d8f6702c412ef08ae3aff3dab3async withexception regions across suspension cleanup1208ec6finallycleanupcd53c9bcb691e1c733a61ef583fad1c7d880bba2c1try8f53b27breakafter iterator disposal05db4017c7a3fd3c77138b2d655d