Repository navigation
Lobbying scraper: Handle amendments where the period row links back to the amendment - #2316
Merged
Merged
Conversation
On some registration pages the portal links an amended period's row to the amendment itself instead of to the original filing, so the same disclosure is linked twice in a row. parse_summary then marked the amendment as superseded by itself: the migration's write phase skipped it (superseded_without_current, which blocks cleanup), and the backfill and weekly runs wrote it and then deleted its filings. A repeated link to a version is now skipped, so the amendment is the current version and supersedes nothing on the page. As a backstop, the backfill and weekly runs no longer delete the filings of a version a page lists as both superseded and current; they keep them and report the page. Verified: pytest in lobbying-scraper (197 passed), including the repeated-link layout and the kept filings in both modes. Parsing all 33,762 archived registration pages in the local archive before and after the change differs on one page only, the one with this layout, which now has no self-superseded versions. The local emulator's filings for that page, which the local rebuild had deleted, were rewritten from the archive and the stats rebuilt without errors. Not verified: the dev write phase reporting superseded_without_current 0; a weekly or backfill run against dev. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
dasanchez
changed the title
dasanchez
marked this pull request as ready for review
October 27, 2026 23:22
dasanchez
requested review from
Mephistic,
alexjball,
kiminkim724,
mertbagt,
mvictor55,
nesanders,
sashamaryl and
timblais
as code owners
October 27, 2026 23:22
Mephistic
approved these changes
Oct 27, 2026
This branch was successfully deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
On some registration pages, the Secretary of the Commonwealth's portal links an amended period's row to the amendment itself rather than to the original filing, so the scraper read the amendment as superseded by itself. That blocks the dev backfill's cleanup, and in the weekly and backfill runs it would delete the amendment's filings right after writing them. The parser now treats a repeated link as the same version, so the amendment stays current. As a backstop, the runs keep the filings of any version a page lists as both current and superseded, and report that page instead.
What's in it
🤖 Generated with Claude Code