Repository navigation
Judge a 4xx or 5xx page by its status, without asking Jev - #19
Merged
Merged
Conversation
Of the 1,781 pages that answered 4xx or 5xx in the classifier's crawl, Jev found none worth reading: every one fired a check, and the seven that fired none on both votes were blocks it scored near 0.6. Asking it is a round trip for an answer the status already gave. The session keeps the status the last navigation or action reported, and judge() returns from it before calling classify: 401 names a login wall, 402 a paywall, 403 and 429 a block, anything else an error page. Scores and answers stay None, so a caller can tell a verdict the model gave from one it did not. A 2xx page still goes to Jev, since bot blocks often arrive as 200.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
judge()now returns a verdict from the HTTP status alone when the page answered 4xx or 5xx, and skips theclassifycall.Why it is safe. In the local classifier's crawl, 1,781 pages answered 4xx or 5xx. Jev fired at least one check on 1,774 of them. The other seven were blocks too, scored around 0.6: Yahoo Japan's EEA notice, zeit.de's bot "Legal Notice", an ipapi rate-limit page and a maintenance page. None of them was a page worth reading, so the call was a round trip for an answer the status had already given.
What the verdict looks like.
is_login_wall, 402 →is_paywall, 403 and 429 →is_blocked, any other 4xx/5xx →is_error_page.verdict.http_statusholds the status. Every score and answer isNone, so a caller can tell a verdict the model gave from one it did not.Why 2xx still goes to Jev. Bot blocks often come back as 200; the first page of that same crawl is an Incapsula block page served with 200.
Not included:
isAccessibleForFree. Treating a page that declaresisAccessibleForFree: falseas a paywall looked like the next shortcut. On 86 hand-labelled pages, 81 of those declaring it were paywalls. But metered sites declare it on articles they still show in full: the Boston Globe did on four of four fresh articles, each 5–8k characters of visible text, and Jev rightly scored them at about 0.1. Skipping on the flag would drop readable pages, so it stays out.