diff --git a/README.md b/README.md
index 601367e..5432a30 100644
--- a/README.md
+++ b/README.md
@@ -83,6 +83,14 @@ via repeatable `--authorized-origin https://approved.example:8443`. Page and
robots redirects outside that scope stop before any destination request and
report `fetch_status=out_of_scope`; robots permission is not authorization.
+The CLI also enforces `--target-timeout 120`: a monotonic budget per target shared
+by robots, redirects, pacing and HTTP acquisition. `--timeout 15` remains the
+additional socket inactivity limit. DNS, connection, TLS, response headers and
+slow-trickling bodies run in one disposable request process at a time; on deadline
+exhaustion it is killed and reaped before the next target. A failed target records
+`fetch_status=deadline_exceeded` and an unknown assessment, rather than a negative
+finding. Process startup and cancellation have small OS scheduling overhead.
+
## Agent skill
The [php2shell-check skill](skills/php2shell-check/SKILL.md) guides an agent
diff --git a/docs/REFERENCE.md b/docs/REFERENCE.md
index dacee89..62aba4a 100644
--- a/docs/REFERENCE.md
+++ b/docs/REFERENCE.md
@@ -137,6 +137,7 @@ are rejected. Input is a text list, **not a CSV or an Excel workbook**.
| `--brand NAME` | `CompleteTech LLC` | Organization name shown on the HTML report. |
| `--delay SECONDS` | `1` | Positive minimum time between requests to an origin. Robots rules may increase it. |
| `--timeout SECONDS` | `15` | Positive per-socket timeout, not a total deadline for a website. |
+| `--target-timeout SECONDS` | `120` | Positive monotonic budget shared by all work for each target. |
| `--authorized-origin ORIGIN` | None | Additional authorized redirect origin; repeat for each approved scheme/host/port. No paths, queries or fragments. |
| `--contact VALUE` | Empty | Public contact URL/email included in the User-Agent. |
| `--help` | — | Show command help. |
@@ -167,6 +168,7 @@ embedded assets, or stale/spoofed metadata can resemble WordPress.
- `ok`: the final page returned HTTP 2xx, even if its content was inconclusive.
- `timeout`: a recognized socket timeout occurred.
+- `deadline_exceeded`: the target's total budget was exhausted; assessment remains unknown and the next input can proceed.
- `dns_error`: a recognized DNS-resolution error occurred.
- `out_of_scope`: a page or robots redirect was stopped before requesting an unauthorized origin; assessment remains unknown.
- `blocked`: other cases preventing a successful fetch, including robots rules,
@@ -271,11 +273,35 @@ upload destination; avoid committing real target lists or scan results.
## Python use
+### Total target deadline
+
+The CLI defaults to `--target-timeout 120`. One monotonic budget covers robots,
+redirects, required pacing, DNS, connection/TLS, response headers/body and local
+assessment. The socket inactivity timeout is an additional bound, never a substitute
+for the total budget. Each HTTP acquisition uses one disposable child process;
+the parent keeps robots caches and request spacing. Expired acquisition is killed
+and reaped before continuing. Windows, macOS and Linux use the same process boundary,
+including for resolver calls that Python cannot cancel in a thread. No child is
+left running intentionally. Process creation/OS scheduling and cancellation have
+small overhead; reaping allows at most two further seconds, then aborts the batch
+rather than launching another request if the OS cannot reap the worker.
+
+Deadline exhaustion produces an error row with `fetch_status=deadline_exceeded`
+and `php2shell_status=unknown`. Other targets get fresh budgets. A completed batch
+can therefore contain incomplete assessments. There are no retries or authorization
+exceptions. Required pacing that cannot fit in the remaining budget fails immediately.
+
+Python callers opt into this same boundary with `Checker(target_timeout=120, ...)`.
+The low-level `Checker` default remains `target_timeout=None` for integrations with
+their own process supervisors; a direct unbounded `request()` is not a total-work
+guarantee. Custom in-process opener changes are not transferred to bounded child
+requests. The CLI's `--recover` stays offline and launches no HTTP worker.
+
```python
from dataclasses import asdict
from wordpress_check import Checker
-checker = Checker(timeout=15, delay=1, contact="https://your-domain.example/contact")
+checker = Checker(timeout=15, target_timeout=120, delay=1, contact="https://your-domain.example/contact")
result = checker.check("https://example.com/")
print(asdict(result))
```
diff --git a/skills/php2shell-check/SKILL.md b/skills/php2shell-check/SKILL.md
index eba63cb..7f61ac4 100644
--- a/skills/php2shell-check/SKILL.md
+++ b/skills/php2shell-check/SKILL.md
@@ -87,7 +87,14 @@ evidence and timestamps must be preserved; HTML does not expose every field.
`--delay` and `--timeout` must be positive finite numbers. Delay defaults to one
second per origin and can be increased by robots rules. Timeout defaults to 15
-seconds **per socket**, not a total run deadline. Add `--contact` only for a
+seconds **per socket**. `--target-timeout` defaults to 120 positive finite seconds
+per target, shared by robots, redirects, pacing, DNS/connect/TLS, headers and body.
+A disposable HTTP acquisition process is killed and reaped on expiry, so stalled
+DNS and trickling responses cannot hold the next target indefinitely on Windows,
+macOS or Linux. Allow small process-startup/cancellation scheduling overhead.
+`fetch_status=deadline_exceeded` means incomplete/unknown, never a negative finding.
+The batch continues after that result; completed-input counts do not imply every
+assessment succeeded. Add `--contact` only for a
public contact value the user intends to disclose in requests.
Inspect each result even if the process exits with `0`: that means the run
diff --git a/test_target_deadline.py b/test_target_deadline.py
new file mode 100644
index 0000000..ab67b35
--- /dev/null
+++ b/test_target_deadline.py
@@ -0,0 +1,170 @@
+import contextlib
+import io
+import json
+import subprocess
+import sys
+import threading
+import time
+import unittest
+from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
+from pathlib import Path
+from unittest.mock import patch
+
+import wordpress_check as checker
+
+
+class TargetDeadlineTests(unittest.TestCase):
+ @classmethod
+ def setUpClass(cls):
+ cls.requests = []
+ class Handler(BaseHTTPRequestHandler):
+ def do_GET(self):
+ cls.requests.append(self.path)
+ try:
+ if self.path == '/robots.txt':
+ self.send_response(200)
+ self.end_headers()
+ self.wfile.write(b'User-agent: *\nDisallow: /private\n')
+ elif self.path == '/slow':
+ self.send_response(200)
+ self.end_headers()
+ for _ in range(40):
+ self.wfile.write(b'x')
+ self.wfile.flush()
+ time.sleep(.04)
+ elif self.path == '/idle':
+ time.sleep(.3)
+ self.send_response(200)
+ self.end_headers()
+ elif self.path == '/unicode':
+ self.send_response(200)
+ self.send_header('Content-Type', 'text/html; charset=utf-8')
+ self.end_headers()
+ self.wfile.write('
\u65e5\u672c\u8a9e \u2603'.encode('utf-8'))
+ elif self.path.startswith('/redirect'):
+ time.sleep(.12)
+ self.send_response(302)
+ self.send_header('Location', '/redirect' + str(int(self.path[9:]) + 1))
+ self.end_headers()
+ else:
+ self.send_response(200)
+ self.end_headers()
+ self.wfile.write(b'synthetic')
+ except (BrokenPipeError, ConnectionResetError, ConnectionAbortedError):
+ pass
+ def log_message(self, *_):
+ pass
+ cls.server = ThreadingHTTPServer(('127.0.0.1', 0), Handler)
+ cls.thread = threading.Thread(target=cls.server.serve_forever, daemon=True)
+ cls.thread.start()
+ cls.url = f'http://127.0.0.1:{cls.server.server_port}'
+
+ @classmethod
+ def tearDownClass(cls):
+ cls.server.shutdown()
+ cls.server.server_close()
+ cls.thread.join()
+
+ def setUp(self):
+ self.requests.clear()
+
+ def assert_incomplete(self, result):
+ self.assertEqual(result.status, 'error')
+ self.assertEqual(result.fetch_status, 'deadline_exceeded')
+ self.assertEqual(result.php2shell_status, 'unknown')
+ self.assertEqual(result.php2shell_exploitability, 'not_tested')
+
+ def test_slow_trickle_is_cancelled_and_next_target_progresses(self):
+ client = checker.Checker(timeout=.2, target_timeout=.65, delay=0)
+ start = time.monotonic()
+ self.assert_incomplete(client.check(self.url + '/slow'))
+ self.assertLess(time.monotonic() - start, 1.5)
+ # Cache and state survive cancellation, but the target gets a fresh budget.
+ result = client.check(self.url + '/fast')
+ self.assertEqual(result.status, 'no_evidence')
+ self.assertEqual(self.requests.count('/robots.txt'), 1)
+
+ def test_redirect_chain_shares_budget(self):
+ client = checker.Checker(timeout=1, target_timeout=.65, delay=0)
+ start = time.monotonic()
+ self.assert_incomplete(client.check(self.url + '/redirect0'))
+ self.assertLess(time.monotonic() - start, 1.5)
+ self.assertLess(len([p for p in self.requests if p.startswith('/redirect')]), 6)
+
+ def test_pacing_does_not_sleep_past_budget_or_fetch_page(self):
+ client = checker.Checker(timeout=1, target_timeout=.5, delay=1)
+ start = time.monotonic()
+ self.assert_incomplete(client.check(self.url + '/fast'))
+ self.assertLess(time.monotonic() - start, 1)
+ self.assertEqual(self.requests, ['/robots.txt'])
+
+ def test_dns_and_connect_stalls_are_killed_without_surviving_workers(self):
+ real_popen = subprocess.Popen
+ for operation in ('dns', 'connect'):
+ with self.subTest(operation=operation):
+ children = []
+ def spawn(command, **kwargs):
+ script = "import runpy,sys,time;m=runpy.run_path(sys.argv[1]);"
+ if operation == 'dns':
+ script += "m['socket'].getaddrinfo=lambda *a,**k:time.sleep(20);"
+ else:
+ script += "m['socket'].socket.connect=lambda *a,**k:time.sleep(20);"
+ script += "m['_request_worker']()"
+ child = real_popen([sys.executable, '-I', '-c', script, str(Path(checker.__file__).resolve())], **kwargs)
+ children.append(child)
+ return child
+ start = time.monotonic()
+ with patch.object(checker.subprocess, 'Popen', side_effect=spawn):
+ result = checker.Checker(timeout=.1, target_timeout=.4, delay=0).check(self.url)
+ self.assert_incomplete(result)
+ self.assertLess(time.monotonic() - start, 1.4)
+ self.assertEqual(len(children), 1)
+ self.assertIsNotNone(children[0].poll())
+
+ def test_cli_deadline_keeps_result_and_continues_batch(self):
+ out = io.StringIO()
+ with contextlib.redirect_stdout(out), contextlib.redirect_stderr(io.StringIO()):
+ code = checker.main([self.url + '/slow', self.url + '/fast', '--format', 'json',
+ '--timeout', '.2', '--target-timeout', '.65', '--delay', '.001'])
+ report = json.loads(out.getvalue())
+ self.assertEqual(code, 0)
+ self.assertTrue(report['complete']) # All inputs handled; first assessment failed.
+ self.assertEqual(report['results'][0]['fetch_status'], 'deadline_exceeded')
+ self.assertEqual(report['results'][0]['php2shell_status'], 'unknown')
+ self.assertEqual(report['results'][1]['status'], 'no_evidence')
+
+ def test_robots_scope_and_socket_timeout_remain_active(self):
+ client = checker.Checker(timeout=.05, target_timeout=2, delay=0)
+ result = client.check(self.url + '/private')
+ self.assertEqual(result.status, 'blocked')
+ self.assertNotIn('/private', self.requests)
+ # An inactivity timeout remains distinct from total deadline exhaustion.
+ with patch.object(client, '_fetch', side_effect=AssertionError('parent HTTP forbidden')):
+ self.assertEqual(client.check(self.url + '/fast').status, 'no_evidence')
+ self.assertEqual(client.check(self.url + '/idle').fetch_status, 'timeout')
+
+ def test_expiry_during_parsing_never_publishes_a_positive_assessment(self):
+ client = checker.Checker(target_timeout=1, delay=0)
+ client.request = lambda *args: checker.Response(200, {}, '', False)
+ client.robots[self.url] = (None, 0, 'synthetic robots')
+ def late_parse(response, result):
+ result.php_version = '8.3.36'
+ client._deadline = 0
+ with patch.object(checker, 'inspect_page', side_effect=late_parse):
+ self.assert_incomplete(client.check(self.url))
+
+ def test_invalid_total_budgets_are_rejected(self):
+ for value in (0, -1, float('nan'), float('inf')):
+ with self.assertRaises(ValueError):
+ checker.Checker(target_timeout=value)
+ with contextlib.redirect_stderr(io.StringIO()), self.assertRaises(SystemExit):
+ checker.main(['example.test', '--target-timeout', '0'])
+
+ def test_unicode_response_survives_worker_boundary(self):
+ result = checker.Checker(timeout=1, target_timeout=2, delay=0).check(self.url + '/unicode')
+ self.assertEqual(result.status, 'no_evidence')
+ self.assertEqual(result.title, '\u65e5\u672c\u8a9e \u2603')
+
+
+if __name__ == '__main__':
+ unittest.main()
diff --git a/wordpress_check.py b/wordpress_check.py
index 501b1be..6606b14 100644
--- a/wordpress_check.py
+++ b/wordpress_check.py
@@ -18,6 +18,7 @@
import mimetypes
import re
import socket
+import subprocess
import sys
import time
from dataclasses import asdict, dataclass, field
@@ -132,6 +133,10 @@ class OutOfScope(CheckStopped):
"""A redirect destination lacks explicit origin authorization."""
+class TargetDeadline(CheckStopped):
+ """The shared monotonic target budget expired."""
+
+
class NoRedirect(HTTPRedirectHandler):
def redirect_request(self, req, fp, code, msg, headers, newurl):
return None
@@ -389,10 +394,14 @@ def inspect_page(response: Response, result: Result) -> None:
class Checker:
- def __init__(self, *, timeout=15.0, delay=1.0, contact="", max_crawl_delay=60.0, authorized_origins=()):
+ def __init__(self, *, timeout=15.0, delay=1.0, contact="", max_crawl_delay=60.0, authorized_origins=(), target_timeout=None):
self.timeout = timeout
self.delay = delay
self.max_crawl_delay = max_crawl_delay
+ if target_timeout is not None and (not math.isfinite(target_timeout) or target_timeout <= 0):
+ raise ValueError("target_timeout must be positive and finite")
+ self.target_timeout = target_timeout
+ self._deadline = None
self.authorized_origins = {origin_key(authorized_origin(value)) for value in authorized_origins}
self.user_agent = f"{BOT}/1.0" + (f" (+{contact})" if contact else "")
self.opener = build_opener(NoRedirect())
@@ -400,11 +409,73 @@ def __init__(self, *, timeout=15.0, delay=1.0, contact="", max_crawl_delay=60.0,
self.robots: dict[str, tuple[RobotsPolicy | None, float, str]] = {}
def request(self, url: str, limit: int, delay: float | None = None) -> Response:
+ own_deadline = self._deadline is None and self.target_timeout is not None
+ if own_deadline:
+ self._deadline = time.monotonic() + self.target_timeout
+ try:
+ return self._request(url, limit, delay)
+ finally:
+ if own_deadline:
+ self._deadline = None
+
+ def _remaining(self):
+ if self._deadline is None:
+ return None
+ remaining = self._deadline - time.monotonic()
+ if remaining <= 0:
+ raise TargetDeadline("Total target deadline exceeded; assessment incomplete.")
+ return remaining
+
+ def _request(self, url, limit, delay):
site = origin(url)
wait = max(self.delay, delay or 0) - (time.monotonic() - self.last_request.get(site, -math.inf))
+ remaining = self._remaining()
if wait > 0:
+ if remaining is not None and wait >= remaining:
+ raise TargetDeadline("Total target deadline cannot accommodate required pacing; assessment incomplete.")
time.sleep(wait)
self.last_request[site] = time.monotonic()
+ remaining = self._remaining()
+ if remaining is None:
+ return self._fetch(url, limit)
+ # A disposable process bounds even resolver calls and trickling headers/body.
+ # One child is active at a time; parent retains robots policy and pacing.
+ command = [sys.executable, "-I", "-c", "import runpy,sys; runpy.run_path(sys.argv[1])['_request_worker']()", str(Path(__file__).resolve())]
+ child = subprocess.Popen(command, stdin=subprocess.PIPE, stdout=subprocess.PIPE, stderr=subprocess.PIPE,
+ creationflags=getattr(subprocess, "CREATE_NO_WINDOW", 0))
+ try:
+ payload = json.dumps({"url": url, "limit": limit, "timeout": min(self.timeout, remaining), "user_agent": self.user_agent})
+ output, _ = child.communicate(payload.encode("utf-8"), timeout=self._remaining())
+ except subprocess.TimeoutExpired:
+ raise TargetDeadline("Total target deadline exceeded during HTTP acquisition; assessment incomplete.") from None
+ finally:
+ if child.poll() is None:
+ child.kill()
+ # Reap the only child before another target can start. No resolver thread survives.
+ try:
+ child.wait(timeout=2)
+ except subprocess.TimeoutExpired:
+ # An OS-level cancellation failure must abort, never launch another worker.
+ raise RuntimeError("Unable to reap HTTP worker after cancellation; batch aborted") from None
+ for pipe in (child.stdin, child.stdout, child.stderr):
+ if pipe is not None:
+ pipe.close()
+ self.last_request[site] = time.monotonic()
+ self._remaining()
+ if child.returncode != 0:
+ raise OSError("HTTP acquisition worker failed")
+ message = json.loads(output)
+ if "error" in message:
+ kind = message.get("kind")
+ error = message["error"]
+ if kind == "timeout":
+ raise TimeoutError(error)
+ if kind == "dns":
+ raise socket.gaierror(error)
+ raise OSError(error)
+ return Response(**message["response"])
+
+ def _fetch(self, url, limit):
req = Request(url, headers={"User-Agent": self.user_agent, "Accept": "text/html,text/plain;q=0.9,*/*;q=0.1", "Accept-Encoding": "identity"})
try:
stream = self.opener.open(req, timeout=self.timeout)
@@ -467,11 +538,13 @@ def policy(self, url: str, authorized=None) -> float:
def check(self, value: str) -> Result:
result = Result(input_url=value)
+ self._deadline = time.monotonic() + self.target_timeout if self.target_timeout is not None else None
try:
url = normalize_url(value)
authorized = self.authorized_origins | {origin_key(url)}
seen = set()
for _ in range(6):
+ self._remaining()
result.final_url = url
if url in seen:
raise CheckStopped("Page redirect loop; stopped.")
@@ -482,6 +555,7 @@ def check(self, value: str) -> Result:
if response.status not in REDIRECTS:
result.notes.append(self.robots[origin(url)][2])
inspect_page(response, result)
+ self._remaining()
return result
location = response.headers.get("location")
if not location:
@@ -495,6 +569,9 @@ def check(self, value: str) -> Result:
raise CheckStopped("Too many page redirects; stopped.")
except CheckStopped as exc:
result.status = "blocked"
+ if isinstance(exc, TargetDeadline):
+ result.status = "error"
+ result.fetch_status = "deadline_exceeded"
if isinstance(exc, OutOfScope):
result.fetch_status = "out_of_scope"
result.notes.append(str(exc))
@@ -507,10 +584,31 @@ def check(self, value: str) -> Result:
result.fetch_status = "dns_error"
result.notes.append(str(exc))
finally:
- annotate_php2shell(result)
+ self._deadline = None
+ if result.fetch_status == "deadline_exceeded":
+ # Evidence parsed just as the budget expired is not a completed assessment.
+ for name, value in assess_php2shell().items():
+ setattr(result, name, value)
+ result.php2shell_notes.append("Total target deadline exceeded; assessment incomplete.")
+ else:
+ annotate_php2shell(result)
return result
+def _request_worker():
+ """Private bounded HTTP acquisition; policy decisions stay in the parent."""
+ payload = json.loads(sys.stdin.buffer.read().decode("utf-8"))
+ checker = Checker(timeout=payload["timeout"], delay=0)
+ checker.user_agent = payload["user_agent"]
+ try:
+ message = {"response": asdict(checker._fetch(payload["url"], payload["limit"]))}
+ except (ValueError, URLError, OSError, HTTPException, UnicodeError) as exc:
+ cause = exc.reason if isinstance(exc, URLError) else exc
+ kind = "timeout" if isinstance(cause, (TimeoutError, socket.timeout)) else "dns" if isinstance(cause, socket.gaierror) else "error"
+ message = {"error": str(exc)[:4096], "kind": kind}
+ sys.stdout.buffer.write(json.dumps(message, ensure_ascii=False).encode("utf-8"))
+
+
def positive_float(value):
number = float(value)
if not math.isfinite(number) or number <= 0:
@@ -742,6 +840,7 @@ def main(argv=None) -> int:
parser.add_argument("--format", choices=("csv", "json", "html"), default="csv", help="csv, json, or a branded html report")
parser.add_argument("--delay", type=positive_float, default=1.0, help="Minimum seconds between requests to an origin (default: 1)")
parser.add_argument("--timeout", type=positive_float, default=15.0, help="Socket timeout per request (default: 15 seconds)")
+ parser.add_argument("--target-timeout", type=positive_float, default=120.0, help="Total monotonic seconds per target, including DNS, HTTP, redirects and pacing (default: 120)")
parser.add_argument("--authorized-origin", action="append", default=[], type=authorized_origin, help="Additional authorized redirect origin (scheme, host and optional port); repeatable")
parser.add_argument("--contact", default="", help="Public contact URL/email to include in the honest User-Agent")
parser.add_argument("--logo", type=Path, default=DEFAULT_LOGO, help="Logo image for the html report (default: assets/logo.png)")
@@ -780,7 +879,7 @@ def main(argv=None) -> int:
if checkpoint:
# Exclusive creation protects a previous interrupted run from accidental reuse.
publish_report(checkpoint, checkpoint_json(report))
- checker = Checker(timeout=args.timeout, delay=args.delay, contact=args.contact, authorized_origins=args.authorized_origin)
+ checker = Checker(timeout=args.timeout, delay=args.delay, contact=args.contact, authorized_origins=args.authorized_origin, target_timeout=args.target_timeout)
try:
for index, url in enumerate(urls):
result = checker.check(url)