diff --git a/README.md b/README.md index 601367e..5432a30 100644 --- a/README.md +++ b/README.md @@ -83,6 +83,14 @@ via repeatable `--authorized-origin https://approved.example:8443`. Page and robots redirects outside that scope stop before any destination request and report `fetch_status=out_of_scope`; robots permission is not authorization. +The CLI also enforces `--target-timeout 120`: a monotonic budget per target shared +by robots, redirects, pacing and HTTP acquisition. `--timeout 15` remains the +additional socket inactivity limit. DNS, connection, TLS, response headers and +slow-trickling bodies run in one disposable request process at a time; on deadline +exhaustion it is killed and reaped before the next target. A failed target records +`fetch_status=deadline_exceeded` and an unknown assessment, rather than a negative +finding. Process startup and cancellation have small OS scheduling overhead. + ## Agent skill The [php2shell-check skill](skills/php2shell-check/SKILL.md) guides an agent diff --git a/docs/REFERENCE.md b/docs/REFERENCE.md index dacee89..62aba4a 100644 --- a/docs/REFERENCE.md +++ b/docs/REFERENCE.md @@ -137,6 +137,7 @@ are rejected. Input is a text list, **not a CSV or an Excel workbook**. | `--brand NAME` | `CompleteTech LLC` | Organization name shown on the HTML report. | | `--delay SECONDS` | `1` | Positive minimum time between requests to an origin. Robots rules may increase it. | | `--timeout SECONDS` | `15` | Positive per-socket timeout, not a total deadline for a website. | +| `--target-timeout SECONDS` | `120` | Positive monotonic budget shared by all work for each target. | | `--authorized-origin ORIGIN` | None | Additional authorized redirect origin; repeat for each approved scheme/host/port. No paths, queries or fragments. | | `--contact VALUE` | Empty | Public contact URL/email included in the User-Agent. | | `--help` | — | Show command help. | @@ -167,6 +168,7 @@ embedded assets, or stale/spoofed metadata can resemble WordPress. - `ok`: the final page returned HTTP 2xx, even if its content was inconclusive. - `timeout`: a recognized socket timeout occurred. +- `deadline_exceeded`: the target's total budget was exhausted; assessment remains unknown and the next input can proceed. - `dns_error`: a recognized DNS-resolution error occurred. - `out_of_scope`: a page or robots redirect was stopped before requesting an unauthorized origin; assessment remains unknown. - `blocked`: other cases preventing a successful fetch, including robots rules, @@ -271,11 +273,35 @@ upload destination; avoid committing real target lists or scan results. ## Python use +### Total target deadline + +The CLI defaults to `--target-timeout 120`. One monotonic budget covers robots, +redirects, required pacing, DNS, connection/TLS, response headers/body and local +assessment. The socket inactivity timeout is an additional bound, never a substitute +for the total budget. Each HTTP acquisition uses one disposable child process; +the parent keeps robots caches and request spacing. Expired acquisition is killed +and reaped before continuing. Windows, macOS and Linux use the same process boundary, +including for resolver calls that Python cannot cancel in a thread. No child is +left running intentionally. Process creation/OS scheduling and cancellation have +small overhead; reaping allows at most two further seconds, then aborts the batch +rather than launching another request if the OS cannot reap the worker. + +Deadline exhaustion produces an error row with `fetch_status=deadline_exceeded` +and `php2shell_status=unknown`. Other targets get fresh budgets. A completed batch +can therefore contain incomplete assessments. There are no retries or authorization +exceptions. Required pacing that cannot fit in the remaining budget fails immediately. + +Python callers opt into this same boundary with `Checker(target_timeout=120, ...)`. +The low-level `Checker` default remains `target_timeout=None` for integrations with +their own process supervisors; a direct unbounded `request()` is not a total-work +guarantee. Custom in-process opener changes are not transferred to bounded child +requests. The CLI's `--recover` stays offline and launches no HTTP worker. + ```python from dataclasses import asdict from wordpress_check import Checker -checker = Checker(timeout=15, delay=1, contact="https://your-domain.example/contact") +checker = Checker(timeout=15, target_timeout=120, delay=1, contact="https://your-domain.example/contact") result = checker.check("https://example.com/") print(asdict(result)) ``` diff --git a/skills/php2shell-check/SKILL.md b/skills/php2shell-check/SKILL.md index eba63cb..7f61ac4 100644 --- a/skills/php2shell-check/SKILL.md +++ b/skills/php2shell-check/SKILL.md @@ -87,7 +87,14 @@ evidence and timestamps must be preserved; HTML does not expose every field. `--delay` and `--timeout` must be positive finite numbers. Delay defaults to one second per origin and can be increased by robots rules. Timeout defaults to 15 -seconds **per socket**, not a total run deadline. Add `--contact` only for a +seconds **per socket**. `--target-timeout` defaults to 120 positive finite seconds +per target, shared by robots, redirects, pacing, DNS/connect/TLS, headers and body. +A disposable HTTP acquisition process is killed and reaped on expiry, so stalled +DNS and trickling responses cannot hold the next target indefinitely on Windows, +macOS or Linux. Allow small process-startup/cancellation scheduling overhead. +`fetch_status=deadline_exceeded` means incomplete/unknown, never a negative finding. +The batch continues after that result; completed-input counts do not imply every +assessment succeeded. Add `--contact` only for a public contact value the user intends to disclose in requests. Inspect each result even if the process exits with `0`: that means the run diff --git a/test_target_deadline.py b/test_target_deadline.py new file mode 100644 index 0000000..ab67b35 --- /dev/null +++ b/test_target_deadline.py @@ -0,0 +1,170 @@ +import contextlib +import io +import json +import subprocess +import sys +import threading +import time +import unittest +from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer +from pathlib import Path +from unittest.mock import patch + +import wordpress_check as checker + + +class TargetDeadlineTests(unittest.TestCase): + @classmethod + def setUpClass(cls): + cls.requests = [] + class Handler(BaseHTTPRequestHandler): + def do_GET(self): + cls.requests.append(self.path) + try: + if self.path == '/robots.txt': + self.send_response(200) + self.end_headers() + self.wfile.write(b'User-agent: *\nDisallow: /private\n') + elif self.path == '/slow': + self.send_response(200) + self.end_headers() + for _ in range(40): + self.wfile.write(b'x') + self.wfile.flush() + time.sleep(.04) + elif self.path == '/idle': + time.sleep(.3) + self.send_response(200) + self.end_headers() + elif self.path == '/unicode': + self.send_response(200) + self.send_header('Content-Type', 'text/html; charset=utf-8') + self.end_headers() + self.wfile.write('\u65e5\u672c\u8a9e \u2603'.encode('utf-8')) + elif self.path.startswith('/redirect'): + time.sleep(.12) + self.send_response(302) + self.send_header('Location', '/redirect' + str(int(self.path[9:]) + 1)) + self.end_headers() + else: + self.send_response(200) + self.end_headers() + self.wfile.write(b'synthetic') + except (BrokenPipeError, ConnectionResetError, ConnectionAbortedError): + pass + def log_message(self, *_): + pass + cls.server = ThreadingHTTPServer(('127.0.0.1', 0), Handler) + cls.thread = threading.Thread(target=cls.server.serve_forever, daemon=True) + cls.thread.start() + cls.url = f'http://127.0.0.1:{cls.server.server_port}' + + @classmethod + def tearDownClass(cls): + cls.server.shutdown() + cls.server.server_close() + cls.thread.join() + + def setUp(self): + self.requests.clear() + + def assert_incomplete(self, result): + self.assertEqual(result.status, 'error') + self.assertEqual(result.fetch_status, 'deadline_exceeded') + self.assertEqual(result.php2shell_status, 'unknown') + self.assertEqual(result.php2shell_exploitability, 'not_tested') + + def test_slow_trickle_is_cancelled_and_next_target_progresses(self): + client = checker.Checker(timeout=.2, target_timeout=.65, delay=0) + start = time.monotonic() + self.assert_incomplete(client.check(self.url + '/slow')) + self.assertLess(time.monotonic() - start, 1.5) + # Cache and state survive cancellation, but the target gets a fresh budget. + result = client.check(self.url + '/fast') + self.assertEqual(result.status, 'no_evidence') + self.assertEqual(self.requests.count('/robots.txt'), 1) + + def test_redirect_chain_shares_budget(self): + client = checker.Checker(timeout=1, target_timeout=.65, delay=0) + start = time.monotonic() + self.assert_incomplete(client.check(self.url + '/redirect0')) + self.assertLess(time.monotonic() - start, 1.5) + self.assertLess(len([p for p in self.requests if p.startswith('/redirect')]), 6) + + def test_pacing_does_not_sleep_past_budget_or_fetch_page(self): + client = checker.Checker(timeout=1, target_timeout=.5, delay=1) + start = time.monotonic() + self.assert_incomplete(client.check(self.url + '/fast')) + self.assertLess(time.monotonic() - start, 1) + self.assertEqual(self.requests, ['/robots.txt']) + + def test_dns_and_connect_stalls_are_killed_without_surviving_workers(self): + real_popen = subprocess.Popen + for operation in ('dns', 'connect'): + with self.subTest(operation=operation): + children = [] + def spawn(command, **kwargs): + script = "import runpy,sys,time;m=runpy.run_path(sys.argv[1]);" + if operation == 'dns': + script += "m['socket'].getaddrinfo=lambda *a,**k:time.sleep(20);" + else: + script += "m['socket'].socket.connect=lambda *a,**k:time.sleep(20);" + script += "m['_request_worker']()" + child = real_popen([sys.executable, '-I', '-c', script, str(Path(checker.__file__).resolve())], **kwargs) + children.append(child) + return child + start = time.monotonic() + with patch.object(checker.subprocess, 'Popen', side_effect=spawn): + result = checker.Checker(timeout=.1, target_timeout=.4, delay=0).check(self.url) + self.assert_incomplete(result) + self.assertLess(time.monotonic() - start, 1.4) + self.assertEqual(len(children), 1) + self.assertIsNotNone(children[0].poll()) + + def test_cli_deadline_keeps_result_and_continues_batch(self): + out = io.StringIO() + with contextlib.redirect_stdout(out), contextlib.redirect_stderr(io.StringIO()): + code = checker.main([self.url + '/slow', self.url + '/fast', '--format', 'json', + '--timeout', '.2', '--target-timeout', '.65', '--delay', '.001']) + report = json.loads(out.getvalue()) + self.assertEqual(code, 0) + self.assertTrue(report['complete']) # All inputs handled; first assessment failed. + self.assertEqual(report['results'][0]['fetch_status'], 'deadline_exceeded') + self.assertEqual(report['results'][0]['php2shell_status'], 'unknown') + self.assertEqual(report['results'][1]['status'], 'no_evidence') + + def test_robots_scope_and_socket_timeout_remain_active(self): + client = checker.Checker(timeout=.05, target_timeout=2, delay=0) + result = client.check(self.url + '/private') + self.assertEqual(result.status, 'blocked') + self.assertNotIn('/private', self.requests) + # An inactivity timeout remains distinct from total deadline exhaustion. + with patch.object(client, '_fetch', side_effect=AssertionError('parent HTTP forbidden')): + self.assertEqual(client.check(self.url + '/fast').status, 'no_evidence') + self.assertEqual(client.check(self.url + '/idle').fetch_status, 'timeout') + + def test_expiry_during_parsing_never_publishes_a_positive_assessment(self): + client = checker.Checker(target_timeout=1, delay=0) + client.request = lambda *args: checker.Response(200, {}, '', False) + client.robots[self.url] = (None, 0, 'synthetic robots') + def late_parse(response, result): + result.php_version = '8.3.36' + client._deadline = 0 + with patch.object(checker, 'inspect_page', side_effect=late_parse): + self.assert_incomplete(client.check(self.url)) + + def test_invalid_total_budgets_are_rejected(self): + for value in (0, -1, float('nan'), float('inf')): + with self.assertRaises(ValueError): + checker.Checker(target_timeout=value) + with contextlib.redirect_stderr(io.StringIO()), self.assertRaises(SystemExit): + checker.main(['example.test', '--target-timeout', '0']) + + def test_unicode_response_survives_worker_boundary(self): + result = checker.Checker(timeout=1, target_timeout=2, delay=0).check(self.url + '/unicode') + self.assertEqual(result.status, 'no_evidence') + self.assertEqual(result.title, '\u65e5\u672c\u8a9e \u2603') + + +if __name__ == '__main__': + unittest.main() diff --git a/wordpress_check.py b/wordpress_check.py index 501b1be..6606b14 100644 --- a/wordpress_check.py +++ b/wordpress_check.py @@ -18,6 +18,7 @@ import mimetypes import re import socket +import subprocess import sys import time from dataclasses import asdict, dataclass, field @@ -132,6 +133,10 @@ class OutOfScope(CheckStopped): """A redirect destination lacks explicit origin authorization.""" +class TargetDeadline(CheckStopped): + """The shared monotonic target budget expired.""" + + class NoRedirect(HTTPRedirectHandler): def redirect_request(self, req, fp, code, msg, headers, newurl): return None @@ -389,10 +394,14 @@ def inspect_page(response: Response, result: Result) -> None: class Checker: - def __init__(self, *, timeout=15.0, delay=1.0, contact="", max_crawl_delay=60.0, authorized_origins=()): + def __init__(self, *, timeout=15.0, delay=1.0, contact="", max_crawl_delay=60.0, authorized_origins=(), target_timeout=None): self.timeout = timeout self.delay = delay self.max_crawl_delay = max_crawl_delay + if target_timeout is not None and (not math.isfinite(target_timeout) or target_timeout <= 0): + raise ValueError("target_timeout must be positive and finite") + self.target_timeout = target_timeout + self._deadline = None self.authorized_origins = {origin_key(authorized_origin(value)) for value in authorized_origins} self.user_agent = f"{BOT}/1.0" + (f" (+{contact})" if contact else "") self.opener = build_opener(NoRedirect()) @@ -400,11 +409,73 @@ def __init__(self, *, timeout=15.0, delay=1.0, contact="", max_crawl_delay=60.0, self.robots: dict[str, tuple[RobotsPolicy | None, float, str]] = {} def request(self, url: str, limit: int, delay: float | None = None) -> Response: + own_deadline = self._deadline is None and self.target_timeout is not None + if own_deadline: + self._deadline = time.monotonic() + self.target_timeout + try: + return self._request(url, limit, delay) + finally: + if own_deadline: + self._deadline = None + + def _remaining(self): + if self._deadline is None: + return None + remaining = self._deadline - time.monotonic() + if remaining <= 0: + raise TargetDeadline("Total target deadline exceeded; assessment incomplete.") + return remaining + + def _request(self, url, limit, delay): site = origin(url) wait = max(self.delay, delay or 0) - (time.monotonic() - self.last_request.get(site, -math.inf)) + remaining = self._remaining() if wait > 0: + if remaining is not None and wait >= remaining: + raise TargetDeadline("Total target deadline cannot accommodate required pacing; assessment incomplete.") time.sleep(wait) self.last_request[site] = time.monotonic() + remaining = self._remaining() + if remaining is None: + return self._fetch(url, limit) + # A disposable process bounds even resolver calls and trickling headers/body. + # One child is active at a time; parent retains robots policy and pacing. + command = [sys.executable, "-I", "-c", "import runpy,sys; runpy.run_path(sys.argv[1])['_request_worker']()", str(Path(__file__).resolve())] + child = subprocess.Popen(command, stdin=subprocess.PIPE, stdout=subprocess.PIPE, stderr=subprocess.PIPE, + creationflags=getattr(subprocess, "CREATE_NO_WINDOW", 0)) + try: + payload = json.dumps({"url": url, "limit": limit, "timeout": min(self.timeout, remaining), "user_agent": self.user_agent}) + output, _ = child.communicate(payload.encode("utf-8"), timeout=self._remaining()) + except subprocess.TimeoutExpired: + raise TargetDeadline("Total target deadline exceeded during HTTP acquisition; assessment incomplete.") from None + finally: + if child.poll() is None: + child.kill() + # Reap the only child before another target can start. No resolver thread survives. + try: + child.wait(timeout=2) + except subprocess.TimeoutExpired: + # An OS-level cancellation failure must abort, never launch another worker. + raise RuntimeError("Unable to reap HTTP worker after cancellation; batch aborted") from None + for pipe in (child.stdin, child.stdout, child.stderr): + if pipe is not None: + pipe.close() + self.last_request[site] = time.monotonic() + self._remaining() + if child.returncode != 0: + raise OSError("HTTP acquisition worker failed") + message = json.loads(output) + if "error" in message: + kind = message.get("kind") + error = message["error"] + if kind == "timeout": + raise TimeoutError(error) + if kind == "dns": + raise socket.gaierror(error) + raise OSError(error) + return Response(**message["response"]) + + def _fetch(self, url, limit): req = Request(url, headers={"User-Agent": self.user_agent, "Accept": "text/html,text/plain;q=0.9,*/*;q=0.1", "Accept-Encoding": "identity"}) try: stream = self.opener.open(req, timeout=self.timeout) @@ -467,11 +538,13 @@ def policy(self, url: str, authorized=None) -> float: def check(self, value: str) -> Result: result = Result(input_url=value) + self._deadline = time.monotonic() + self.target_timeout if self.target_timeout is not None else None try: url = normalize_url(value) authorized = self.authorized_origins | {origin_key(url)} seen = set() for _ in range(6): + self._remaining() result.final_url = url if url in seen: raise CheckStopped("Page redirect loop; stopped.") @@ -482,6 +555,7 @@ def check(self, value: str) -> Result: if response.status not in REDIRECTS: result.notes.append(self.robots[origin(url)][2]) inspect_page(response, result) + self._remaining() return result location = response.headers.get("location") if not location: @@ -495,6 +569,9 @@ def check(self, value: str) -> Result: raise CheckStopped("Too many page redirects; stopped.") except CheckStopped as exc: result.status = "blocked" + if isinstance(exc, TargetDeadline): + result.status = "error" + result.fetch_status = "deadline_exceeded" if isinstance(exc, OutOfScope): result.fetch_status = "out_of_scope" result.notes.append(str(exc)) @@ -507,10 +584,31 @@ def check(self, value: str) -> Result: result.fetch_status = "dns_error" result.notes.append(str(exc)) finally: - annotate_php2shell(result) + self._deadline = None + if result.fetch_status == "deadline_exceeded": + # Evidence parsed just as the budget expired is not a completed assessment. + for name, value in assess_php2shell().items(): + setattr(result, name, value) + result.php2shell_notes.append("Total target deadline exceeded; assessment incomplete.") + else: + annotate_php2shell(result) return result +def _request_worker(): + """Private bounded HTTP acquisition; policy decisions stay in the parent.""" + payload = json.loads(sys.stdin.buffer.read().decode("utf-8")) + checker = Checker(timeout=payload["timeout"], delay=0) + checker.user_agent = payload["user_agent"] + try: + message = {"response": asdict(checker._fetch(payload["url"], payload["limit"]))} + except (ValueError, URLError, OSError, HTTPException, UnicodeError) as exc: + cause = exc.reason if isinstance(exc, URLError) else exc + kind = "timeout" if isinstance(cause, (TimeoutError, socket.timeout)) else "dns" if isinstance(cause, socket.gaierror) else "error" + message = {"error": str(exc)[:4096], "kind": kind} + sys.stdout.buffer.write(json.dumps(message, ensure_ascii=False).encode("utf-8")) + + def positive_float(value): number = float(value) if not math.isfinite(number) or number <= 0: @@ -742,6 +840,7 @@ def main(argv=None) -> int: parser.add_argument("--format", choices=("csv", "json", "html"), default="csv", help="csv, json, or a branded html report") parser.add_argument("--delay", type=positive_float, default=1.0, help="Minimum seconds between requests to an origin (default: 1)") parser.add_argument("--timeout", type=positive_float, default=15.0, help="Socket timeout per request (default: 15 seconds)") + parser.add_argument("--target-timeout", type=positive_float, default=120.0, help="Total monotonic seconds per target, including DNS, HTTP, redirects and pacing (default: 120)") parser.add_argument("--authorized-origin", action="append", default=[], type=authorized_origin, help="Additional authorized redirect origin (scheme, host and optional port); repeatable") parser.add_argument("--contact", default="", help="Public contact URL/email to include in the honest User-Agent") parser.add_argument("--logo", type=Path, default=DEFAULT_LOGO, help="Logo image for the html report (default: assets/logo.png)") @@ -780,7 +879,7 @@ def main(argv=None) -> int: if checkpoint: # Exclusive creation protects a previous interrupted run from accidental reuse. publish_report(checkpoint, checkpoint_json(report)) - checker = Checker(timeout=args.timeout, delay=args.delay, contact=args.contact, authorized_origins=args.authorized_origin) + checker = Checker(timeout=args.timeout, delay=args.delay, contact=args.contact, authorized_origins=args.authorized_origin, target_timeout=args.target_timeout) try: for index, url in enumerate(urls): result = checker.check(url)