Skip to content

New integration: WhoisFreaks WHOIS threat intelligence - #20754

Open
qayyum-jfreaks wants to merge 10 commits into
elastic:mainfrom
WhoisFreaks:feat/whoisfreaks
Open

New integration: WhoisFreaks WHOIS threat intelligence#20754
qayyum-jfreaks wants to merge 10 commits into
elastic:mainfrom
WhoisFreaks:feat/whoisfreaks

Conversation

@qayyum-jfreaks

@qayyum-jfreaks qayyum-jfreaks commented Aug 17, 2026

Copy link
Copy Markdown

Add the WhoisFreaks integration: collects daily gTLD and ccTLD "domainer"
WHOIS feeds from WhoisFreaks and indexes them as threat intelligence.

WHAT:

  • New whois data stream using the Elastic Agent CEL input. On each
    poll it checks the public WhoisFreaks status endpoint
    (/v3.1/status, reading newly.gtld.last_update /
    newly.cctld.last_update) against two independent per-feed cursors kept
    in the input's persistent state, and only pages a feed's CSV stream
    endpoint when a newer date is available — so a short polling interval
    (e.g. 1h) never re-downloads a file that's already been ingested.
  • Each feed is paged via offset/limit query parameters against the
    stream endpoint and parsed as headered CSV, tagged with
    whoisfreaks.tld_type (gtld or cctld), until a page comes back
    shorter than the page size.
  • Ingest pipeline sets @timestamp, parses WHOIS dates, derives
    whoisfreaks.days_until_expiry and whoisfreaks.is_newly_registered,
    populates ECS threat-intel fields (threat.indicator.*,
    domain.registered_domain, domain.registrar.name), and fingerprints
    tld_type + domain_name + query_time into the document _id for
    idempotent re-polling.
  • Ships an [WhoisFreaks] Overview dashboard (total/unique domain
    counts, records over time, top registrars/name servers, top domains).
  • docs/README.md includes a "Suggested detection rules" section with
    three ready-to-use KQL queries (newly-registered domain observed,
    domain expiring within 30 days, ingest pipeline/API error) for users to
    create as custom rules via Security → Rules → Create a rule — the
    package does not ship prebuilt security-rule assets, so nothing here
    depends on that installation path.
  • System tests run against a small Python mock server
    (_dev/deploy/docker/files/mock_server.py) that mirrors the real
    WhoisFreaks status-endpoint and stream-endpoint response shapes, so
    elastic-package test system exercises the full poll → status-check →
    page → ingest flow without a live API key.

WHY:
WhoisFreaks' newly-registered-domain feed is useful for phishing/brand-
impersonation detection, domain portfolio/expiration tracking, and
registrar/name-server pivoting. This package brings that feed into
Elastic Security as threat intelligence with minimal setup — an API key
and a Resource Interval are the only required configuration.

Checklist

  • I have reviewed tips for building integrations and this pull request is aligned with them.
  • I have verified that all data streams collect metrics or logs.
  • I have added an entry to my package's changelog.yml file.
  • I have verified that Kibana version constraints are current according to guidelines.
  • I have verified that any added dashboard complies with Kibana's Dashboard good practices

Author's Checklist

  • elastic-package check passes clean
  • elastic-package test system -v installs the package and ingests both gTLD and ccTLD mock records (no stuck "found 0 hits" polling)
  • elastic-package test pipeline -v passes
  • Verified the dashboard renders with the system-test mock data
  • Verified docs/README.md matches _dev/build/docs/README.md (rebuilt via elastic-package build after any template edit)

How to test this PR locally

  • git clone && cd integrations/packages/ti_whoisfreaks
  • elastic-package check
  • elastic-package test system -v
  • elastic-package test pipeline -v

Then install the package against a local stack (elastic-package stack up)
and confirm data lands in logs-ti_whoisfreaks.whois-* and the
[WhoisFreaks] Overview dashboard renders.

Related issues

Screenshots

Screenshot from 2026-08-17 14-48-56

@qayyum-jfreaks
qayyum-jfreaks requested a review from a team as a code owner August 17, 2026 10:15
@elastic-vault-github-plugin-prod

Copy link
Copy Markdown
Contributor

Reviewers

Buildkite won't run for external contributors automatically; you need to add a comment:

  • /test : will kick off a build in Buildkite.

NOTE: https://github.com/elastic/integrations/blob/main/.buildkite/pull-requests.json contains all those details.

@qayyum-jfreaks

Copy link
Copy Markdown
Author
  • /test

@qayyum-jfreaks

Copy link
Copy Markdown
Author

/test

@qcorporation

Copy link
Copy Markdown
Contributor

/test

@infra-vault-gh-plugin-prod

infra-vault-gh-plugin-prod Bot commented Aug 17, 2026

Copy link
Copy Markdown

💔 Build Failed

Failed CI Steps

History

@elastic-vault-github-plugin-prod

Copy link
Copy Markdown
Contributor

✅ All changelog entries have the correct PR link.

@qayyum-jfreaks

Copy link
Copy Markdown
Author

💔 Build Failed

Failed CI Steps

History

  • /test

@qayyum-jfreaks

Copy link
Copy Markdown
Author
  • /test

@cla-checker-service

cla-checker-service Bot commented Aug 18, 2026

Copy link
Copy Markdown

💚 CLA has been signed

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants