New integration: WhoisFreaks WHOIS threat intelligence - #20754
Open
qayyum-jfreaks wants to merge 10 commits into
Open
New integration: WhoisFreaks WHOIS threat intelligence#20754qayyum-jfreaks wants to merge 10 commits into
qayyum-jfreaks wants to merge 10 commits into
Conversation
Contributor
ReviewersBuildkite won't run for external contributors automatically; you need to add a comment:
NOTE: https://github.com/elastic/integrations/blob/main/.buildkite/pull-requests.json contains all those details. |
Author
|
Author
|
/test |
Contributor
|
/test |
💔 Build Failed
Failed CI StepsHistory |
Contributor
|
✅ All changelog entries have the correct PR link. |
…egrations into feat/whoisfreaks
Author
|
Author
|
|
💚 CLA has been signed |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Add the WhoisFreaks integration: collects daily gTLD and ccTLD "domainer"
WHOIS feeds from WhoisFreaks and indexes them as threat intelligence.
WHAT:
whoisdata stream using the Elastic Agent CEL input. On eachpoll it checks the public WhoisFreaks status endpoint
(
/v3.1/status, readingnewly.gtld.last_update/newly.cctld.last_update) against two independent per-feed cursors keptin the input's persistent state, and only pages a feed's CSV stream
endpoint when a newer date is available — so a short polling interval
(e.g.
1h) never re-downloads a file that's already been ingested.offset/limitquery parameters against thestream endpoint and parsed as headered CSV, tagged with
whoisfreaks.tld_type(gtldorcctld), until a page comes backshorter than the page size.
@timestamp, parses WHOIS dates, deriveswhoisfreaks.days_until_expiryandwhoisfreaks.is_newly_registered,populates ECS threat-intel fields (
threat.indicator.*,domain.registered_domain,domain.registrar.name), and fingerprintstld_type+domain_name+query_timeinto the document_idforidempotent re-polling.
counts, records over time, top registrars/name servers, top domains).
docs/README.mdincludes a "Suggested detection rules" section withthree ready-to-use KQL queries (newly-registered domain observed,
domain expiring within 30 days, ingest pipeline/API error) for users to
create as custom rules via Security → Rules → Create a rule — the
package does not ship prebuilt
security-ruleassets, so nothing heredepends on that installation path.
(
_dev/deploy/docker/files/mock_server.py) that mirrors the realWhoisFreaks status-endpoint and stream-endpoint response shapes, so
elastic-package test systemexercises the full poll → status-check →page → ingest flow without a live API key.
WHY:
WhoisFreaks' newly-registered-domain feed is useful for phishing/brand-
impersonation detection, domain portfolio/expiration tracking, and
registrar/name-server pivoting. This package brings that feed into
Elastic Security as threat intelligence with minimal setup — an API key
and a Resource Interval are the only required configuration.
Checklist
changelog.ymlfile.Author's Checklist
elastic-package checkpasses cleanelastic-package test system -vinstalls the package and ingests both gTLD and ccTLD mock records (no stuck "found 0 hits" polling)elastic-package test pipeline -vpassesdocs/README.mdmatches_dev/build/docs/README.md(rebuilt viaelastic-package buildafter any template edit)How to test this PR locally
Then install the package against a local stack (
elastic-package stack up)and confirm data lands in
logs-ti_whoisfreaks.whois-*and the[WhoisFreaks] Overview dashboard renders.
Related issues
Screenshots