-
Notifications
You must be signed in to change notification settings - Fork 11
feat: GeoIP support #315
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Open
hhvrc
wants to merge
6
commits into
develop
Choose a base branch
from
feature/ip-enrichment-geo-vpn
base: develop
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Open
feat: GeoIP support #315
Changes from all commits
Commits
Show all changes
6 commits
Select commit
Hold shift + click to select a range
3647333
Add GeoIP support
hhvrc 8c86aa9
Merge branch 'develop' into feature/ip-enrichment-geo-vpn
hhvrc b24627d
fix(geo): address PR review comments
hhvrc 72f8aff
Merge remote-tracking branch 'origin/develop' into feature/ip-enrichm…
hhvrc dc76755
Merge branch 'develop' into feature/ip-enrichment-geo-vpn
hhvrc fbfcf2e
fix: Report only real VPN providers in IsVpn
hhvrc File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Some comments aren't visible on the classic Files Changed page.
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,50 @@ | ||
| using OpenShock.Common.Services.Geo; | ||
|
|
||
| namespace OpenShock.Common.Tests.Services; | ||
|
|
||
| public class IpEnrichmentServiceTests | ||
| { | ||
| [Test] | ||
| [Arguments("NordVPN")] | ||
| [Arguments("Mullvad VPN AB")] | ||
| [Arguments("Private Internet Access, Inc.")] | ||
| [Arguments("privateinternetaccess.com")] | ||
| [Arguments("Hotspot Shield")] | ||
| [Arguments("PIA")] | ||
| public async Task MatchesVpnProvider_KnownVpnOrgs_ReturnsTrue(string asnOrg) | ||
| { | ||
| await Assert.That(IpEnrichmentService.MatchesVpnProvider(asnOrg)).IsTrue(); | ||
| } | ||
|
|
||
| // The datacenter and hosting ASNs that used to live in the keyword list. They carry ordinary | ||
| // traffic and must not be reported as VPNs. | ||
| [Test] | ||
| [Arguments("Amazon.com, Inc.")] | ||
| [Arguments("Google LLC")] | ||
| [Arguments("Microsoft Corporation")] | ||
| [Arguments("Akamai Technologies, Inc.")] | ||
| [Arguments("OVH SAS")] | ||
| [Arguments("Hetzner Online GmbH")] | ||
| [Arguments("DigitalOcean, LLC")] | ||
| public async Task MatchesVpnProvider_HostingOrgs_ReturnsFalse(string asnOrg) | ||
| { | ||
| await Assert.That(IpEnrichmentService.MatchesVpnProvider(asnOrg)).IsFalse(); | ||
| } | ||
|
|
||
| // "pia" is a real vendor abbreviation and also a substring of ordinary words. Whole-token | ||
| // matching is what keeps the second case from being reported as a VPN. | ||
| [Test] | ||
| [Arguments("Olympia Networks")] | ||
| [Arguments("Compia Telecom")] | ||
| [Arguments("Utopia Broadband")] | ||
| public async Task MatchesVpnProvider_SubstringLookalikes_ReturnsFalse(string asnOrg) | ||
| { | ||
| await Assert.That(IpEnrichmentService.MatchesVpnProvider(asnOrg)).IsFalse(); | ||
| } | ||
|
|
||
| [Test] | ||
| public async Task MatchesVpnProvider_EmptyOrg_ReturnsFalse() | ||
| { | ||
| await Assert.That(IpEnrichmentService.MatchesVpnProvider(string.Empty)).IsFalse(); | ||
| } | ||
| } |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,16 @@ | ||
| namespace OpenShock.Common.Options; | ||
|
|
||
| public sealed class GeoOptions | ||
| { | ||
| public const string SectionName = "OpenShock:Geo"; | ||
|
|
||
| /// <summary> | ||
| /// Path to the MaxMind GeoLite2-ASN.mmdb file. | ||
| /// </summary> | ||
| public string? AsnDbPath { get; init; } | ||
|
|
||
| /// <summary> | ||
| /// Path to the MaxMind GeoLite2-City.mmdb file. | ||
| /// </summary> | ||
| public string? CityDbPath { get; init; } | ||
| } | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,11 @@ | ||
| using System.Net; | ||
|
|
||
| namespace OpenShock.Common.Services.Geo; | ||
|
|
||
| public interface IIpEnrichmentService | ||
| { | ||
| /// <summary> | ||
| /// Returns null when neither GeoLite2 database is configured or available. | ||
| /// </summary> | ||
| IpEnrichmentData? Enrich(IPAddress ip); | ||
| } |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,8 @@ | ||
| namespace OpenShock.Common.Services.Geo; | ||
|
|
||
| public sealed record IpEnrichmentData( | ||
| string? AsnOrg, | ||
| bool? IsVpn, | ||
| string? CountryCode, | ||
| string? City | ||
| ); |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,139 @@ | ||
| using System.Net; | ||
| using MaxMind.GeoIP2; | ||
| using Microsoft.Extensions.Logging; | ||
| using OpenShock.Common.Options; | ||
|
|
||
| namespace OpenShock.Common.Services.Geo; | ||
|
|
||
| public sealed class IpEnrichmentService : IIpEnrichmentService, IDisposable | ||
| { | ||
| // Consumer VPN providers only. Datacenter and hosting ASNs are deliberately NOT listed here: | ||
| // AWS, Google, Microsoft, Akamai, OVH and friends carry an enormous amount of ordinary traffic | ||
| // (corporate egress, mobile carrier NAT, CDN-fronted clients), so treating them as VPN evidence | ||
| // flags a large share of legitimate users. That signal is worth surfacing one day, but as its own | ||
| // "hosting provider" field rather than folded into a flag the UI presents as "VPN". | ||
| // | ||
| // Matched against whole tokens, not raw substrings, so "pia" cannot match "Olympia". Multi-word | ||
| // vendors are listed in both spaced and concatenated form because ASN org names use both. | ||
| private static readonly string[] VpnProviderKeywords = | ||
| [ | ||
| "mullvad", "nordvpn", "expressvpn", "protonvpn", "ipvanish", "surfshark", | ||
| "privateinternetaccess", "private internet access", "pia", | ||
| "hidemyass", "hide my ass", "purevpn", "cyberghost", | ||
| "windscribe", "tunnelbear", "hotspot shield", "hotspotshield", | ||
| "vyprvpn", "airvpn", "perfect privacy", "perfectprivacy", "ivpn", "ovpn", | ||
| ]; | ||
|
|
||
| private readonly DatabaseReader? _asnReader; | ||
| private readonly DatabaseReader? _cityReader; | ||
| private readonly ILogger<IpEnrichmentService> _logger; | ||
|
|
||
| public IpEnrichmentService(GeoOptions options, ILogger<IpEnrichmentService> logger) | ||
| { | ||
| _logger = logger; | ||
|
|
||
| _asnReader = TryOpen(options.AsnDbPath, "ASN"); | ||
| _cityReader = TryOpen(options.CityDbPath, "City"); | ||
| } | ||
|
|
||
| /// <summary> | ||
| /// Whole-token match of an ASN organisation name against <see cref="VpnProviderKeywords"/>. | ||
| /// Punctuation becomes whitespace so "Amazon.com, Inc." tokenises the way a reader expects, and | ||
| /// the padded haystack lets a single Contains express a word-boundary match for phrases too. | ||
| /// </summary> | ||
| public static bool MatchesVpnProvider(string asnOrg) | ||
| { | ||
| var normalized = string.Create(asnOrg.Length, asnOrg, static (span, source) => | ||
| { | ||
| for (var i = 0; i < source.Length; i++) | ||
| { | ||
| var c = char.ToLowerInvariant(source[i]); | ||
| span[i] = char.IsLetterOrDigit(c) ? c : ' '; | ||
| } | ||
| }); | ||
|
|
||
| var haystack = $" {string.Join(' ', normalized.Split(' ', StringSplitOptions.RemoveEmptyEntries))} "; | ||
|
|
||
| return Array.Exists(VpnProviderKeywords, k => haystack.Contains($" {k} ", StringComparison.Ordinal)); | ||
| } | ||
|
|
||
| private DatabaseReader? TryOpen(string? path, string dbName) | ||
| { | ||
| if (string.IsNullOrWhiteSpace(path)) | ||
| { | ||
| _logger.LogInformation("GeoLite2 {DbName} database path not configured, skipping", dbName); | ||
| return null; | ||
| } | ||
|
|
||
| if (!File.Exists(path)) | ||
| { | ||
| _logger.LogWarning("GeoLite2 {DbName} database not found at {Path}", dbName, path); | ||
| return null; | ||
| } | ||
|
|
||
| try | ||
| { | ||
| return new DatabaseReader(path); | ||
| } | ||
| catch (Exception ex) | ||
| { | ||
| _logger.LogError(ex, "Failed to open GeoLite2 {DbName} database at {Path}", dbName, path); | ||
| return null; | ||
| } | ||
| } | ||
|
|
||
| public IpEnrichmentData? Enrich(IPAddress ip) | ||
| { | ||
| if (_asnReader is null && _cityReader is null) return null; | ||
|
|
||
| string? asnOrg = null; | ||
| // Null means "unknown" (no ASN DB, lookup miss, or failure); only a resolved ASN org yields a verdict. | ||
| bool? isVpn = null; | ||
|
|
||
|
hhvrc marked this conversation as resolved.
|
||
| if (_asnReader is not null) | ||
| { | ||
| try | ||
| { | ||
| if (_asnReader.TryAsn(ip, out var asn) && asn is not null) | ||
| { | ||
| asnOrg = asn.AutonomousSystemOrganization; | ||
| if (asnOrg is not null) | ||
| { | ||
| isVpn = MatchesVpnProvider(asnOrg); | ||
| } | ||
| } | ||
| } | ||
| catch (Exception ex) | ||
| { | ||
| _logger.LogDebug(ex, "ASN lookup failed for {Ip}", ip); | ||
| } | ||
| } | ||
|
|
||
| string? countryCode = null; | ||
| string? city = null; | ||
|
|
||
| if (_cityReader is not null) | ||
| { | ||
| try | ||
| { | ||
| if (_cityReader.TryCity(ip, out var cityResponse) && cityResponse is not null) | ||
| { | ||
| countryCode = cityResponse.Country.IsoCode; | ||
| city = cityResponse.City.Name; | ||
| } | ||
| } | ||
| catch (Exception ex) | ||
| { | ||
| _logger.LogDebug(ex, "City lookup failed for {Ip}", ip); | ||
| } | ||
| } | ||
|
|
||
| return new IpEnrichmentData(asnOrg, isVpn, countryCode, city); | ||
| } | ||
|
|
||
| public void Dispose() | ||
| { | ||
| _asnReader?.Dispose(); | ||
| _cityReader?.Dispose(); | ||
| } | ||
| } | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.