NethVoice: Alerts for call quality, registrations and CTI clients #8274

Second phase of the built-in NethVoice alerts (#8066). It adds alerts for problems that users notice before administrators do: many phones losing registration, many CTI clients disconnecting, a flood of rejected registrations, many failed calls, and poor audio quality.

Phase 2 alerts could warnings, never critical. They are published only when the "Alerts (beta)" switch of the module is on, like the phase 1 alerts.

Proposed Alerts

#2 Many devices unregistered (NethVoice)

• New export-metrics collector devices: from the Asterisk /metrics page (res_prometheus, already read by phase 1), count asterisk_endpoints_state == 2 whose resource is not a trunk and not anonymous. Export nethvoice_registered_devices (and nethvoice_configured_devices, same set in any state).
• Rule:

  (max_over_time(nethvoice_registered_devices[15m] offset 10m) - nethvoice_registered_devices)
    / max_over_time(nethvoice_registered_devices[15m] offset 10m) > 0.30
  and
  (max_over_time(nethvoice_registered_devices[15m] offset 10m) - nethvoice_registered_devices) >= 5
  

• Event alert: it fires about 15 minutes after the drop and clears by itself after about 25 minutes. A long outage stays covered by ServiceDown. A slow decrease (for example at the end of the working day) does not fire.

#3 Many failed registrations (Proxy)

• Kamailio counts REGISTER requests rejected with 403 or 404, per NethVoice instance, in a new htable nvstats (key regrej|<setid>|<code>, incremented with $shtinc). 401/407 challenges are not counted.
• The proxy kamailio collector reads nvstats with JSON-RPC htable.dump (http://127.0.0.1:5060/RPC, already used for nvtrunkreg) and maps the dispatcher set id to the NethVoice module id. Export counter nethvoice_proxy_register_rejected_total{nethvoice,code}.
• Rule: sum by (nethvoice) (rate(nethvoice_proxy_register_rejected_total[15m])) * 60 > 20.
• Overlaps with the CrowdSec kamailio-bf scenario: CrowdSec bans the source, this alert only notifies.

#4 Many CTI disconnections (NethVoice)

• New collector cti_clients, only after the wizard has started the CTI (same condition as the CTI services in phase 1): GET http://127.0.0.1:${NETHCTI_INTERNAL_REST_PORT11}/profiling/conn_clients on nethcti-server, ws_conn_clients + tcp_conn_clients.tot, exported as gauge nethvoice_cti_connected_clients. No change to nethcti-server.
• Rule: same shape as #2 on nethvoice_cti_connected_clients.
• It counts connections, not logouts.

#5 High failed-call rate (Proxy)

• Kamailio tags every initial INVITE with the NethVoice instance (dispatcher set id, 0 if unknown) and a category: in and out for trunk legs, internal for device legs (is_from_local() for incoming, is_domain_local("$td") for outgoing). Use dedicated $avp(nv_*) variables, because onreply_route[MANAGE_REPLY] overwrites $avp(direction).
• On the final reply, count once per call (guarded by a dialog variable) in nvstats, key inv|<setid>|<cat>|<result>:
  • answered: 2xx
  • failed: 5xx, 408, 488, and 403 on trunk legs
  • excluded: 486, 487, 480, 404, and 403 on device legs
  • other: everything else
• Export counter nethvoice_proxy_invite_final_total{nethvoice,direction,result}. Also export zero values for the missing results of every seen (nethvoice, direction), so the first failure is not lost by increase().
• Rule:

  (
    sum by (nethvoice, direction) (increase(nethvoice_proxy_invite_final_total{result="failed"}[15m]))
    /
    sum by (nethvoice, direction) (increase(nethvoice_proxy_invite_final_total[15m]))
  ) > 0.20
  and
  sum by (nethvoice, direction) (increase(nethvoice_proxy_invite_final_total[15m])) >= 20
  

• Per instance and direction only, not per trunk. A version based on call records (CDR) was rejected.

#6 and #7 Audio quality (Proxy)

• Enable measure-rtp in rtpengine, applied by update-module (it restarts rtpengine). With it, jitter is measured on all relayed calls, not only on calls with RTCP reports.
• New proxy collector rtpengine: podman exec rtpengine rtpengine-ctl -ip 127.0.0.1 -port 2224 list jsonstats. Export counters nethvoice_proxy_rtp_{jitter,packetloss,mos}_sum_total (from jitter_total, packetloss_total, mos_total) and nethvoice_proxy_rtp_{jitter,packetloss,mos}_samples_total. The rtpengine HTTP /metrics endpoint is not used: it needs a new listening port and an rtpengine restart.
• Rules: average = increase(sum[15m]) / increase(samples[15m]), with increase(samples[15m]) >= 10. Jitter > 30 (ms), packet loss > 2 (%), MOS < 3.5.
• One value for the whole proxy, not per trunk or per NethVoice instance. Only media that goes through rtpengine is measured.

Checks before implementing

• V3: with a real provider and a phone registered through the proxy, confirm the trunk/device split in Kamailio (is_from_local(), is_domain_local("$td")) and that $avp(nv_*) are readable in onreply_route.
• V4: with a real call, confirm the units reported by rtpengine (jitter in ms, MOS 1–5, packet loss in %) and that typical phones and providers send RTCP.

References

• Parent issue: NethVoice: Configurable Alerting for Critical NethVoice Events #8066
• Phase 1: feat(alerts): add first NethVoice alerts nethesis/ns8-nethvoice#1067, feat(alerts): add ServiceDown and trunk alerts nethesis/ns8-nethvoice-proxy#231
• Module export-metrics: feat(metrics): publish module metric files ns8-core#1365 (core 3.24.0)
• Module alert rules: feat(alerts): allow module-provided rules ns8-metrics#91 (metrics 1.7.0)