Design: metrics.rules_counters_extra_labels — promote output fields to rules_matches_total Prometheus labels

[ahbeigi]

Background

Issues #3652 (Jul 2025) and #3826 (Mar 2026) both request k8s_ns_name / k8s_pod_name as labels on falcosecurity_falco_rules_matches_total. PR #3839 attempted this but was correctly rejected: it hardcoded "n/a" as placeholder values for both labels, which provides no useful information and violates Prometheus best practices — labels should distinguish time series, not carry constant values.

In the review of #3839, @leogr diagnosed the root cause and proposed a concrete design direction:

stats_manager::on_event() only receives a const falco_rule& and has no access to per-event data. That said, the sinsp_evt* IS available at match time (in falco_engine::process_event()), and falco_outputs::handle_event() already resolves all output fields (including k8s.ns.name, k8s.pod.name from the container plugin) into a map<string, string>. So the data is there, it just doesn't reach the stats_manager.

A more generic solution would be to allow users to configure which output fields should be promoted to Prometheus labels on rules_matches_total. Something like:

metrics:
  rules_counters_extra_labels:
    - k8s.ns.name
    - k8s.pod.name

This would require changes to the stats_manager to track per-(rule, label_values) counters instead of just per-rule counters, but the architecture supports it since the event data is available at the right point in the pipeline.

This discussion is to work through that design before a PR is submitted.

---

Proposed design

Config (falco.yaml, under metrics):

rules_counters_extra_labels: []   # default: empty — no behaviour change

When non-empty and rules_counters_enabled: true, the listed Falco output field names are promoted to additional Prometheus label dimensions on rules_matches_total. Label names are sanitized from Falco dot-notation to underscores (e.g. k8s.ns.name → k8s_ns_name) for compatibility with tooling that enforces the [a-zA-Z_][a-zA-Z0-9_]* label name convention.

Event capture point

At match time in falco_engine::process_event() / falco_outputs::handle_event(), where sinsp_evt* is available and output fields are already resolved into a map<string, string>, extract the values for the configured extra label fields and pass them alongside the rule reference to stats_manager.

stats_manager change

Change the rule counter map from per-rule to per-(rule, label-values-tuple):

// before
map<rule_name, uint64_t>

// after
map<tuple<rule_name, vector<label_values>>, uint64_t>

The label names (keys) are fixed at startup from config; the label values are captured per-event. When rules_counters_extra_labels is empty the behaviour is identical to today.

Prometheus exposition

falco_metrics.cpp emits one series per unique (rule_name, label_values_tuple). The extra label names and values appear alongside the existing rule_name, priority, and source labels.

---

Open questions

1. Cardinality and stale series. A pod that triggers a rule match and is then terminated leaves a stale series in the counter map until Falco restarts. This behaviour is common in the Prometheus ecosystem (e.g. kube-state-metrics produces series for resources that no longer exist until the next scrape cycle completes). Is accepting stale series reasonable here, or should the implementation include a TTL/eviction mechanism? If eviction, what form — bounded LRU map, periodic sweep on a configurable TTL, or none for the initial implementation with a future follow-up?

2. Label name sanitization. Falco output field names use dots (k8s.ns.name). The Prometheus data model recommends label names match [a-zA-Z_][a-zA-Z0-9_]* for best compatibility with tooling. Proposed: replace . with _ at exposition time. Are there field names where this produces collisions or ambiguity?

3. Field availability. Some Falco output fields may resolve to <NA> when the relevant plugin (e.g. the container plugin) is not loaded or when metadata is not yet available for a given process. Should events where a requested extra-label field resolves to <NA> or empty string be counted under that value as a distinct label, skipped entirely, or cause a validation warning at startup?

4. Scope of accepted fields. Should rules_counters_extra_labels accept any output field name, or should there be validation at startup against the set of fields resolvable by the loaded plugins? Accepting arbitrary names with no validation could lead to silently empty labels if a field is misspelled or its plugin is not loaded.

---

References

• Issue Prometheus metrics should include pod name and namespace as labels #3652
• Issue Prometheus metrics should include pod name and namespace as labels #3826
• PR feat(metrics): add optional k8s metadata labels to rules_matches #3839 and @leogr's review comment

cc @falcosecurity/core-maintainers

[ahbeigi]

Hi! I was wondering if we can get any update on this topic?

[leogr]

Hi! I was wondering if we can get any update on this topic?

Hi @ahbeigi

This is for sure worth a deep discussion. Just so you know, the maintainers are a bit unresponsive because we are busy preparing the release for next month.

cc @falcosecurity/core-maintainers