Ability to forget and reclaim MetricPoint (Delta)

[cijothomas]

Each Metric keeps track of its MetricPoints. This points are never removed, even if there are no new updates to it.
For delta aggregation:
We should reclaim any un-used MetricPoints, when we hit Metric Point limits.
Due to pre-allocation of MetricPoints in current implementation, there won't be any memory savings due to this. But we'll be able to accomodate more MetricPoints, by reclaiming unused points. This would mean the MetricPoint limit is the "limit of active points", rather than "limit of points seen since the beginning of process".

For cumulative aggregation:
Its possible to "forget" an un-used Metricpoint. However, if the forgotten point is reported again, we'll not be able to calculate cumulative since the process start. This would result in a reset (i.e start time will be reset).

EDIT by utpilla:

This issue has been updated to track MetricPoint reclaim for Delta aggregation temporality only.

[cijothomas]

This issue is slightly related to #2524, but more complex.
A few attempts were done to achieve this, see PRs: #2466
#2600 #2598

Some observations:

1. To account for the possibility of a MetricPoint being reclaimed by someone else, the hot path has to do some synchronization mechanism. This has significantly affected throughput, in all the above PRs.
2. At the root, the issue is "atomically lookup which MetricPoint should be used for given tags, and do the update, without affecting performance".
3. As proved by the PR attempts above, plain locks/ or even ReadWriteLockSlim cannot be used as is, without sacrificing performance.

This requires more investigation and time to address. For 1.2 release, #2524 and #2358 will be focused, and this issue will be re-investigated post 1.2

[reyang]

Throwing couple ideas here:

1. the synchronization (lock) approach - break down the lock granularity - e.g. instead of having a single R/W lock for each point, having a per CPU core/group lock then decide if there is a need to escalate to a bigger lock.
2. non synchronization approach (even better) - leverage the page faults or segment access from memory manager to prevent stale writes (and the stale writes should result in a recoverable error + retry).