Skip to content

Improve Akka.Cluster.Metrics collected values#6203

Merged
Aaronontheweb merged 9 commits into
akkadotnet:v1.4from
Arkatufus:Improve_metrics_collection
Oct 28, 2022
Merged

Improve Akka.Cluster.Metrics collected values#6203
Aaronontheweb merged 9 commits into
akkadotnet:v1.4from
Arkatufus:Improve_metrics_collection

Conversation

@Arkatufus

@Arkatufus Arkatufus commented Oct 19, 2022

Copy link
Copy Markdown
Contributor

close #4142

This improves the values collected by the metrics plugin, it does not fix the heap selector since it is still missing informations.
Heap metric based routing can only be implemented after we implement the new EventCounters API.

@Aaronontheweb Aaronontheweb left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM but needs API approvals

// VirtualMemorySize64 is not best idea here...
NodeMetrics.Types.Metric.Create(StandardMetrics.MemoryAvailable, process.VirtualMemorySize64).Value,
NodeMetrics.Types.Metric.Create(StandardMetrics.MemoryUsed, GC.GetTotalMemory(false)).Value,
NodeMetrics.Types.Metric.Create(StandardMetrics.MemoryAvailable, process.WorkingSet64).Value,

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@Aaronontheweb

Copy link
Copy Markdown
Member

We will also need to backport this to v1.5 @Arkatufus

@Aaronontheweb

Copy link
Copy Markdown
Member

@Arkatufus looks like some of the MNTR specs for Akka.Cluster.Metrics are not passing. Could you investigate?

@Arkatufus

Arkatufus commented Oct 20, 2022

Copy link
Copy Markdown
Contributor Author

Need a re-review.
Changed the code, available memory is the virtual size, used memory is the working set

@Aaronontheweb

Copy link
Copy Markdown
Member

@Arkatufus DocFx warning check here

NodeMetrics.Types.Metric.Create(StandardMetrics.MemoryUsed, GC.GetTotalMemory(true)).Value,
// VirtualMemorySize64 is not best idea here...
NodeMetrics.Types.Metric.Create(StandardMetrics.MemoryUsed, process.WorkingSet64).Value,
NodeMetrics.Types.Metric.Create(StandardMetrics.MemoryAvailable, process.VirtualMemorySize64).Value,

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this should be working set

{
// Memory
// Forcing garbage collection to keep metrics more resilent to occasional allocations
NodeMetrics.Types.Metric.Create(StandardMetrics.MemoryUsed, GC.GetTotalMemory(true)).Value,

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This should be GC

@@ -55,12 +55,11 @@ public NodeMetrics Sample()
{
using (var process = Process.GetCurrentProcess())

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I wonder if this is wrong - shouldn't we just cache this value somewhere?

@Aaronontheweb

Copy link
Copy Markdown
Member

The right algorithm, which we can't implement on .NET Standard 2.0 without a lot of hacks:

used memory = gc.totalsize OR process.workingset64
available memory = GC.GetGCMemoryInfo().TotalAvailableMemoryBytes

We should be measuring how close we're getting to hitting the total amount of available memory on the system as a percentage. In the meantime, we can do the following on .NET Standard 2.0:

used memory = gc.totalsize
available memory = process.workingset64 + process.pagedmemorysize64

This will tell us how much memory is being consumed by the .NET application itself - it won't be able to factor in issues like system memory and some unmanaged memory without the user manually calling GC.AddMemoryPressure but that's probably the best compromise we can make.

@Aaronontheweb

Copy link
Copy Markdown
Member

Testing out the latter now - we'll go back to the drawing board if we keep running into problems getting the MNTR to pass.

@Aaronontheweb

Copy link
Copy Markdown
Member

@Arkatufus needs a port to v1.5

@Aaronontheweb Aaronontheweb merged commit dca908b into akkadotnet:v1.4 Oct 28, 2022
Arkatufus added a commit to Arkatufus/akka.net that referenced this pull request Nov 2, 2022
* Improve collected values

* Update API Verify list

* Fix metrics collection

* Fix documentation

* Update API Verify list

* Update DefaultCollector.cs

* Update DefaultCollector.cs

Co-authored-by: Aaron Stannard <aaron@petabridge.com>
(cherry picked from commit dca908b)
Aaronontheweb added a commit that referenced this pull request Nov 3, 2022
* Improve collected values

* Update API Verify list

* Fix metrics collection

* Fix documentation

* Update API Verify list

* Update DefaultCollector.cs

* Update DefaultCollector.cs

Co-authored-by: Aaron Stannard <aaron@petabridge.com>
(cherry picked from commit dca908b)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants