Improved Usage Tracking

[sirwolfgang]

This is to improved usage tracking and instrumentation for cost monitoring and APM integration. The original instrumentation was a slop coded prototype, that didn't actually work in a way that is useful. This cuts it down and restructures it for APM integration while adding usage normalization.

Multi-Turn Usage Tracking

Cumulative token tracking across tool calling cycles:

response = agent.generate("Research the weather in NYC")
# After 3 API calls (initial + 2 tool rounds):

response.usage.input_tokens   #=> 350 (cumulative across all calls)
response.usage.output_tokens  #=> 120 (cumulative across all calls)
response.usage.total_tokens   #=> 470 (cumulative across all calls)

# Access per-call details
response.usages.size          #=> 3
response.usages[0].input_tokens  #=> 100 (first call)
response.usages[1].input_tokens  #=> 150 (after tool execution)
response.usages[2].input_tokens  #=> 100 (final call)

Usage Statistics

Consistent token tracking across all providers:

response = agent.generate("Hello")
response.usage.input_tokens   #=> 8
response.usage.output_tokens  #=> 12
response.usage.total_tokens   #=> 20
response.usage.cached_tokens  #=> 0 (when available)

Provider-specific fields: reasoning_tokens, audio_tokens, cache_creation_tokens, service_tier, duration_ms

Combine usage objects:

total_usage = response1.usage + response2.usage
total_usage.input_tokens  #=> Combined from both

APM Integration

Unified instrumentation events with comprehensive payloads:

Top-level events (overall request lifecycle):

# Fires once per agent.generate call
ActiveSupport::Notifications.subscribe("prompt.active_agent") do |event|
  event.payload[:model]           #=> "gpt-4"
  event.payload[:temperature]     #=> 0.7
  event.payload[:message_count]   #=> 3
  event.payload[:stream]          #=> false
  
  # Cumulative across all API calls in multi-turn conversations
  event.payload[:usage]           #=> { input_tokens: 350, output_tokens: 120, total_tokens: 470 }
  event.payload[:finish_reason]   #=> "stop"
  event.payload[:response_id]     #=> "chatcmpl-123"
end

Provider-level events (per API call):

# Fires for each API call (multiple times during tool calling)
ActiveSupport::Notifications.subscribe("prompt.provider.active_agent") do |event|
  # Per-call usage for granular tracking
  event.payload[:usage]           #=> { input_tokens: 100, output_tokens: 50, total_tokens: 150 }
end

Event payloads include:

• Request params: model, temperature, max_tokens, top_p, stream, message_count, has_tools
• Usage data: input_tokens, output_tokens, total_tokens, cached_tokens, reasoning_tokens, audio_tokens, cache_creation_tokens
• Response metadata: finish_reason, response_model, response_id, embedding_count

Embedding events:

ActiveSupport::Notifications.subscribe("embed.active_agent") do |event|
  event.payload[:model]           #=> "text-embedding-ada-002"
  event.payload[:input_size]      #=> 5
  event.payload[:embedding_count] #=> 5
  event.payload[:usage]           #=> { input_tokens: 150, output_tokens: 0, total_tokens: 150 }
end

New response methods:

response.id              # Response ID
response.model           # Model used
response.finish_reason   # Why generation stopped
response.stop_reason     # Alias for Anthropic

Breaking Change

# Before
response.usage["prompt_tokens"]

# After
response.usage.input_tokens

[Copilot]

Pull Request Overview

This pull request introduces normalized usage statistics across all providers (OpenAI, Anthropic, Ollama, and OpenRouter) by adding a unified Usage model. The changes enable consistent access to token usage statistics regardless of which provider is being used.

Key changes:

• Added comprehensive unit tests for provider transform modules to verify gem-to-hash conversions and parameter normalization
• Implemented Usage model tests covering all provider formats with auto-detection capabilities
• Added integration tests with VCR cassettes for all providers to validate response handling
• Updated documentation and added inline documentation for model classes

Reviewed Changes

Copilot reviewed 214 out of 216 changed files in this pull request and generated 1 comment.

Show a summary per file

File	Description
test/providers/open_router/transforms_test.rb	Unit tests for OpenRouter-specific transforms including parameter extraction, serialization, and response format handling
test/providers/open_ai/responses/transforms_test.rb	Unit tests for OpenAI Responses API transforms covering input normalization, response format handling, and message simplification
test/providers/open_ai/embedding/transforms_test.rb	Unit tests for OpenAI Embedding transforms including input normalization, parameter handling, and serialization cleanup
test/providers/open_ai/chat/transforms_test.rb	Comprehensive unit tests for OpenAI Chat transforms covering message normalization, content handling, and instruction processing
test/providers/ollama/embedding/transforms_test.rb	Unit tests for Ollama Embedding transforms including Ollama-specific parameter extraction and input validation
test/providers/ollama/chat/transforms_test.rb	Unit tests for Ollama Chat transforms covering message grouping and Ollama-specific parameter handling
test/providers/common/usage_test.rb	Core unit tests for Usage model covering all provider formats, auto-detection, and field normalization
test/providers/anthropic/transforms_test.rb	Unit tests for Anthropic transforms covering message normalization, content handling, and response cleanup
test/integration/*	Integration tests with VCR cassettes validating real API response handling across all providers
test/fixtures/vcr_cassettes/*	VCR cassettes capturing real API responses for reliable integration testing
test/docs/providers/anthropic_examples_test.rb	Fixed assertion argument order to follow Minitest conventions
lib/active_agent/providers/open_router/requests/prediction.rb	Added comprehensive inline documentation for Prediction model
lib/active_agent/providers/open_router/requests/message.rb	Removed unused require statement

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[sirwolfgang]

Took some back and fourth with New Relic Support, but I'm now able to confirm that these Notifications allow you to correctly hook into all the details that are needed for at least supporting New Relic's APM.

[TonsOfFun]

I have heard honey badger support too

…

On Wed, Nov 19, 2025 at 5:42 PM Zane Pickett ***@***.***> wrote: *sirwolfgang* left a comment (activeagents/activeagent#274) <#274 (comment)> Took some back and fourth with New Relic Support, but I'm now able to confirm that these Notifications allow you to correctly hook into all the details that are needed for at least supporting New Relic's APM. — Reply to this email directly, view it on GitHub <#274 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAG4S3DMGDPTRWZ5VNH7JKD35UL7JAVCNFSM6AAAAACL6DW4PSVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZTKNJVGM4TAMRTGE> . You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>