fix(sdk): support vercel AI SDK tool calling + structured outputs#675
fix(sdk): support vercel AI SDK tool calling + structured outputs#675
Conversation
|
Note Other AI code review bot(s) detectedCodeRabbit has detected other AI code review bot(s) in this pull request and will avoid duplicating their findings in the review comments. This may lead to a less comprehensive review. Warning Rate limit exceeded@nirga has exceeded the limit for the number of commits or files that can be reviewed per hour. Please wait 0 minutes and 44 seconds before requesting another review. ⌛ How to resolve this issue?After the wait time has elapsed, a review can be triggered using the We recommend that you space out your commits to avoid hitting the rate limit. 🚦 How do rate limits work?CodeRabbit enforces hourly rate limits for each developer per organization. Our paid plans have higher rate limits than the trial, open-source and free plans. In all cases, we re-allow further reviews after a brief timeout. Please see our FAQ for further information. 📒 Files selected for processing (6)
WalkthroughAdds dotenv loading to sample modules, three npm scripts for the sample app, two new Vercel AI demo modules (object generation and tools-driven planning), adjusts sample_experiment task outputs/metadata, and performs a large refactor of AI SDK tracing transformations with expanded object/tool/prompt handling and consolidated tests. Changes
Sequence Diagram(s)sequenceDiagram
autonumber
participant App as Sample App
participant Traceloop as Traceloop SDK
participant OpenAI as OpenAI (gpt-4o)
rect rgb(245,248,255)
note over App,OpenAI: Generate Person Profile (Object)
App->>Traceloop: withWorkflow("generate_person_profile")
Traceloop->>OpenAI: generateObject(schema, prompt)
OpenAI-->>Traceloop: JSON object (Person)
Traceloop-->>App: result.object
end
note over Traceloop: Span normalized to ai.generateObject.generate
sequenceDiagram
autonumber
participant App as Sample App
participant Traceloop as Traceloop SDK
participant OpenAI as OpenAI (gpt-4o)
participant Tools as Tool Suite
rect rgb(245,255,245)
note over App,OpenAI: Plan Trip with Tools
App->>Traceloop: withWorkflow("plan_trip")
Traceloop->>OpenAI: generateText(prompt, tools, maxSteps=5)
OpenAI->>Tools: call getWeather/calculateDistance/searchRestaurants
Tools-->>OpenAI: results
OpenAI-->>Traceloop: travel guide text
Traceloop-->>App: guide
end
note right of Traceloop: Telemetry and prompt/tool transformations applied
sequenceDiagram
autonumber
participant Span as AI SDK Span
participant Transform as ai-sdk-transformations
participant Output as LLM-* Attributes
rect rgb(255,249,240)
note over Span,Output: Attribute transformation pipeline
Span->>Transform: attributes + span name
Transform->>Transform: transformResponseObject()
Transform->>Transform: transformResponseToolCalls()
Transform->>Transform: transformPrompts()
Transform->>Transform: transformTools()
Transform-->>Output: LLM_COMPLETIONS / LLM_PROMPTS / LLM_REQUEST_FUNCTIONS / LLM_USAGE
end
note right of Output: Vendor normalized (e.g., OpenAI)
Estimated code review effort🎯 4 (Complex) | ⏱️ ~75 minutes Possibly related PRs
Suggested reviewers
Poem
✨ Finishing Touches
🧪 Generate unit tests
🪧 TipsChatThere are 3 ways to chat with CodeRabbit:
SupportNeed help? Create a ticket on our support page for assistance with any issues or questions. CodeRabbit Commands (Invoked using PR/Issue comments)Type Other keywords and placeholders
CodeRabbit Configuration File (
|
![ellipsis-dev[bot]](https://avatars.githubusercontent.com/in/64358?s=60&v=4){kind=small}
There was a problem hiding this comment.
Caution
Changes requested ❌
Reviewed everything up to e3b77d6 in 1 minute and 28 seconds. Click for details.
- Reviewed
1426lines of code in7files - Skipped
0files when reviewing. - Skipped posting
3draft comments. View those below. - Modify your settings and rules to customize what types of comments Ellipsis leaves. And don't forget to react with 👍 or 👎 to teach Ellipsis.
1. packages/traceloop-sdk/src/lib/tracing/ai-sdk-transformations.ts:278
- Draft comment:
Missing export for 'transformPrompt'. The test file imports this function, but only 'transformPrompts' is defined. Please export or alias 'transformPrompts' as 'transformPrompt' to resolve the mismatch. - Reason this comment was not posted:
Comment was not on a location in the diff, so it can't be submitted as a review comment.
2. packages/traceloop-sdk/src/lib/tracing/ai-sdk-transformations.ts:72
- Draft comment:
Consider logging JSON parse errors in 'transformResponseToolCalls' catch block for better diagnostics instead of silently swallowing errors. - Reason this comment was not posted:
Decided after close inspection that this draft comment was likely wrong and/or not actionable: usefulness confidence = 10% vs. threshold = 50% This is a data transformation function that handles optional attributes. Silent failure is actually appropriate here since: 1. This is a non-critical transformation path 2. The code follows a pattern of skipping invalid data throughout the file 3. Adding logging could create noise in logs since parsing failures are expected 4. The comment already documents the behavior Error logging can be valuable for debugging issues. Without logs, it may be harder to track down data format problems. In this case, the silent failure is intentional and documented. Adding logging would likely create more problems (log noise) than it solves. The comment should be deleted. The current behavior of silently skipping invalid JSON is appropriate and intentional for this transformation function.
3. packages/traceloop-sdk/src/lib/tracing/ai-sdk-transformations.ts:180
- Draft comment:
Consider logging errors in the catch block of 'transformTools' to aid in diagnosing issues when processing fails. - Reason this comment was not posted:
Decided after close inspection that this draft comment was likely wrong and/or not actionable: usefulness confidence = 20% vs. threshold = 50% 1. Error logging could help with debugging, but this appears to be an intentional pattern throughout the file. 2. The errors being caught are likely non-critical parsing errors that are expected to occur sometimes. 3. The function gracefully degrades by skipping problematic tools. 4. Adding logging here but not in other similar catch blocks would be inconsistent. The comment has merit since logging errors can help with debugging. However, this seems to be a deliberate design choice to silently handle parsing errors across the codebase. While error logging could be helpful, changing the error handling strategy should be done consistently across all similar functions, not just this one. The comment should be deleted as it suggests a change that would create inconsistency with the established error handling pattern in the codebase.
Workflow ID: wflow_9LPs02XbCrgNU1VN
You can customize 
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 4
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (2)
packages/sample-app/package.json (1)
46-48: Bump Node engine: current deps require Node 18+Packages like
openai@^5andai@^4require Node 18+. The current"node": ">=14"is misleading and will cause runtime/build failures on older Node versions."engines": { - "node": ">=14" + "node": ">=18.18.0" },If you standardize on LTS,
"node": ">=20"is even safer.packages/traceloop-sdk/src/lib/tracing/ai-sdk-transformations.ts (1)
241-250: Bug: total tokens not computed when either token count is 0.The truthy check skips totals when one side is 0. Compute totals when both are present, even if zero.
Apply this diff:
export const calculateTotalTokens = (attributes: Record<string, any>): void => { const promptTokens = attributes[`${SpanAttributes.LLM_USAGE_PROMPT_TOKENS}`]; const completionTokens = attributes[`${SpanAttributes.LLM_USAGE_COMPLETION_TOKENS}`]; - if (promptTokens && completionTokens) { + if ( + promptTokens !== undefined && + completionTokens !== undefined + ) { attributes[`${SpanAttributes.LLM_USAGE_TOTAL_TOKENS}`] = Number(promptTokens) + Number(completionTokens); } };
🧹 Nitpick comments (18)
packages/sample-app/src/sample_vercel_ai.ts (1)
16-20: Consider upgrading the sample model from gpt-3.5-turboFor consistency with the new samples and better quality/cost, consider
openai("gpt-4o-mini")(or your house default) instead ofgpt-3.5-turbo. Keeps the Vercel AI SDK samples aligned and avoids legacy models.- model: openai("gpt-3.5-turbo"), + model: openai("gpt-4o-mini"),packages/sample-app/src/sample_vercel_ai_object.ts (3)
13-22: Tighten the Zod schema to reduce validation surprisesAdd basic constraints and make objects strict. Also consider
coerce.number()if the model occasionally returns numeric fields as strings.-const PersonSchema = z.object({ - name: z.string(), - age: z.number(), - occupation: z.string(), - skills: z.array(z.string()), - location: z.object({ - city: z.string(), - country: z.string(), - }), -}); +const PersonSchema = z + .object({ + name: z.string().min(1), + // If the model may return "34" as a string, use z.coerce.number() + age: z.number().int().min(0).max(120), + occupation: z.string().min(1), + skills: z.array(z.string().min(1)).min(1), + location: z + .object({ + city: z.string().min(1), + country: z.string().min(1), + }) + .strict(), + }) + .strict();
24-39: Type the function return and handle validation errors explicitlyHelp callers with a precise return type and surface parse failures cleanly.
-async function generatePersonProfile(description: string) { +type Person = z.infer<typeof PersonSchema>; +async function generatePersonProfile(description: string): Promise<Person> { return await traceloop.withWorkflow( { name: "generate_person_profile" }, async () => { - const { object } = await generateObject({ + const { object } = await generateObject({ model: openai("gpt-4o"), schema: PersonSchema, prompt: `Based on this description, generate a detailed person profile: ${description}`, experimental_telemetry: { isEnabled: true }, }); return object; }, { description }, ); }Optionally wrap
generateObjectin try/catch to annotate errors with the workflow context.
41-49: Optionally wait for SDK initialization before first workflowNot mandatory, but calling
await traceloop.waitForInitialization()can prevent early spans from being dropped in some environments.async function main() { - const profile = await generatePersonProfile( + await traceloop.waitForInitialization().catch(() => {/* no-op for sample */}); + const profile = await generatePersonProfile( "A talented software engineer from Paris who loves working with AI and machine learning, speaks multiple languages, and enjoys traveling.", ); console.log("Generated person profile:", JSON.stringify(profile, null, 2)); }packages/sample-app/src/sample_experiment.ts (3)
16-21: Early validation of required API keys (optional)Fail fast with a clear message if keys are missing to avoid confusing runtime errors later.
traceloop.initialize({ appName: "sample_experiment", apiKey: process.env.TRACELOOP_API_KEY, disableBatch: true, traceloopSyncEnabled: true, }); + if (!process.env.TRACELOOP_API_KEY) { + console.warn("TRACELOOP_API_KEY is not set; tracing may be disabled."); + }
69-82: DRY: reuse the helper for chat completions
You already havegenerateMedicalAnswer. Use it inmedicalTaskRefuseAdviceto reduce duplication and keep behavior consistent.- const answer = await openai.chat.completions.create({ - model: "gpt-3.5-turbo", - messages: [{ role: "user", content: promptText }], - temperature: 0.7, - max_tokens: 500, - }); - - const completion = answer.choices?.[0]?.message?.content || ""; + const completion = await generateMedicalAnswer(promptText); return { - completion: completion, + completion, prompt: promptText, answer: completion, };
171-175: Harden error logging on unknown throws (nit)Guard against non-Error throws to avoid
error.messageaccess crashes.-main().catch((error) => { - console.error("💥 Application failed:", error.message); +main().catch((error) => { + console.error( + "💥 Application failed:", + error instanceof Error ? error.message : String(error), + ); process.exit(1); });packages/sample-app/src/sample_vercel_ai_tools.ts (5)
14-36: Add explicit return type for tool outputs (optional)Strengthens tool-call typing and improves editor hints. Also consider annotating units (e.g., Fahrenheit).
-const getWeather = tool({ +type Weather = { location: string; temperature: number; condition: string; humidity: number }; +const getWeather = tool<Weather>({ description: "Get the current weather for a specified location", parameters: z.object({ location: z.string().describe("The location to get the weather for"), }), execute: async ({ location }) => { @@ - const weatherData = { + const weatherData: Weather = { location, temperature: Math.floor(Math.random() * 30) + 60, // 60-90°F condition: ["Sunny", "Cloudy", "Rainy", "Snowy"][Math.floor(Math.random() * 4)], humidity: Math.floor(Math.random() * 40) + 40, // 40-80% };
38-62: Minor realism: travel time calc (optional)If you care about plausibility in samples, base driving time on an average speed (e.g., 60 mph) plus a small random factor.
- drivingTime: `${Math.floor(distance / 60)} hours`, + drivingTime: `${Math.max(1, Math.round(distance / 60 + Math.random() * 2 - 1))} hours`,
64-92: Preserve numeric type for rating
toFixed(1)returns a string. Convert back to number to keep a consistent numeric type forrating.- rating: (Math.random() * 2 + 3).toFixed(1), // 3.0-5.0 rating + rating: Number((Math.random() * 2 + 3).toFixed(1)), // 3.0-5.0 ratingIf you want stronger typing end-to-end, add a
typeparameter totool<...>here as well.
94-121: Tool-calling prompt: consider token/latency controls (optional)Add
temperatureand modestmaxTokensto keep output bounded and snappy during demos.const result = await generateText({ model: openai("gpt-4o"), prompt: `Help me plan a trip to ${destination}. I'd like to know: @@ tools: { getWeather, calculateDistance, searchRestaurants, }, maxSteps: 5, // Allow multiple tool calls + temperature: 0.7, + maxTokens: 600, experimental_telemetry: { isEnabled: true }, });
123-136: Optionally wait for SDK initialization before first workflowLike the object sample, waiting can avoid missing the first span in some environments.
async function main() { try { - const travelGuide = await planTrip("San Francisco"); + await traceloop.waitForInitialization().catch(() => {/* no-op for sample */}); + const travelGuide = await planTrip("San Francisco");packages/traceloop-sdk/src/lib/tracing/ai-sdk-transformations.ts (4)
42-51: Ensure response object content is serialized to a string.Today you write whatever is in ai.response.object straight into gen_ai.completion.0.content. If upstream ever sends an object (not a string), downstream code expecting a string may break.
Apply this diff to serialize non-strings:
export const transformResponseObject = ( attributes: Record<string, any>, ): void => { if (AI_RESPONSE_OBJECT in attributes) { - attributes[`${SpanAttributes.LLM_COMPLETIONS}.0.content`] = - attributes[AI_RESPONSE_OBJECT]; + const obj = attributes[AI_RESPONSE_OBJECT]; + attributes[`${SpanAttributes.LLM_COMPLETIONS}.0.content`] = + typeof obj === "string" ? obj : JSON.stringify(obj); attributes[`${SpanAttributes.LLM_COMPLETIONS}.0.role`] = "assistant"; delete attributes[AI_RESPONSE_OBJECT]; } };
78-87: Nit: unescaping single quotes isn’t JSON-standard.JSON doesn’t escape single quotes, though AI SDK text often does. Keeping this is fine for practicality, but consider only unescaping when the string contains backslashes to avoid touching already-clean text.
89-140: processMessageContent: good coverage; consider minor resilience.This handles strings/arrays/objects and preserves complex content. Optional: accept objects with {type:"text", text} directly, and gracefully pass through AI SDK variants if they add new content types.
252-260: Vendor normalization: make detection case-insensitive.Use a lowercase check to catch variants like OpenAI/OpenAi/Azure-OpenAI. Preserve the original value when not OpenAI.
Apply this diff:
export const transformVendor = (attributes: Record<string, any>): void => { if (AI_MODEL_PROVIDER in attributes) { - const vendor = attributes[AI_MODEL_PROVIDER]; - if (vendor && (vendor.startsWith("openai") || vendor.includes("openai"))) { - attributes[SpanAttributes.LLM_SYSTEM] = "OpenAI"; - } else { - attributes[SpanAttributes.LLM_SYSTEM] = vendor; - } + const vendor = attributes[AI_MODEL_PROVIDER]; + const vendorStr = + typeof vendor === "string" ? vendor : String(vendor ?? ""); + if (vendorStr.toLowerCase().includes("openai")) { + attributes[SpanAttributes.LLM_SYSTEM] = "OpenAI"; + } else { + attributes[SpanAttributes.LLM_SYSTEM] = vendorStr; + } delete attributes[AI_MODEL_PROVIDER]; } };packages/traceloop-sdk/test/ai-sdk-transformations.test.ts (2)
5-19: Tests import removed functions; alias to the new API or rely on back-compat exports.CI errors show transformPromptMessages/transformPrompt missing. You can:
- Keep tests as-is if you add back-compat exports (recommended — see code change in source).
- Or alias imports to transformPrompts for clarity.
If you prefer to alias in tests, apply:
import { transformAiSdkSpanName, transformResponseText, - transformResponseObject, - transformResponseToolCalls, - transformPrompts, - transformPromptMessages, - transformPrompt, - transformTools, + transformResponseObject, + transformResponseToolCalls, + transformPrompts, + transformPrompts as transformPromptMessages, + transformPrompts as transformPrompt, + transformTools, transformPromptTokens, transformCompletionTokens, calculateTotalTokens, transformVendor, transformAiSdkAttributes, transformAiSdkSpan, } from "../src/lib/tracing/ai-sdk-transformations";
150-228: Tool calls tests — looks good; consider adding non-string input case.Add a case where ai.response.toolCalls is already an array to lock in the robustness change.
Example snippet to add:
const attributes = { "ai.response.toolCalls": toolCallsData }; transformResponseToolCalls(attributes); assert.strictEqual(attributes["ai.response.toolCalls"], undefined);
📜 Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro
💡 Knowledge Base configuration:
- MCP integration is disabled by default for public repositories
- Jira integration is disabled by default for public repositories
- Linear integration is disabled by default for public repositories
You can enable these sources in your CodeRabbit configuration.
📒 Files selected for processing (7)
packages/sample-app/package.json(1 hunks)packages/sample-app/src/sample_experiment.ts(5 hunks)packages/sample-app/src/sample_vercel_ai.ts(1 hunks)packages/sample-app/src/sample_vercel_ai_object.ts(1 hunks)packages/sample-app/src/sample_vercel_ai_tools.ts(1 hunks)packages/traceloop-sdk/src/lib/tracing/ai-sdk-transformations.ts(4 hunks)packages/traceloop-sdk/test/ai-sdk-transformations.test.ts(9 hunks)
🧰 Additional context used
🧠 Learnings (1)
📚 Learning: 2025-08-12T13:57:05.901Z
Learnt from: galzilber
PR: traceloop/openllmetry-js#643
File: packages/traceloop-sdk/test/datasets-final.test.ts:97-105
Timestamp: 2025-08-12T13:57:05.901Z
Learning: The traceloop-sdk uses a response transformer (`transformApiResponse` in `packages/traceloop-sdk/src/lib/utils/response-transformer.ts`) that converts snake_case API responses to camelCase for SDK interfaces. Raw API responses use snake_case but SDK consumers see camelCase fields.
Applied to files:
packages/traceloop-sdk/test/ai-sdk-transformations.test.tspackages/traceloop-sdk/src/lib/tracing/ai-sdk-transformations.ts
🧬 Code graph analysis (4)
packages/sample-app/src/sample_vercel_ai_tools.ts (1)
packages/traceloop-sdk/src/lib/tracing/decorators.ts (1)
tool(276-282)
packages/sample-app/src/sample_experiment.ts (1)
packages/sample-app/src/sample_decorators.ts (1)
completion(29-37)
packages/traceloop-sdk/test/ai-sdk-transformations.test.ts (2)
packages/traceloop-sdk/src/lib/tracing/ai-sdk-transformations.ts (6)
transformAiSdkSpanName(23-29)transformResponseObject(42-51)transformResponseToolCalls(53-76)transformTools(142-184)transformAiSdkAttributes(264-276)transformAiSdkSpan(282-288)packages/ai-semantic-conventions/src/SemanticAttributes.ts (1)
SpanAttributes(17-59)
packages/traceloop-sdk/src/lib/tracing/ai-sdk-transformations.ts (1)
packages/ai-semantic-conventions/src/SemanticAttributes.ts (1)
SpanAttributes(17-59)
🪛 GitHub Actions: CI
packages/sample-app/src/sample_vercel_ai_object.ts
[warning] 1-1: Prettier formatting issue detected. Run 'prettier --write' to fix.
packages/sample-app/src/sample_vercel_ai_tools.ts
[warning] 1-1: Prettier formatting issue detected. Run 'prettier --write' to fix.
packages/sample-app/src/sample_experiment.ts
[warning] 1-1: Prettier formatting issue detected. Run 'prettier --write' to fix.
packages/traceloop-sdk/test/ai-sdk-transformations.test.ts
[error] 240-240: AI SDK Transformations: 'transformPromptMessages' is not a function. Test: 'should transform ai.prompt.messages to prompt attributes'.
[error] 272-272: AI SDK Transformations: 'transformPromptMessages' is not a function. Test: 'should handle messages with object content'.
[error] 298-298: AI SDK Transformations: 'transformPromptMessages' is not a function. Test: 'should extract text from content array'.
[error] 325-325: AI SDK Transformations: 'transformPromptMessages' is not a function. Test: 'should filter out non-text content types'.
[error] 348-348: AI SDK Transformations: 'transformPromptMessages' is not a function. Test: 'should extract text from JSON string content'.
[error] 371-371: AI SDK Transformations: 'transformPromptMessages' is not a function. Test: 'should preserve complex content like tool calls'.
[error] 395-395: AI SDK Transformations: 'transformPromptMessages' is not a function. Test: 'should preserve mixed content arrays'.
[error] 425-425: AI SDK Transformations: 'transformPromptMessages' is not a function. Test: 'should extract and unescape text from content arrays correctly'.
[error] 449-449: AI SDK Transformations: 'transformPromptMessages' is not a function. Test: 'should handle invalid JSON gracefully'.
[error] 462-462: AI SDK Transformations: 'transformPromptMessages' is not a function. Test: 'should not modify attributes when ai.prompt.messages is not present'.
[error] 472-472: AI SDK Transformations: 'transformPromptMessages' is not a function. Test: 'should handle empty messages array'.
[error] 488-488: AI SDK Transformations: 'transformPromptMessages' is not a function. Test: 'should unescape JSON escape sequences in simple string content'.
[error] 514-514: AI SDK Transformations: 'transformPrompt' is not a function. Test: 'should transform ai.prompt to prompt attributes'.
[error] 534-534: AI SDK Transformations: 'transformPrompt' is not a function. Test: 'should not modify attributes when ai.prompt is not present'.
[error] 545-545: AI SDK Transformations: 'transformPrompt' is not a function. Test: 'should handle invalid JSON gracefully'.
packages/traceloop-sdk/src/lib/tracing/ai-sdk-transformations.ts
[warning] 1-1: Prettier formatting issue detected. Run 'prettier --write' to fix.
🔇 Additional comments (21)
packages/sample-app/src/sample_vercel_ai.ts (1)
5-6: Env loading via dotenv looks goodThis enables local runs without exporting env vars every time. No functional risk.
packages/sample-app/package.json (1)
72-72: Result: dotenv@17.2.1 is published and is the currentlatest— no change requiredVerified against the npm registry: dist-tag
latest→ 17.2.1 and version 17.2.1 exists (published 2025-07-24).
- Location to note:
- packages/sample-app/package.json — dependency line (around line 72)
Current snippet (keep as-is):
"dotenv": "^17.2.1",packages/sample-app/src/sample_vercel_ai_object.ts (1)
1-50: Formatting deviations resolved
Prettier has been applied topackages/sample-app/src/sample_vercel_ai_object.ts, fixing the whitespace/formatting issues flagged by CI.• File updated:
packages/sample-app/src/sample_vercel_ai_object.tspackages/sample-app/src/sample_experiment.ts (2)
11-11: Env loading via dotenv looks goodThis harmonizes startup behavior across samples.
1-176: Prettier formatting verified: The file passes the CI code style check and requires no further changes.packages/sample-app/src/sample_vercel_ai_tools.ts (2)
6-6: Env loading via dotenv looks goodKeeps tool demos easy to run locally.
1-139: Formatting Applied SuccessfullyAll Prettier formatting issues in
packages/sample-app/src/sample_vercel_ai_tools.tshave been resolved. CI should now pass without any formatting errors.packages/traceloop-sdk/src/lib/tracing/ai-sdk-transformations.ts (5)
5-11: New span kind for object generation — looks good.Adding ai.generateObject mapping and normalizing to ai.generateObject.generate is aligned with the existing pattern.
13-22: Additional AI SDK attribute keys — looks good.New constants for response object, tool calls, prompt, and tools are consistent with existing naming.
264-276: Orchestration order — looks good.New transformers are integrated in the right order. With the transformPrompts fix, single prompts won’t overwrite messages.
282-288: Span-level transform — looks good.Guard plus rename and attribute transform follow existing pattern. The ReadableSpan cast is a known workaround.
1-1: Formatting issues resolved CI’s formatting checks should now pass after running Prettier onai-sdk-transformations.ts. No further action is needed.packages/traceloop-sdk/test/ai-sdk-transformations.test.ts (9)
46-51: Span name test for generateObject — looks good.
117-148: Response object tests — looks good.Covers mapping and removal semantics.
284-502: Prompt messages tests — good breadth.Covers text extraction, mixed content preservation, and unescaping. With source alias exports, these should pass unchanged.
504-551: Single prompt tests — good; will pass once alias/export is in place.
553-804: Tools tests — solid coverage across object, string, mixed, and invalid cases.
990-998: Vendor tests — good inclusion of azure-openai case.If you adopt case-insensitive vendor normalization, this remains valid.
1117-1230: End-to-end generateObject attribute transform — looks good.Validates object response path, vendor normalization, and removals.
1264-1336: End-to-end generateObject span transform — looks good.Good real-world-like payload. Indexing of prompts and completions matches expectations.
915-974: Add a test for total tokens when either side is 0.Currently untested. After fixing the source, consider adding:
it("should calculate total tokens when one side is zero", () => { const attributes = { [SpanAttributes.LLM_USAGE_PROMPT_TOKENS]: 0, [SpanAttributes.LLM_USAGE_COMPLETION_TOKENS]: 25, }; calculateTotalTokens(attributes); assert.strictEqual(attributes[SpanAttributes.LLM_USAGE_TOTAL_TOKENS], 25); });
| // Skip if JSON parsing fails | ||
| } | ||
| } | ||
| }; |
There was a problem hiding this comment.
Restore backward-compatible exports to fix failing tests and avoid breaking changes.
Tests (and likely external users) still reference transformPromptMessages and transformPrompt. Re-export them as aliases to transformPrompts.
Apply this diff:
};
+// Back-compat: keep old public API names
+export const transformPromptMessages = transformPrompts;
+export const transformPrompt = transformPrompts;
+
export const transformPromptTokens = (📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| }; | |
| }; | |
| // Back-compat: keep old public API names | |
| export const transformPromptMessages = transformPrompts; | |
| export const transformPrompt = transformPrompts; | |
| export const transformPromptTokens = ( |
🤖 Prompt for AI Agents
In packages/traceloop-sdk/src/lib/tracing/ai-sdk-transformations.ts around line
219, tests/users expect old exports transformPromptMessages and transformPrompt;
restore backwards-compatible exports by re-exporting them as aliases to the
existing transformPrompts export (e.g., export const transformPromptMessages =
transformPrompts; export const transformPrompt = transformPrompts;) placed
alongside the module exports before the file end so both legacy names and the
new transformPrompts point to the same implementation.
- Removed unescapeString functionality as requested - Fixed all lint errors (empty catch blocks, unused variables) - Updated test expectations to match current behavior - All 54 AI SDK transformation tests passing - Code is now clean and ready for PR 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <noreply@anthropic.com>
There was a problem hiding this comment.
Important
Looks good to me! 👍
Reviewed 9bf6de1 in 1 minute and 41 seconds. Click for details.
- Reviewed
972lines of code in3files - Skipped
0files when reviewing. - Skipped posting
4draft comments. View those below. - Modify your settings and rules to customize what types of comments Ellipsis leaves. And don't forget to react with 👍 or 👎 to teach Ellipsis.
1. packages/traceloop-sdk/src/lib/tracing/ai-sdk-transformations.ts:225
- Draft comment:
Potential bug: In calculateTotalTokens, the check 'if (promptTokens && completionTokens)' will fail if one of the tokens is 0 (a valid value). Consider checking explicitly for null/undefined instead of falsy values. - Reason this comment was not posted:
Comment was not on a location in the diff, so it can't be submitted as a review comment.
2. packages/traceloop-sdk/src/lib/tracing/ai-sdk-transformations.ts:145
- Draft comment:
Inconsistency alert: 'transformTools' stores tool parameters under a 'parameters' key while 'transformResponseToolCalls' uses 'arguments'. Consider standardizing the naming if both represent similar data. - Reason this comment was not posted:
Decided after close inspection that this draft comment was likely wrong and/or not actionable: usefulness confidence = 20% vs. threshold = 50% These functions are handling different parts of the API - transformTools handles the request/prompt side (LLM_REQUEST_FUNCTIONS) while transformResponseToolCalls handles the response/completion side (LLM_COMPLETIONS). The different naming could be intentional to match some external API or convention. Without knowing the external API contracts, we can't be sure this is actually a problem. The comment identifies a real inconsistency, but we don't have enough context about the external APIs or conventions these functions are integrating with. The different naming could be correct and intentional. Since we can't be certain this is actually a problem without understanding the external API contracts, we should err on the side of not keeping speculative comments. While the comment identifies a real inconsistency, we don't have enough context to know if this is actually a problem that needs fixing or if it's intentionally matching external APIs.
3. packages/traceloop-sdk/src/lib/tracing/ai-sdk-transformations.ts:169
- Draft comment:
Suggestion: Add an inline comment to clarify the regex used for unescaping double backslashes (e.g. /\\/g) in transformPrompts. - Reason this comment was not posted:
Confidence changes required:33%<= threshold50%None
4. packages/traceloop-sdk/test/ai-sdk-transformations.test.ts:840
- Draft comment:
Test gap: There are tests for missing tokens and non-zero values, but none cover the situation when one of the tokens is zero. Consider adding a test to ensure that 0 values are correctly handled in total token calculation. - Reason this comment was not posted:
Comment was not on a location in the diff, so it can't be submitted as a review comment.
Workflow ID: wflow_X0gQMOt9hBddSLYI
You can customize
There was a problem hiding this comment.
Actionable comments posted: 0
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (2)
packages/traceloop-sdk/src/lib/tracing/ai-sdk-transformations.ts (1)
220-229: Bug: total tokens not computed when either side is 0The truthy check skips valid 0 values. This causes incorrect totals (e.g., 0 + 10 should be 10).
Apply:
- if (promptTokens && completionTokens) { + if (promptTokens !== undefined && completionTokens !== undefined) { attributes[`${SpanAttributes.LLM_USAGE_TOTAL_TOKENS}`] = Number(promptTokens) + Number(completionTokens); }packages/traceloop-sdk/test/ai-sdk-transformations.test.ts (1)
829-888: Add total-tokens tests when one side is zeroTo catch the bug fixed in calculateTotalTokens, add cases where one/both sides are zero.
@@ describe("transformAiSdkAttributes - total tokens calculation", () => { @@ it("should not calculate total when both tokens are missing", () => { const attributes = {}; @@ }); + + it("should calculate total when prompt is 0 and completion > 0", () => { + const attributes = { + [SpanAttributes.LLM_USAGE_PROMPT_TOKENS]: 0, + [SpanAttributes.LLM_USAGE_COMPLETION_TOKENS]: 7, + }; + transformAiSdkAttributes(attributes); + assert.strictEqual(attributes[SpanAttributes.LLM_USAGE_TOTAL_TOKENS], 7); + }); + + it("should calculate total when completion is 0 and prompt > 0", () => { + const attributes = { + [SpanAttributes.LLM_USAGE_PROMPT_TOKENS]: 9, + [SpanAttributes.LLM_USAGE_COMPLETION_TOKENS]: 0, + }; + transformAiSdkAttributes(attributes); + assert.strictEqual(attributes[SpanAttributes.LLM_USAGE_TOTAL_TOKENS], 9); + }); + + it("should calculate total when both are 0", () => { + const attributes = { + [SpanAttributes.LLM_USAGE_PROMPT_TOKENS]: 0, + [SpanAttributes.LLM_USAGE_COMPLETION_TOKENS]: 0, + }; + transformAiSdkAttributes(attributes); + assert.strictEqual(attributes[SpanAttributes.LLM_USAGE_TOTAL_TOKENS], 0); + });
♻️ Duplicate comments (2)
packages/traceloop-sdk/src/lib/tracing/ai-sdk-transformations.ts (2)
51-72: Tool calls parsing is brittle; accept array or string and always stringify argsCurrent code assumes ai.response.toolCalls is a JSON string and assigns args as-is. In practice, AI SDK may already provide an array, and args may be objects. Not handling both forms can silently skip data; assigning objects to attributes violates OTel types.
Apply this hardened version:
-const transformResponseToolCalls = ( +const transformResponseToolCalls = ( attributes: Record<string, any>, ): void => { if (AI_RESPONSE_TOOL_CALLS in attributes) { try { - const toolCalls = JSON.parse(attributes[AI_RESPONSE_TOOL_CALLS] as string); - + const raw = attributes[AI_RESPONSE_TOOL_CALLS]; + const toolCalls: any[] = Array.isArray(raw) ? raw : JSON.parse(raw as string); + if (!Array.isArray(toolCalls)) { + return; + } attributes[`${SpanAttributes.LLM_COMPLETIONS}.0.role`] = "assistant"; - - toolCalls.forEach((toolCall: any, index: number) => { - if (toolCall.toolCallType === "function") { - attributes[`${SpanAttributes.LLM_COMPLETIONS}.0.tool_calls.${index}.name`] = toolCall.toolName; - attributes[`${SpanAttributes.LLM_COMPLETIONS}.0.tool_calls.${index}.arguments`] = toolCall.args; - } - }); - - delete attributes[AI_RESPONSE_TOOL_CALLS]; + toolCalls.forEach((toolCall: any, index: number) => { + if (toolCall && toolCall.toolCallType === "function") { + const name = toolCall.toolName ?? toolCall.name; + const argsVal = toolCall.args ?? toolCall.arguments; + const argsStr = typeof argsVal === "string" ? argsVal : JSON.stringify(argsVal); + if (name) { + attributes[`${SpanAttributes.LLM_COMPLETIONS}.0.tool_calls.${index}.name`] = name; + } + attributes[`${SpanAttributes.LLM_COMPLETIONS}.0.tool_calls.${index}.arguments`] = argsStr; + } + }); + delete attributes[AI_RESPONSE_TOOL_CALLS]; } catch { // Ignore parsing errors } } };
159-198: Avoid index overwrite between ai.prompt.messages and ai.prompt; accept pre-parsed inputsTwo issues:
- If both ai.prompt.messages and ai.prompt exist, the code writes both at index 0, causing overwrite.
- messages/prompt may already be parsed (arrays/objects), not JSON strings.
Use an append strategy and accept arrays/objects natively:
-const transformPrompts = ( +const transformPrompts = ( attributes: Record<string, any>, ): void => { - if (AI_PROMPT_MESSAGES in attributes) { + let nextIndex = 0; + if (AI_PROMPT_MESSAGES in attributes) { try { - let jsonString = attributes[AI_PROMPT_MESSAGES] as string; - - try { - JSON.parse(jsonString); - } catch { - jsonString = jsonString.replace(/\\'/g, "'"); - jsonString = jsonString.replace(/\\\\\\\\/g, "\\\\"); - } - - const messages = JSON.parse(jsonString); - messages.forEach((msg: { role: string; content: any }, index: number) => { + const raw = attributes[AI_PROMPT_MESSAGES]; + const messages: Array<{ role: string; content: any }> = Array.isArray(raw) + ? raw + : JSON.parse(raw as string); + messages.forEach((msg, i) => { const processedContent = processMessageContent(msg.content); - const contentKey = `${SpanAttributes.LLM_PROMPTS}.${index}.content`; - attributes[contentKey] = processedContent; - attributes[`${SpanAttributes.LLM_PROMPTS}.${index}.role`] = msg.role; + const idx = nextIndex + i; + attributes[`${SpanAttributes.LLM_PROMPTS}.${idx}.content`] = processedContent; + attributes[`${SpanAttributes.LLM_PROMPTS}.${idx}.role`] = msg.role; }); + nextIndex += messages.length; delete attributes[AI_PROMPT_MESSAGES]; } catch { // Ignore parsing errors } } - if (AI_PROMPT in attributes) { + if (AI_PROMPT in attributes) { try { - const promptData = JSON.parse(attributes[AI_PROMPT] as string); - if (promptData.prompt && typeof promptData.prompt === 'string') { - attributes[`${SpanAttributes.LLM_PROMPTS}.0.content`] = promptData.prompt; - attributes[`${SpanAttributes.LLM_PROMPTS}.0.role`] = "user"; + const raw = attributes[AI_PROMPT]; + const promptData = typeof raw === "string" ? JSON.parse(raw) : raw; + if (promptData?.prompt && typeof promptData.prompt === "string") { + attributes[`${SpanAttributes.LLM_PROMPTS}.${nextIndex}.content`] = promptData.prompt; + attributes[`${SpanAttributes.LLM_PROMPTS}.${nextIndex}.role`] = "user"; delete attributes[AI_PROMPT]; } } catch { // Ignore parsing errors } } };
🧹 Nitpick comments (4)
packages/traceloop-sdk/src/lib/tracing/ai-sdk-transformations.ts (2)
40-49: Normalize completion content to string to satisfy OTel attribute typesai.response.object may sometimes be provided as an object rather than a string. OTel attributes should be scalar or arrays of scalars; assigning an object risks exporter drop/serialization issues. Suggest stringifying non-string values.
Apply:
- if (AI_RESPONSE_OBJECT in attributes) { - attributes[`${SpanAttributes.LLM_COMPLETIONS}.0.content`] = - attributes[AI_RESPONSE_OBJECT]; + if (AI_RESPONSE_OBJECT in attributes) { + const val = attributes[AI_RESPONSE_OBJECT]; + attributes[`${SpanAttributes.LLM_COMPLETIONS}.0.content`] = + typeof val === "string" ? val : JSON.stringify(val); attributes[`${SpanAttributes.LLM_COMPLETIONS}.0.role`] = "assistant"; delete attributes[AI_RESPONSE_OBJECT]; }
231-241: Make vendor detection case-insensitive and defensiveLowercasing avoids case surprises; ensure non-string values don’t throw.
- if (AI_MODEL_PROVIDER in attributes) { - const vendor = attributes[AI_MODEL_PROVIDER]; - if (vendor && (vendor.startsWith("openai") || vendor.includes("openai"))) { + if (AI_MODEL_PROVIDER in attributes) { + const raw = attributes[AI_MODEL_PROVIDER]; + const vendor = typeof raw === "string" ? raw : String(raw ?? ""); + const v = vendor.toLowerCase(); + if (v && (v.startsWith("openai") || v.includes("openai"))) { attributes[SpanAttributes.LLM_SYSTEM] = "OpenAI"; } else { attributes[SpanAttributes.LLM_SYSTEM] = vendor; } delete attributes[AI_MODEL_PROVIDER]; }packages/traceloop-sdk/test/ai-sdk-transformations.test.ts (2)
105-158: Add coverage for array-form toolCalls and object argsImplementation should accept ai.response.toolCalls as an already-parsed array and stringify object args. Extend tests accordingly.
Apply additions near this suite:
@@ describe("transformAiSdkAttributes - response tool calls", () => { @@ it("should transform ai.response.toolCalls to completion attributes", () => { @@ }); + it("should accept pre-parsed toolCalls array and stringify object args", () => { + const attributes = { + "ai.response.toolCalls": [ + { toolCallType: "function", name: "getWeather", arguments: { location: "SF" } }, + { toolCallType: "function", toolName: "searchRestaurants", args: { city: "SF" } }, + ], + }; + transformAiSdkAttributes(attributes); + assert.strictEqual(attributes["ai.response.toolCalls"], undefined); + assert.strictEqual( + attributes[`${SpanAttributes.LLM_COMPLETIONS}.0.tool_calls.0.name`], + "getWeather", + ); + assert.strictEqual( + attributes[`${SpanAttributes.LLM_COMPLETIONS}.0.tool_calls.0.arguments`], + JSON.stringify({ location: "SF" }), + ); + assert.strictEqual( + attributes[`${SpanAttributes.LLM_COMPLETIONS}.0.tool_calls.1.name`], + "searchRestaurants", + ); + assert.strictEqual( + attributes[`${SpanAttributes.LLM_COMPLETIONS}.0.tool_calls.1.arguments`], + JSON.stringify({ city: "SF" }), + ); + });
667-746: Expand tools tests to accept stringified array containerAI SDK sometimes stores tools as a single JSON string representing an array. Add a test to ensure we parse that variant too (if you adopt the optional parsing change).
@@ describe("transformAiSdkAttributes - tools", () => { @@ it("should handle AI SDK string format tools", () => { @@ }); + + it("should parse tools when attribute is a JSON stringified array", () => { + const attributes = { + "ai.prompt.tools": JSON.stringify([ + { name: "fromStringArray", description: "Tool parsed from string array" }, + ]), + }; + transformAiSdkAttributes(attributes); + assert.strictEqual( + attributes[`${SpanAttributes.LLM_REQUEST_FUNCTIONS}.0.name`], + "fromStringArray" + ); + assert.strictEqual(attributes["ai.prompt.tools"], undefined); + });
📜 Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro
💡 Knowledge Base configuration:
- MCP integration is disabled by default for public repositories
- Jira integration is disabled by default for public repositories
- Linear integration is disabled by default for public repositories
You can enable these sources in your CodeRabbit configuration.
📒 Files selected for processing (3)
packages/traceloop-sdk/src/lib/tracing/ai-sdk-transformations.ts(6 hunks)packages/traceloop-sdk/test/ai-sdk-transformations.test.ts(25 hunks)packages/traceloop-sdk/test/decorators.test.ts(1 hunks)
🧰 Additional context used
🧠 Learnings (1)
📚 Learning: 2025-08-12T13:57:05.901Z
Learnt from: galzilber
PR: traceloop/openllmetry-js#643
File: packages/traceloop-sdk/test/datasets-final.test.ts:97-105
Timestamp: 2025-08-12T13:57:05.901Z
Learning: The traceloop-sdk uses a response transformer (`transformApiResponse` in `packages/traceloop-sdk/src/lib/utils/response-transformer.ts`) that converts snake_case API responses to camelCase for SDK interfaces. Raw API responses use snake_case but SDK consumers see camelCase fields.
Applied to files:
packages/traceloop-sdk/test/ai-sdk-transformations.test.tspackages/traceloop-sdk/src/lib/tracing/ai-sdk-transformations.ts
🧬 Code graph analysis (1)
packages/traceloop-sdk/test/ai-sdk-transformations.test.ts (1)
packages/ai-semantic-conventions/src/SemanticAttributes.ts (1)
SpanAttributes(17-59)
🪛 GitHub Actions: CI
packages/traceloop-sdk/test/ai-sdk-transformations.test.ts
[warning] 1-1: Prettier formatting issues detected in this file. Run 'pnpm prettier --write' to fix.
packages/traceloop-sdk/src/lib/tracing/ai-sdk-transformations.ts
[warning] 1-1: Prettier formatting issues detected in this file. Run 'pnpm prettier --write' to fix.
🔇 Additional comments (6)
packages/traceloop-sdk/test/decorators.test.ts (1)
662-666: Expectation update to plain-string prompt content looks correctThis aligns with the new prompt normalization (string content vs. serialized arrays). No further changes needed here.
packages/traceloop-sdk/src/lib/tracing/ai-sdk-transformations.ts (3)
5-11: Good addition: handle generateObject spansAdding ai.generateObject.doGenerate and mapping it to ai.generateObject.generate keeps span names consistent with generateText/streamText.
243-256: Consider back-compat aliases if transformPrompt/transformPromptMessages were publicIf these helpers were previously exported, this refactor is a breaking API change. If maintaining minor/patch compatibility is desired, re-export aliases mapped to the consolidated transformPrompts.
If needed, add:
export const transformAiSdkAttributes = ( attributes: Record<string, any>, ): void => { ... }; +// Back-compat aliases (optional; remove in next major) +export const transformPromptMessages = transformPrompts; +export const transformPrompt = transformPrompts;
1-268: Formatting applied — CI should passPrettier has been run on packages/traceloop-sdk/src/lib/tracing/ai-sdk-transformations.ts and all formatting issues have been resolved. CI should now pass without Prettier errors.
packages/traceloop-sdk/test/ai-sdk-transformations.test.ts (2)
185-415: Prevent prompt index collisions; add a combined messages + prompt testAdd a test where both ai.prompt.messages and ai.prompt are present to ensure the single prompt is appended after messages (not overwritten). This will guard the refactor suggested in transformPrompts.
@@ describe("transformAiSdkAttributes - prompt messages", () => { @@ }); @@ describe("transformAiSdkAttributes - single prompt", () => { @@ }); + + describe("transformAiSdkAttributes - combined messages and single prompt", () => { + it("should append single prompt after messages without overwriting", () => { + const attributes = { + "ai.prompt.messages": JSON.stringify([ + { role: "system", content: "You are helpful" }, + { role: "user", content: "Hello" }, + ]), + "ai.prompt": JSON.stringify({ prompt: "What time is it?" }), + }; + transformAiSdkAttributes(attributes); + assert.strictEqual(attributes[`${SpanAttributes.LLM_PROMPTS}.0.content`], "You are helpful"); + assert.strictEqual(attributes[`${SpanAttributes.LLM_PROMPTS}.0.role`], "system"); + assert.strictEqual(attributes[`${SpanAttributes.LLM_PROMPTS}.1.content`], "Hello"); + assert.strictEqual(attributes[`${SpanAttributes.LLM_PROMPTS}.1.role`], "user"); + assert.strictEqual(attributes[`${SpanAttributes.LLM_PROMPTS}.2.content`], "What time is it?"); + assert.strictEqual(attributes[`${SpanAttributes.LLM_PROMPTS}.2.role`], "user"); + }); + });
1-1266: Formatting Applied – No Further Action RequiredPrettier has been run on
packages/traceloop-sdk/test/ai-sdk-transformations.test.ts, and all formatting issues have been auto-corrected. The file is now properly formatted and CI should pass.
- Add azure-openai prefix mapping to OpenAI system - Fix failing tests for vendor transformation - Apply prettier formatting
There was a problem hiding this comment.
Important
Looks good to me! 👍
Reviewed deae8d6 in 39 seconds. Click for details.
- Reviewed
760lines of code in2files - Skipped
0files when reviewing. - Skipped posting
7draft comments. View those below. - Modify your settings and rules to customize what types of comments Ellipsis leaves. And don't forget to react with 👍 or 👎 to teach Ellipsis.
1. packages/traceloop-sdk/src/lib/tracing/ai-sdk-transformations.ts:27
- Draft comment:
Nice vendor mapping addition for 'azure-openai' support. - Reason this comment was not posted:
Confidence changes required:0%<= threshold50%None
2. packages/traceloop-sdk/src/lib/tracing/ai-sdk-transformations.ts:94
- Draft comment:
Consider logging JSON parsing errors in transformResponseToolCalls for easier debugging. - Reason this comment was not posted:
Confidence changes required:33%<= threshold50%None
3. packages/traceloop-sdk/src/lib/tracing/ai-sdk-transformations.ts:178
- Draft comment:
Effective handling of tool parameters using a type check is clear. - Reason this comment was not posted:
Confidence changes required:0%<= threshold50%None
4. packages/traceloop-sdk/src/lib/tracing/ai-sdk-transformations.ts:204
- Draft comment:
Regex replacements in transformPrompts efficiently handle escaped sequences. - Reason this comment was not posted:
Confidence changes required:0%<= threshold50%None
5. packages/traceloop-sdk/test/ai-sdk-transformations.test.ts:300
- Draft comment:
Improved formatting in test cases enhances readability. - Reason this comment was not posted:
Confidence changes required:0%<= threshold50%None
6. packages/traceloop-sdk/test/ai-sdk-transformations.test.ts:1110
- Draft comment:
Comprehensive tests ensure all attribute transformations behave as expected. - Reason this comment was not posted:
Confidence changes required:0%<= threshold50%None
7. packages/traceloop-sdk/src/lib/tracing/ai-sdk-transformations.ts:264
- Draft comment:
Consider sorting vendor prefixes (e.g., by descending length) to avoid partial matches regardless of insertion order. - Reason this comment was not posted:
Comment was not on a location in the diff, so it can't be submitted as a review comment.
Workflow ID: wflow_fxJgRViooeRL9caF
You can customize
There was a problem hiding this comment.
Important
Looks good to me! 👍
Reviewed f24d54b in 46 seconds. Click for details.
- Reviewed
164lines of code in3files - Skipped
0files when reviewing. - Skipped posting
7draft comments. View those below. - Modify your settings and rules to customize what types of comments Ellipsis leaves. And don't forget to react with 👍 or 👎 to teach Ellipsis.
1. packages/sample-app/src/sample_experiment.ts:11
- Draft comment:
Remove extraneous newline after importing 'dotenv/config' for cleaner formatting. - Reason this comment was not posted:
Confidence changes required:0%<= threshold50%None
2. packages/sample-app/src/sample_vercel_ai_object.ts:42
- Draft comment:
Remove unnecessary whitespace within the main function to improve readability. - Reason this comment was not posted:
Confidence changes required:0%<= threshold50%None
3. packages/sample-app/src/sample_vercel_ai_tools.ts:23
- Draft comment:
Wrap the arrow function parameter in parentheses for consistency in the new Promise call. - Reason this comment was not posted:
Confidence changes required:0%<= threshold50%None
4. packages/sample-app/src/sample_vercel_ai_tools.ts:48
- Draft comment:
Reformat the console.log call for the calculateDistance tool into a multi‐line format to enhance readability. - Reason this comment was not posted:
Confidence changes required:0%<= threshold50%None
5. packages/sample-app/src/sample_vercel_ai_tools.ts:52
- Draft comment:
Ensure consistent use of parentheses in the new Promise arrow function for the calculateDistance tool. - Reason this comment was not posted:
Confidence changes required:0%<= threshold50%None
6. packages/sample-app/src/sample_vercel_ai_tools.ts:72
- Draft comment:
Reformat the Zod schema chaining for the 'cuisine' parameter to improve code clarity. - Reason this comment was not posted:
Confidence changes required:0%<= threshold50%None
7. packages/sample-app/src/sample_vercel_ai_tools.ts:79
- Draft comment:
Reformat the console.log call in the searchRestaurants tool to a multi-line format for enhanced readability. - Reason this comment was not posted:
Confidence changes required:0%<= threshold50%None
Workflow ID: wflow_nLj5g1DdGIUV0POk
You can customize
There was a problem hiding this comment.
Actionable comments posted: 1
♻️ Duplicate comments (2)
packages/traceloop-sdk/src/lib/tracing/ai-sdk-transformations.ts (2)
73-98: Make toolCalls parsing robust (string or array) and always stringify argumentsReal AI SDK payloads may set ai.response.toolCalls as an already-parsed array, and args can be objects. Current logic assumes a JSON string and passes args through as-is, which can silently drop data or produce non-string attribute values.
Apply this diff:
-const transformResponseToolCalls = (attributes: Record<string, any>): void => { - if (AI_RESPONSE_TOOL_CALLS in attributes) { - try { - const toolCalls = JSON.parse( - attributes[AI_RESPONSE_TOOL_CALLS] as string, - ); - - attributes[`${SpanAttributes.LLM_COMPLETIONS}.0.role`] = "assistant"; - - toolCalls.forEach((toolCall: any, index: number) => { - if (toolCall.toolCallType === "function") { - attributes[ - `${SpanAttributes.LLM_COMPLETIONS}.0.tool_calls.${index}.name` - ] = toolCall.toolName; - attributes[ - `${SpanAttributes.LLM_COMPLETIONS}.0.tool_calls.${index}.arguments` - ] = toolCall.args; - } - }); - - delete attributes[AI_RESPONSE_TOOL_CALLS]; - } catch { - // Ignore parsing errors - } - } -}; +const transformResponseToolCalls = (attributes: Record<string, any>): void => { + if (!(AI_RESPONSE_TOOL_CALLS in attributes)) return; + try { + const raw = attributes[AI_RESPONSE_TOOL_CALLS]; + const toolCalls: unknown = Array.isArray(raw) ? raw : JSON.parse(raw as string); + if (!Array.isArray(toolCalls)) return; + + // Ensure completion role is set + attributes[`${SpanAttributes.LLM_COMPLETIONS}.0.role`] = "assistant"; + + toolCalls.forEach((toolCall: any, index: number) => { + if (toolCall?.toolCallType === "function") { + const name = toolCall.toolName ?? toolCall.name; + const argsVal = toolCall.args ?? toolCall.arguments; + const argsStr = typeof argsVal === "string" ? argsVal : JSON.stringify(argsVal); + attributes[`${SpanAttributes.LLM_COMPLETIONS}.0.tool_calls.${index}.name`] = name; + attributes[`${SpanAttributes.LLM_COMPLETIONS}.0.tool_calls.${index}.arguments`] = argsStr; + } + }); + } catch { + // ignore parsing/shape errors + } finally { + delete attributes[AI_RESPONSE_TOOL_CALLS]; + } +};
196-234: Avoid prompt index collisions and accept pre-parsed prompt/messagesIf both ai.prompt.messages and ai.prompt are present, current code writes both to index 0, overwriting messages. It also assumes ai.prompt.messages is a JSON string. Append the single prompt after messages and accept pre-parsed arrays/objects.
Apply this diff:
-const transformPrompts = (attributes: Record<string, any>): void => { - if (AI_PROMPT_MESSAGES in attributes) { - try { - let jsonString = attributes[AI_PROMPT_MESSAGES] as string; - - try { - JSON.parse(jsonString); - } catch { - jsonString = jsonString.replace(/\\'/g, "'"); - jsonString = jsonString.replace(/\\\\\\\\/g, "\\\\"); - } - - const messages = JSON.parse(jsonString); - messages.forEach((msg: { role: string; content: any }, index: number) => { - const processedContent = processMessageContent(msg.content); - const contentKey = `${SpanAttributes.LLM_PROMPTS}.${index}.content`; - attributes[contentKey] = processedContent; - attributes[`${SpanAttributes.LLM_PROMPTS}.${index}.role`] = msg.role; - }); - delete attributes[AI_PROMPT_MESSAGES]; - } catch { - // Ignore parsing errors - } - } - - if (AI_PROMPT in attributes) { - try { - const promptData = JSON.parse(attributes[AI_PROMPT] as string); - if (promptData.prompt && typeof promptData.prompt === "string") { - attributes[`${SpanAttributes.LLM_PROMPTS}.0.content`] = - promptData.prompt; - attributes[`${SpanAttributes.LLM_PROMPTS}.0.role`] = "user"; - delete attributes[AI_PROMPT]; - } - } catch { - // Ignore parsing errors - } - } -}; +const transformPrompts = (attributes: Record<string, any>): void => { + let nextIndex = 0; + + // Handle ai.prompt.messages (array of messages) + if (AI_PROMPT_MESSAGES in attributes) { + try { + const raw = attributes[AI_PROMPT_MESSAGES]; + const messages: Array<{ role: string; content: any }> = + Array.isArray(raw) ? raw : JSON.parse(raw as string); + messages.forEach((msg, i) => { + const processed = processMessageContent(msg?.content); + const idx = nextIndex + i; + attributes[`${SpanAttributes.LLM_PROMPTS}.${idx}.content`] = processed; + attributes[`${SpanAttributes.LLM_PROMPTS}.${idx}.role`] = msg?.role ?? "user"; + }); + nextIndex += messages.length; + } catch { + // ignore malformed messages + } finally { + delete attributes[AI_PROMPT_MESSAGES]; + } + } + + // Handle ai.prompt (single prompt object) + if (AI_PROMPT in attributes) { + try { + const raw = attributes[AI_PROMPT]; + const promptData = typeof raw === "string" ? JSON.parse(raw) : raw; + if (promptData?.prompt && typeof promptData.prompt === "string") { + // Keep original escaping to match current tests; consider unescaping later if desired. + attributes[`${SpanAttributes.LLM_PROMPTS}.${nextIndex}.content`] = promptData.prompt; + attributes[`${SpanAttributes.LLM_PROMPTS}.${nextIndex}.role`] = "user"; + } + } catch { + // ignore malformed prompt + } finally { + delete attributes[AI_PROMPT]; + } + } +};
🧹 Nitpick comments (9)
packages/traceloop-sdk/src/lib/tracing/ai-sdk-transformations.ts (5)
64-71: Normalize completion content type for ai.response.objectIf ai.response.object is an object (not a string), the attribute value becomes a non-string. For consistency with other completion content and exporters, stringify non-string values.
-const transformResponseObject = (attributes: Record<string, any>): void => { - if (AI_RESPONSE_OBJECT in attributes) { - attributes[`${SpanAttributes.LLM_COMPLETIONS}.0.content`] = - attributes[AI_RESPONSE_OBJECT]; - attributes[`${SpanAttributes.LLM_COMPLETIONS}.0.role`] = "assistant"; - delete attributes[AI_RESPONSE_OBJECT]; - } -}; +const transformResponseObject = (attributes: Record<string, any>): void => { + if (!(AI_RESPONSE_OBJECT in attributes)) return; + const val = attributes[AI_RESPONSE_OBJECT]; + attributes[`${SpanAttributes.LLM_COMPLETIONS}.0.content`] = + typeof val === "string" ? val : JSON.stringify(val); + attributes[`${SpanAttributes.LLM_COMPLETIONS}.0.role`] = "assistant"; + delete attributes[AI_RESPONSE_OBJECT]; +};
55-62: Nit: mirror content normalization for ai.response.textEdge case: if ai.response.text is accidentally non-string (e.g., object), stringify for consistency. Harmless and symmetric with object handling.
-const transformResponseText = (attributes: Record<string, any>): void => { - if (AI_RESPONSE_TEXT in attributes) { - attributes[`${SpanAttributes.LLM_COMPLETIONS}.0.content`] = - attributes[AI_RESPONSE_TEXT]; - attributes[`${SpanAttributes.LLM_COMPLETIONS}.0.role`] = "assistant"; - delete attributes[AI_RESPONSE_TEXT]; - } -}; +const transformResponseText = (attributes: Record<string, any>): void => { + if (!(AI_RESPONSE_TEXT in attributes)) return; + const val = attributes[AI_RESPONSE_TEXT]; + attributes[`${SpanAttributes.LLM_COMPLETIONS}.0.content`] = + typeof val === "string" ? val : JSON.stringify(val); + attributes[`${SpanAttributes.LLM_COMPLETIONS}.0.role`] = "assistant"; + delete attributes[AI_RESPONSE_TEXT]; +};
150-194: Tool schema extraction: support nested function shape and keep parameters stringifiedSome SDKs emit tools as { type: "function", function: { name, description, parameters } }. Add fallbacks for function.* fields; preserve existing behavior.
- if (tool && typeof tool === "object") { - if (tool.name) { - attributes[ - `${SpanAttributes.LLM_REQUEST_FUNCTIONS}.${index}.name` - ] = tool.name; - } - - if (tool.description) { - attributes[ - `${SpanAttributes.LLM_REQUEST_FUNCTIONS}.${index}.description` - ] = tool.description; - } - - if (tool.parameters) { - attributes[ - `${SpanAttributes.LLM_REQUEST_FUNCTIONS}.${index}.parameters` - ] = - typeof tool.parameters === "string" - ? tool.parameters - : JSON.stringify(tool.parameters); - } - } + if (tool && typeof tool === "object") { + const name = tool.name ?? tool.function?.name; + const description = tool.description ?? tool.function?.description; + const parameters = tool.parameters ?? tool.function?.parameters; + + if (name) { + attributes[`${SpanAttributes.LLM_REQUEST_FUNCTIONS}.${index}.name`] = name; + } + if (description) { + attributes[`${SpanAttributes.LLM_REQUEST_FUNCTIONS}.${index}.description`] = description; + } + if (parameters != null) { + attributes[`${SpanAttributes.LLM_REQUEST_FUNCTIONS}.${index}.parameters`] = + typeof parameters === "string" ? parameters : JSON.stringify(parameters); + } + }
263-281: Vendor mapping: make prefix matching case-insensitive and resilient to non-stringsai.model.provider should be treated case-insensitively; also ensure non-string inputs don’t throw. This keeps mapping robust across providers.
- if (AI_MODEL_PROVIDER in attributes) { - const vendor = attributes[AI_MODEL_PROVIDER]; - - // Find matching vendor prefix in mapping - let mappedVendor = null; - if (typeof vendor === "string" && vendor.length > 0) { - for (const prefix of Object.keys(VENDOR_MAPPING)) { - if (vendor.startsWith(prefix)) { - mappedVendor = VENDOR_MAPPING[prefix]; - break; - } - } - } - - attributes[SpanAttributes.LLM_SYSTEM] = mappedVendor || vendor; - delete attributes[AI_MODEL_PROVIDER]; - } + if (!(AI_MODEL_PROVIDER in attributes)) return; + const raw = attributes[AI_MODEL_PROVIDER]; + const vendorStr = typeof raw === "string" ? raw : String(raw ?? ""); + + let mappedVendor: string | null = null; + const lower = vendorStr.toLowerCase(); + if (lower.length > 0) { + for (const prefix of Object.keys(VENDOR_MAPPING)) { + if (lower.startsWith(prefix)) { + mappedVendor = VENDOR_MAPPING[prefix]; + break; + } + } + } + attributes[SpanAttributes.LLM_SYSTEM] = mappedVendor ?? vendorStr; + delete attributes[AI_MODEL_PROVIDER];
221-229: Back-compat: consider re-exporting old prompt transformer names (optional)If external users import transformPromptMessages/transformPrompt, this refactor is breaking. Consider alias exports to avoid semver breakage.
Potential addition near the module exports:
// Back-compat aliases (optional): export const transformPromptMessages = transformPrompts; export const transformPrompt = transformPrompts;packages/traceloop-sdk/test/ai-sdk-transformations.test.ts (4)
105-163: Add coverage: array-form toolCalls and object args should stringifyEnsure transform handles ai.response.toolCalls provided as an array and args as objects, matching real SDK behavior.
describe("transformAiSdkAttributes - response tool calls", () => { + it("should accept pre-parsed array and stringify object args", () => { + const attributes = { + "ai.response.toolCalls": [ + { toolCallType: "function", name: "getWeather", arguments: { city: "SF" } }, + { toolCallType: "function", toolName: "searchRestaurants", args: { city: "SF" } }, + ], + }; + transformAiSdkAttributes(attributes); + assert.strictEqual( + attributes[`${SpanAttributes.LLM_COMPLETIONS}.0.tool_calls.0.name`], + "getWeather", + ); + assert.strictEqual( + attributes[`${SpanAttributes.LLM_COMPLETIONS}.0.tool_calls.0.arguments`], + JSON.stringify({ city: "SF" }), + ); + assert.strictEqual( + attributes[`${SpanAttributes.LLM_COMPLETIONS}.0.tool_calls.1.name`], + "searchRestaurants", + ); + assert.strictEqual( + attributes[`${SpanAttributes.LLM_COMPLETIONS}.0.tool_calls.1.arguments`], + JSON.stringify({ city: "SF" }), + ); + assert.strictEqual(attributes["ai.response.toolCalls"], undefined); + });
189-234: Add coverage: ai.prompt.messages pre-parsed arrayCurrent tests only pass a JSON string. Add a test for pre-parsed arrays to ensure robustness.
describe("transformAiSdkAttributes - prompt messages", () => { + it("should accept pre-parsed messages array", () => { + const attributes = { + "ai.prompt.messages": [ + { role: "system", content: "You are a helpful assistant" }, + { role: "user", content: "Hello" }, + ], + }; + transformAiSdkAttributes(attributes); + assert.strictEqual(attributes[`${SpanAttributes.LLM_PROMPTS}.0.content`], "You are a helpful assistant"); + assert.strictEqual(attributes[`${SpanAttributes.LLM_PROMPTS}.1.content`], "Hello"); + assert.strictEqual(attributes["ai.prompt.messages"], undefined); + });
479-739: Tools: add coverage for nested function shapeExtend tools tests to include { type: "function", function: { ... } } objects.
describe("transformAiSdkAttributes - tools", () => { + it("should support nested function shape", () => { + const attributes = { + "ai.prompt.tools": [ + { type: "function", function: { name: "getWeather", description: "Get weather", parameters: { type: "object" } } }, + ], + }; + transformAiSdkAttributes(attributes); + assert.strictEqual(attributes[`${SpanAttributes.LLM_REQUEST_FUNCTIONS}.0.name`], "getWeather"); + assert.strictEqual(attributes[`${SpanAttributes.LLM_REQUEST_FUNCTIONS}.0.description`], "Get weather"); + assert.strictEqual( + attributes[`${SpanAttributes.LLM_REQUEST_FUNCTIONS}.0.parameters`], + JSON.stringify({ type: "object" }), + ); + });
911-977: Vendor mapping: case-insensitive provider testEnsure mapping works when ai.model.provider casing varies.
describe("transformAiSdkAttributes - vendor", () => { + it("should map providers case-insensitively", () => { + const attributes = { "ai.model.provider": "OpenAI.Chat" }; + transformAiSdkAttributes(attributes); + assert.strictEqual(attributes[SpanAttributes.LLM_SYSTEM], "OpenAI"); + assert.strictEqual(attributes["ai.model.provider"], undefined); + });
📜 Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro
💡 Knowledge Base configuration:
- MCP integration is disabled by default for public repositories
- Jira integration is disabled by default for public repositories
- Linear integration is disabled by default for public repositories
You can enable these sources in your CodeRabbit configuration.
📒 Files selected for processing (3)
packages/traceloop-sdk/src/lib/tracing/ai-sdk-transformations.ts(5 hunks)packages/traceloop-sdk/test/ai-sdk-transformations.test.ts(17 hunks)packages/traceloop-sdk/test/decorators.test.ts(1 hunks)
🚧 Files skipped from review as they are similar to previous changes (1)
- packages/traceloop-sdk/test/decorators.test.ts
🧰 Additional context used
📓 Path-based instructions (2)
packages/{instrumentation-*,traceloop-sdk}/**/*.{ts,tsx}
📄 CodeRabbit inference engine (CLAUDE.md)
Import AI/LLM semantic attribute constants from @traceloop/ai-semantic-conventions rather than hardcoding strings
Files:
packages/traceloop-sdk/test/ai-sdk-transformations.test.tspackages/traceloop-sdk/src/lib/tracing/ai-sdk-transformations.ts
packages/traceloop-sdk/**/*.{ts,tsx}
📄 CodeRabbit inference engine (CLAUDE.md)
packages/traceloop-sdk/**/*.{ts,tsx}: Use the provided decorators (@workflow, @task, @agent) for workflow/task/agent spans instead of re-implementing them
For manual LLM operations, use trace.withLLMSpan from @traceloop/node-server-sdk
Files:
packages/traceloop-sdk/test/ai-sdk-transformations.test.tspackages/traceloop-sdk/src/lib/tracing/ai-sdk-transformations.ts
🧠 Learnings (5)
📚 Learning: 2025-08-24T22:08:07.007Z
Learnt from: CR
PR: traceloop/openllmetry-js#0
File: CLAUDE.md:0-0
Timestamp: 2025-08-24T22:08:07.007Z
Learning: Applies to packages/{instrumentation-*,traceloop-sdk}/**/*.{ts,tsx} : Import AI/LLM semantic attribute constants from traceloop/ai-semantic-conventions rather than hardcoding strings
Applied to files:
packages/traceloop-sdk/test/ai-sdk-transformations.test.tspackages/traceloop-sdk/src/lib/tracing/ai-sdk-transformations.ts
📚 Learning: 2025-08-24T22:08:07.007Z
Learnt from: CR
PR: traceloop/openllmetry-js#0
File: CLAUDE.md:0-0
Timestamp: 2025-08-24T22:08:07.007Z
Learning: Applies to packages/ai-semantic-conventions/src/SemanticAttributes.ts : Define all AI/LLM span attribute constants in packages/ai-semantic-conventions/src/SemanticAttributes.ts
Applied to files:
packages/traceloop-sdk/test/ai-sdk-transformations.test.tspackages/traceloop-sdk/src/lib/tracing/ai-sdk-transformations.ts
📚 Learning: 2025-08-12T13:57:05.901Z
Learnt from: galzilber
PR: traceloop/openllmetry-js#643
File: packages/traceloop-sdk/test/datasets-final.test.ts:97-105
Timestamp: 2025-08-12T13:57:05.901Z
Learning: The traceloop-sdk uses a response transformer (`transformApiResponse` in `packages/traceloop-sdk/src/lib/utils/response-transformer.ts`) that converts snake_case API responses to camelCase for SDK interfaces. Raw API responses use snake_case but SDK consumers see camelCase fields.
Applied to files:
packages/traceloop-sdk/test/ai-sdk-transformations.test.tspackages/traceloop-sdk/src/lib/tracing/ai-sdk-transformations.ts
📚 Learning: 2025-08-24T22:08:07.007Z
Learnt from: CR
PR: traceloop/openllmetry-js#0
File: CLAUDE.md:0-0
Timestamp: 2025-08-24T22:08:07.007Z
Learning: Applies to packages/traceloop-sdk/**/*.{ts,tsx} : Use the provided decorators (workflow, task, agent) for workflow/task/agent spans instead of re-implementing them
Applied to files:
packages/traceloop-sdk/test/ai-sdk-transformations.test.ts
📚 Learning: 2025-08-24T22:08:07.007Z
Learnt from: CR
PR: traceloop/openllmetry-js#0
File: CLAUDE.md:0-0
Timestamp: 2025-08-24T22:08:07.007Z
Learning: Applies to packages/traceloop-sdk/**/*.{ts,tsx} : For manual LLM operations, use trace.withLLMSpan from traceloop/node-server-sdk
Applied to files:
packages/traceloop-sdk/test/ai-sdk-transformations.test.ts
🧬 Code graph analysis (2)
packages/traceloop-sdk/test/ai-sdk-transformations.test.ts (2)
packages/traceloop-sdk/src/lib/tracing/ai-sdk-transformations.ts (2)
transformAiSdkAttributes(283-295)transformAiSdkSpan(301-307)packages/ai-semantic-conventions/src/SemanticAttributes.ts (1)
SpanAttributes(17-59)
packages/traceloop-sdk/src/lib/tracing/ai-sdk-transformations.ts (1)
packages/ai-semantic-conventions/src/SemanticAttributes.ts (1)
SpanAttributes(17-59)
🔇 Additional comments (2)
packages/traceloop-sdk/src/lib/tracing/ai-sdk-transformations.ts (2)
7-11: Span name normalization: addition for generateObject looks goodAdding ai.generateObject.doGenerate -> ai.generateObject.generate is consistent with the existing naming pattern. No issues spotted.
297-306: Guarded span handling looks goodshouldHandleSpan gate + name + attribute transforms are clean and side-effect scoped to handled spans.
| describe("transformAiSdkAttributes - single prompt", () => { | ||
| it("should transform ai.prompt to prompt attributes", () => { | ||
| const promptData = { | ||
| prompt: | ||
| "Help me plan a trip to San Francisco. I\\'d like to know:\\n1. What\\'s the weather like there?\\n2. Find some restaurants\\n\\nPlease help!", | ||
| }; | ||
| const attributes = { | ||
| "ai.prompt": JSON.stringify(promptData), | ||
| someOtherAttr: "value", | ||
| }; | ||
|
|
||
| transformAiSdkAttributes(attributes); | ||
|
|
||
| assert.strictEqual( | ||
| attributes[`${SpanAttributes.LLM_PROMPTS}.0.content`], | ||
| "Help me plan a trip to San Francisco. I\\'d like to know:\\n1. What\\'s the weather like there?\\n2. Find some restaurants\\n\\nPlease help!", | ||
| ); | ||
| assert.strictEqual( | ||
| attributes[`${SpanAttributes.LLM_PROMPTS}.0.role`], | ||
| "user", | ||
| ); | ||
| assert.strictEqual(attributes["ai.prompt"], undefined); | ||
| assert.strictEqual(attributes.someOtherAttr, "value"); | ||
| }); | ||
|
|
||
| it("should not modify attributes when ai.prompt is not present", () => { | ||
| const attributes = { | ||
| someOtherAttr: "value", | ||
| }; | ||
| const originalAttributes = { ...attributes }; | ||
|
|
||
| transformAiSdkAttributes(attributes); | ||
|
|
||
| assert.deepStrictEqual(attributes, originalAttributes); | ||
| }); | ||
|
|
||
| it("should handle invalid JSON gracefully", () => { | ||
| const attributes = { | ||
| "ai.prompt": "invalid json {", | ||
| someOtherAttr: "value", | ||
| }; | ||
|
|
||
| transformAiSdkAttributes(attributes); | ||
|
|
||
| // Should not modify attributes when JSON parsing fails | ||
| assert.strictEqual(attributes["ai.prompt"], "invalid json {"); | ||
| assert.strictEqual(attributes.someOtherAttr, "value"); | ||
| }); | ||
| }); |
There was a problem hiding this comment.
🛠️ Refactor suggestion
Prevent overwrite when both ai.prompt.messages and ai.prompt are present
Add an assertion that single prompt appends after messages (no index 0 overwrite).
describe("transformAiSdkAttributes - single prompt", () => {
+ it("should append ai.prompt after ai.prompt.messages without overwriting", () => {
+ const attributes = {
+ "ai.prompt.messages": JSON.stringify([{ role: "system", content: "Sys" }]),
+ "ai.prompt": JSON.stringify({ prompt: "User prompt" }),
+ };
+ transformAiSdkAttributes(attributes);
+ assert.strictEqual(attributes[`${SpanAttributes.LLM_PROMPTS}.0.content`], "Sys");
+ assert.strictEqual(attributes[`${SpanAttributes.LLM_PROMPTS}.1.content`], "User prompt");
+ });📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| describe("transformAiSdkAttributes - single prompt", () => { | |
| it("should transform ai.prompt to prompt attributes", () => { | |
| const promptData = { | |
| prompt: | |
| "Help me plan a trip to San Francisco. I\\'d like to know:\\n1. What\\'s the weather like there?\\n2. Find some restaurants\\n\\nPlease help!", | |
| }; | |
| const attributes = { | |
| "ai.prompt": JSON.stringify(promptData), | |
| someOtherAttr: "value", | |
| }; | |
| transformAiSdkAttributes(attributes); | |
| assert.strictEqual( | |
| attributes[`${SpanAttributes.LLM_PROMPTS}.0.content`], | |
| "Help me plan a trip to San Francisco. I\\'d like to know:\\n1. What\\'s the weather like there?\\n2. Find some restaurants\\n\\nPlease help!", | |
| ); | |
| assert.strictEqual( | |
| attributes[`${SpanAttributes.LLM_PROMPTS}.0.role`], | |
| "user", | |
| ); | |
| assert.strictEqual(attributes["ai.prompt"], undefined); | |
| assert.strictEqual(attributes.someOtherAttr, "value"); | |
| }); | |
| it("should not modify attributes when ai.prompt is not present", () => { | |
| const attributes = { | |
| someOtherAttr: "value", | |
| }; | |
| const originalAttributes = { ...attributes }; | |
| transformAiSdkAttributes(attributes); | |
| assert.deepStrictEqual(attributes, originalAttributes); | |
| }); | |
| it("should handle invalid JSON gracefully", () => { | |
| const attributes = { | |
| "ai.prompt": "invalid json {", | |
| someOtherAttr: "value", | |
| }; | |
| transformAiSdkAttributes(attributes); | |
| // Should not modify attributes when JSON parsing fails | |
| assert.strictEqual(attributes["ai.prompt"], "invalid json {"); | |
| assert.strictEqual(attributes.someOtherAttr, "value"); | |
| }); | |
| }); | |
| describe("transformAiSdkAttributes - single prompt", () => { | |
| it("should append ai.prompt after ai.prompt.messages without overwriting", () => { | |
| const attributes = { | |
| "ai.prompt.messages": JSON.stringify([{ role: "system", content: "Sys" }]), | |
| "ai.prompt": JSON.stringify({ prompt: "User prompt" }), | |
| }; | |
| transformAiSdkAttributes(attributes); | |
| assert.strictEqual(attributes[`${SpanAttributes.LLM_PROMPTS}.0.content`], "Sys"); | |
| assert.strictEqual(attributes[`${SpanAttributes.LLM_PROMPTS}.1.content`], "User prompt"); | |
| }); | |
| it("should transform ai.prompt to prompt attributes", () => { | |
| const promptData = { | |
| prompt: | |
| "Help me plan a trip to San Francisco. I\\'d like to know:\\n1. What\\'s the weather like there?\\n2. Find some restaurants\\n\\nPlease help!", | |
| }; | |
| const attributes = { | |
| "ai.prompt": JSON.stringify(promptData), | |
| someOtherAttr: "value", | |
| }; | |
| transformAiSdkAttributes(attributes); | |
| assert.strictEqual( | |
| attributes[`${SpanAttributes.LLM_PROMPTS}.0.content`], | |
| "Help me plan a trip to San Francisco. I\\'d like to know:\\n1. What\\'s the weather like there?\\n2. Find some restaurants\\n\\nPlease help!", | |
| ); | |
| assert.strictEqual( | |
| attributes[`${SpanAttributes.LLM_PROMPTS}.0.role`], | |
| "user", | |
| ); | |
| assert.strictEqual(attributes["ai.prompt"], undefined); | |
| assert.strictEqual(attributes.someOtherAttr, "value"); | |
| }); | |
| it("should not modify attributes when ai.prompt is not present", () => { | |
| const attributes = { | |
| someOtherAttr: "value", | |
| }; | |
| const originalAttributes = { ...attributes }; | |
| transformAiSdkAttributes(attributes); | |
| assert.deepStrictEqual(attributes, originalAttributes); | |
| }); | |
| it("should handle invalid JSON gracefully", () => { | |
| const attributes = { | |
| "ai.prompt": "invalid json {", | |
| someOtherAttr: "value", | |
| }; | |
| transformAiSdkAttributes(attributes); | |
| // Should not modify attributes when JSON parsing fails | |
| assert.strictEqual(attributes["ai.prompt"], "invalid json {"); | |
| assert.strictEqual(attributes.someOtherAttr, "value"); | |
| }); | |
| }); |
🤖 Prompt for AI Agents
packages/traceloop-sdk/test/ai-sdk-transformations.test.ts around lines 429-477:
the test for "single prompt" needs to ensure a standalone ai.prompt does not
overwrite existing ai.prompt.messages entries; modify the test input so
ai.prompt JSON includes a messages array (e.g. one message) and then call
transformAiSdkAttributes, and add assertions that the original messages occupy
index .0 and the single prompt was appended at the next index (e.g. .1) — verify
both content and role for each index, that ai.prompt key is removed, and other
attributes remain unchanged.
There was a problem hiding this comment.
Important
Looks good to me! 👍
Reviewed a8adf1d in 45 seconds. Click for details.
- Reviewed
66lines of code in2files - Skipped
0files when reviewing. - Skipped posting
4draft comments. View those below. - Modify your settings and rules to customize what types of comments Ellipsis leaves. And don't forget to react with 👍 or 👎 to teach Ellipsis.
1. packages/traceloop-sdk/src/lib/tracing/ai-sdk-transformations.ts:27
- Draft comment:
Vendor mapping update: 'azure-openai' now maps to 'Azure'. Note that the 'azure' key was removed. Ensure this change covers all expected azure provider cases. - Reason this comment was not posted:
Confidence changes required:33%<= threshold50%None
2. packages/traceloop-sdk/test/ai-sdk-transformations.test.ts:927
- Draft comment:
Removed 'azure-openai.chat' from openaiProviders array. Verify that all intended azure provider variants are now covered by dedicated tests. - Reason this comment was not posted:
Confidence changes required:33%<= threshold50%None
3. packages/traceloop-sdk/test/ai-sdk-transformations.test.ts:941
- Draft comment:
Added test for transforming 'azure-openai' provider to system 'Azure'. This helps confirm correct vendor normalization. - Reason this comment was not posted:
Confidence changes required:0%<= threshold50%None
4. packages/traceloop-sdk/test/ai-sdk-transformations.test.ts:1093
- Draft comment:
Updated vendor transformation assertion: expected value changed from 'OpenAI' to 'Azure'. Ensure consistency with the vendor mapping changes. - Reason this comment was not posted:
Confidence changes required:0%<= threshold50%None
Workflow ID: wflow_w79HrRojw37QKlp1
You can customize
There was a problem hiding this comment.
Important
Looks good to me! 👍
Reviewed fb2f250 in 46 seconds. Click for details.
- Reviewed
12lines of code in1files - Skipped
0files when reviewing. - Skipped posting
1draft comments. View those below. - Modify your settings and rules to customize what types of comments Ellipsis leaves. And don't forget to react with 👍 or 👎 to teach Ellipsis.
1. packages/sample-app/package.json:22
- Draft comment:
Removed 'debug:tool_calls' script. Ensure its removal is intentional and that any debugging functionality is maintained elsewhere or documented accordingly. - Reason this comment was not posted:
Comment looked like it was already resolved.
Workflow ID: wflow_ZzLZa7eFd4P6c5P6
You can customize
There was a problem hiding this comment.
Important
Looks good to me! 👍
Reviewed 4ffe0ca in 23 seconds. Click for details.
- Reviewed
15lines of code in1files - Skipped
0files when reviewing. - Skipped posting
1draft comments. View those below. - Modify your settings and rules to customize what types of comments Ellipsis leaves. And don't forget to react with 👍 or 👎 to teach Ellipsis.
1. packages/traceloop-sdk/test/ai-sdk-transformations.test.ts:945
- Draft comment:
Minor formatting change: the single-element array for 'azure-openai' is now in one line. This is a clear style improvement. - Reason this comment was not posted:
Confidence changes required:0%<= threshold50%None
Workflow ID: wflow_JYGsH7whcFdvdgXd
You can customize
Co-authored-by: Claude <noreply@anthropic.com>
2 participants
Important
Enhances SDK with Vercel AI SDK tool calling, structured outputs, and expanded tracing and testing capabilities.
sample_vercel_ai_object.tsandsample_vercel_ai_tools.ts.sample_experiment.tsnow return a newanswerfield and updated dataset/evaluator identifiers.ai-sdk-transformations.tsto capture AI object responses, tool calls, prompts, and tools with improved span naming and vendor normalization.package.jsonto build and run the demos and a debug script for tool-call flows.ai-sdk-transformations.test.tsfor object responses, tool-call parsing, prompts/tools handling, vendor normalization, and aggregated transformations.This description was created by
for 4ffe0ca. You can customize this summary. It will automatically update as commits are pushed.
Summary by CodeRabbit
New Features
Chores
Tests