Skip to content

Builtin Tools

Jump to bottom
Arian Amiramjadi edited this page Dec 24, 2025 · 1 revision

Built-in Tools

OpenAI's Responses API provides powerful built-in tools that extend model capabilities with web search, file search, code execution, image generation, computer control, and more. These tools are hosted and managed by OpenAI — you just enable them and the model handles the rest.

Overview

Tool Description Use Case
Web Search Search the internet for real-time information Current events, recent data, fact-checking
File Search Search your vector stores RAG, knowledge bases, document Q&A
Code Interpreter Execute Python in a sandbox Data analysis, calculations, chart generation
MCP Connect to remote MCP servers Third-party integrations, custom tools
Image Generation Generate images using GPT Image models Art, visualizations, creative content
Computer Use Control computer interfaces (CUA) Automation, browser tasks, UI testing
Shell Execute shell commands File operations, system tasks, builds
Apply Patch Structured file editing with diffs Code refactoring, migrations, multi-file edits

Note: Built-in tools are only available when using the OpenAI provider directly. They use the Responses API (/v1/responses) which is automatically selected when you add any built-in tool.

Web Search

Enable the model to search the web for up-to-date information.

Basic Usage

resp, err := ai.GPT5().
    WebSearch().
    User("What happened in tech news today?").
    Send()

With Options

meta := ai.GPT5().
    WebSearchWith(ai.WebSearchOptions{
        // Location for geo-targeted results
        Country:  "US",           // ISO country code
        City:     "San Francisco",
        Region:   "California",
        Timezone: "America/Los_Angeles",
        
        // Limit to specific domains
        AllowedDomains: []string{
            "nytimes.com",
            "bbc.com",
            "reuters.com",
        },
    }).
    User("What's the latest local news?").
    SendWithMeta()

// Access search citations
if meta.ResponsesOutput != nil {
    for _, cite := range meta.ResponsesOutput.Citations {
        fmt.Printf("[%s](%s)\n", cite.Title, cite.URL)
    }
}

Response Fields

When using SendWithMeta(), you can access:

  • ResponsesOutput.Text - The model's response text
  • ResponsesOutput.Citations - URL citations with titles
  • ResponsesOutput.Sources - All URLs consulted (more than citations)
  • ResponsesOutput.ToolCalls - Details of web_search_call operations

File Search

Search your OpenAI vector stores for relevant document chunks.

Basic Usage

// Single vector store
resp, err := ai.GPT5().
    FileSearch("vs_abc123").
    User("What's our refund policy?").
    Send()

// Multiple vector stores
resp, err := ai.GPT5().
    FileSearch("vs_policies", "vs_faqs", "vs_docs").
    User("How do I reset my password?").
    Send()

With Options

resp, err := ai.GPT5().
    FileSearchWith(ai.FileSearchOptions{
        VectorStoreIDs: []string{"vs_abc123"},
        MaxNumResults:  5,  // Default 10, max 50
        
        // Attribute filtering (optional)
        Filters: map[string]any{
            "type": "eq",
            "key":  "department",
            "value": "engineering",
        },
    }).
    User("Find engineering docs about deployments").
    Send()

Response Fields

  • ResponsesOutput.Citations - File citations with FileID and Filename

Code Interpreter

Let the model write and execute Python code in a sandboxed environment.

Basic Usage

resp, err := ai.GPT5().
    CodeInterpreter().
    User("Calculate the first 100 prime numbers").
    Send()

With More Memory

For large datasets or complex computations:

resp, err := ai.GPT5().
    CodeInterpreterWith(ai.CodeInterpreterOptions{
        MemoryLimit: "4g",  // Options: "1g" (default), "4g", "16g", "64g"
    }).
    User("Analyze this large CSV and create visualizations").
    Send()

With Files

resp, err := ai.GPT5().
    CodeInterpreterWith(ai.CodeInterpreterOptions{
        MemoryLimit: "4g",
        FileIDs:     []string{"file-abc123", "file-def456"},
    }).
    User("Process these data files").
    Send()

Using an Existing Container

resp, err := ai.GPT5().
    CodeInterpreterWith(ai.CodeInterpreterOptions{
        ContainerID: "cntr_abc123",  // Reuse a container
    }).
    User("Continue the analysis from before").
    Send()

MCP (Model Context Protocol)

Connect to remote MCP servers or built-in connectors for third-party integrations.

Remote MCP Server

resp, err := ai.GPT5().
    MCP("dice", "https://dmcp-server.deno.dev/sse").
    User("Roll 2d6+3").
    Send()

With Full Options

resp, err := ai.GPT5().
    MCPWith(ai.MCPOptions{
        Label:           "stripe",
        URL:             "https://mcp.stripe.com",
        Description:     "Stripe payment processing",
        Authorization:   os.Getenv("STRIPE_OAUTH_TOKEN"),
        RequireApproval: "never",  // or "always"
        AllowedTools:    []string{"create_payment_link", "list_invoices"},
    }).
    User("Create a $50 payment link").
    Send()

Built-in Connectors

OpenAI provides pre-configured connectors for popular services:

// Google Calendar
ai.GPT5().
    MCPConnector("calendar", ai.ConnectorGoogleCalendar, googleToken).
    User("What's on my calendar today?").
    Send()

// Gmail
ai.GPT5().
    MCPConnector("gmail", ai.ConnectorGmail, googleToken).
    User("Summarize my unread emails").
    Send()

// Dropbox
ai.GPT5().
    MCPConnector("dropbox", ai.ConnectorDropbox, dropboxToken).
    User("Find the Q4 report").
    Send()

Available Connectors

Connector Constant Description
Dropbox ai.ConnectorDropbox File search and retrieval
Gmail ai.ConnectorGmail Email search and reading
Google Calendar ai.ConnectorGoogleCalendar Calendar events
Google Drive ai.ConnectorGoogleDrive Document search
Microsoft Teams ai.ConnectorMicrosoftTeams Chat and messages
Outlook Calendar ai.ConnectorOutlookCalendar Calendar events
Outlook Email ai.ConnectorOutlookEmail Email access
SharePoint ai.ConnectorSharePoint Document search

Image Generation

Let the model generate images using GPT Image models (gpt-image-1, gpt-image-1.5).

Basic Usage

resp, err := ai.GPT5().
    ImageGeneration().
    User("Generate an image of a cat hugging an otter with an orange scarf").
    Send()

With Options

resp, err := ai.GPT5().
    ImageGenerationWith(ai.ImageGenerationOptions{
        Size:       "1024x1536",  // or "1024x1024", "auto"
        Quality:    "high",       // "low", "medium", "high", "auto"
        Format:     "png",        // "png", "jpeg", "webp"
        Background: "transparent", // "transparent", "opaque", "auto"
    }).
    User("Generate a portrait of a sunset over the ocean").
    Send()

Accessing Generated Images

meta := ai.GPT5().
    ImageGeneration().
    User("Draw a futuristic city").
    SendWithMeta()

if meta.ResponsesOutput != nil {
    for _, tc := range meta.ResponsesOutput.ToolCalls {
        if tc.Type == "image_generation_call" {
            // tc.ImageResult contains base64-encoded image
            // tc.RevisedPrompt contains the optimized prompt used
            imageBytes, _ := base64.StdEncoding.DecodeString(tc.ImageResult)
            os.WriteFile("city.png", imageBytes, 0644)
        }
    }
}

Streaming Partial Images

For faster visual feedback, you can request partial images during generation:

resp, err := ai.GPT5().
    ImageGenerationWith(ai.ImageGenerationOptions{
        PartialImages: 2, // 1-3 partial images during generation
    }).
    User("Generate a detailed landscape").
    Send()

Computer Use (CUA)

Build computer-using agents that can control interfaces, click buttons, type text, and navigate applications.

Note: Computer Use is in beta. Always run in sandboxed environments. See OpenAI's safety guidelines.

Basic Usage

// Use the computer-use-preview model
resp, err := ai.ComputerUsePreview().
    ComputerUse(1024, 768, "browser").
    User("Go to bing.com and search for OpenAI news").
    Send()

The CUA Loop

Computer Use works in a loop: the model sends actions, you execute them, and return screenshots.

// 1. Initial request
meta := ai.ComputerUsePreview().
    ComputerUse(1024, 768, "browser").
    User("Book a flight to Paris").
    SendWithMeta()

// 2. Check for computer_call items
for _, tc := range meta.ResponsesOutput.ToolCalls {
    if tc.Type == "computer_call" {
        // 3. Execute the action
        action := tc.Action
        switch action.Type {
        case "click":
            // Click at (action.X, action.Y) with action.Button
        case "type":
            // Type action.Text
        case "scroll":
            // Scroll by (action.ScrollX, action.ScrollY)
        case "keypress":
            // Press action.Keys
        case "wait":
            // Wait ~2 seconds
        case "screenshot":
            // Just take a screenshot
        }
        
        // 4. Take screenshot and send back
        // (continue the conversation with the screenshot)
    }
}

Computer Actions

Action Fields Description
click X, Y, Button Click at coordinates
double_click X, Y Double-click
scroll X, Y, ScrollX, ScrollY Scroll at position
keypress Keys Press keyboard keys
type Text Type text
wait - Wait for page to load
screenshot - Request a screenshot

Safety Checks

The model may return PendingSafetyChecks that require acknowledgment:

if len(tc.PendingSafetyChecks) > 0 {
    // Show to user for confirmation before proceeding
    for _, check := range tc.PendingSafetyChecks {
        fmt.Printf("Safety check: %s - %s\n", check.Code, check.Message)
    }
}

Shell

Let GPT-5.1+ execute shell commands in your environment. Your code handles the actual execution.

Basic Usage

resp, err := ai.GPT51().
    Shell().
    User("Find the largest PDF file in ~/Documents").
    Send()

The Shell Loop

meta := ai.GPT51().
    Shell().
    User("List all Go files and count lines of code").
    SendWithMeta()

for _, tc := range meta.ResponsesOutput.ToolCalls {
    if tc.Type == "shell_call" {
        action := tc.ShellAction
        
        // Execute commands (can be run concurrently)
        for _, cmd := range action.Commands {
            // Execute cmd in your sandbox
            // Respect action.TimeoutMs if set
            stdout, stderr, exitCode := executeCommand(cmd)
            
            // Send results back to the model
        }
    }
}

Shell Action Fields

Field Description
Commands List of shell commands to execute
TimeoutMs Suggested timeout in milliseconds
MaxOutputLength For truncating large outputs

Best Practices

  1. Always sandbox execution - Use Docker, firejail, or restricted users
  2. Filter dangerous commands - Block rm -rf, curl | bash, etc.
  3. Impose resource limits - CPU, memory, network, disk
  4. Log everything - For debugging and auditing

Apply Patch

Let GPT-5.1+ edit files using structured diffs. The model emits V4A diffs, your code applies them.

Basic Usage

resp, err := ai.GPT51().
    ApplyPatch().
    Context("lib/fib.py").
    Context("run.py").
    User("Rename the fib() function to fibonacci() everywhere").
    Send()

The Patch Loop

meta := ai.GPT51().
    ApplyPatch().
    Context("src/").
    User("Add error handling to all database calls").
    SendWithMeta()

for _, tc := range meta.ResponsesOutput.ToolCalls {
    if tc.Type == "apply_patch_call" {
        op := tc.PatchOperation
        
        switch op.Type {
        case "create_file":
            // Create new file at op.Path with content from op.Diff
        case "update_file":
            // Apply op.Diff to existing file at op.Path
        case "delete_file":
            // Delete file at op.Path
        }
        
        // Send result back to model
    }
}

Patch Operations

Operation Description Fields
create_file Create a new file Path, Diff (full content)
update_file Modify existing file Path, Diff (changes)
delete_file Remove a file Path

Combining with Shell

For powerful coding agents, combine Shell and ApplyPatch:

resp, err := ai.GPT51().
    Shell().
    ApplyPatch().
    User("Find all TODO comments and fix them").
    Send()

Combining Tools

You can enable multiple tools in a single request:

// Web search + code interpreter
resp, err := ai.GPT5().
    WebSearch().
    CodeInterpreter().
    User("Find today's AAPL stock price and create a chart comparing it to last week").
    Send()

// File search + code interpreter
resp, err := ai.GPT5().
    FileSearch("vs_sales_data").
    CodeInterpreter().
    User("Analyze our Q4 sales data and create a summary report").
    Send()

// Shell + Apply Patch for coding agents
resp, err := ai.GPT51().
    Shell().
    ApplyPatch().
    User("Find all deprecated API calls and update them to the new API").
    Send()

// Image generation + web search
resp, err := ai.GPT5().
    WebSearch().
    ImageGeneration().
    User("Search for the latest SpaceX rocket and generate an artistic rendering").
    Send()

Accessing Response Details

Use SendWithMeta() to get full response details:

meta := ai.GPT5().
    WebSearch().
    User("Latest AI news").
    SendWithMeta()

if meta.Error != nil {
    log.Fatal(meta.Error)
}

// Main response
fmt.Println(meta.Content)

// Token usage
fmt.Printf("Tokens: %d (prompt: %d, completion: %d)\n",
    meta.Tokens, meta.PromptTokens, meta.CompletionTokens)

// Responses API details
if meta.ResponsesOutput != nil {
    // Citations
    fmt.Println("\nCitations:")
    for _, c := range meta.ResponsesOutput.Citations {
        if c.URL != "" {
            fmt.Printf("  - [%s](%s)\n", c.Title, c.URL)
        } else {
            fmt.Printf("  - %s (file: %s)\n", c.Filename, c.FileID)
        }
    }
    
    // Tool calls made
    fmt.Println("\nTool Calls:")
    for _, tc := range meta.ResponsesOutput.ToolCalls {
        fmt.Printf("  - %s: %s\n", tc.Type, tc.Name)
    }
}

Error Handling

resp, err := ai.GPT5().
    WebSearch().
    User("Search query").
    Send()

if err != nil {
    // Check if it's a provider error
    if provErr, ok := err.(*ai.ProviderError); ok {
        fmt.Printf("Provider: %s\n", provErr.Provider)
        fmt.Printf("Code: %s\n", provErr.Code)
        fmt.Printf("Message: %s\n", provErr.Message)
    }
}

Best Practices

  1. Use the right tool for the job

    • Web Search for current information
    • File Search for your own documents
    • Code Interpreter for calculations and data processing
    • MCP for third-party service integrations
    • Image Generation for visual content
    • Computer Use for browser/UI automation
    • Shell for system operations
    • Apply Patch for code editing

  2. Limit domains for focused web search

    • Use AllowedDomains to restrict results to trusted sources

  3. Choose appropriate memory limits

    • Start with default "1g" for Code Interpreter
    • Scale up only when needed for large datasets

  4. Handle rate limits

    • Use RetryWithBackoff() for automatic retry on rate limits

  5. Check capabilities

    • Built-in tools only work with OpenAI provider
    • The library will warn if you try to use them with unsupported providers

  6. Security for agentic tools

    • Always sandbox Shell and Computer Use execution
    • Validate Apply Patch operations before applying
    • Require human approval for sensitive actions
    • Log all tool operations for auditing

  7. Use truncation for Responses API

    • Computer Use requires truncation: "auto" in raw API calls
    • The library handles this automatically

Clone this wiki locally
⚡