Function Calling Internals: How LLM Tool Use Works Under the Hood

What Function Calling Actually Is

LLMs are text-in, text-out. There is no special "function execution" capability inside the model. What actually happens:

1. You describe available tools as JSON Schema in your API request.
2. The model, trained to recognize when tools are useful, generates a specially formatted response that indicates it wants to call a tool with specific arguments.
3. Your code detects this response format, executes the actual function, and sends the result back to the model as a new message.
4. The model incorporates the result into its response.

The model never executes code. It generates a JSON description of what it wants your code to execute. This distinction matters for security: a malicious instruction in retrieved content cannot directly execute a function — it can only influence what the model asks you to execute, which your code can validate.

The Wire Format (OpenAI API)

Tools are defined as JSON Schema objects in the tools array of your request:

# Request to OpenAI chat completions API
{
  "model": "gpt-4o",
  "messages": [{"role": "user", "content": "What's the weather in Tokyo and London?"}],
  "tools": [
    {
      "type": "function",
      "function": {
        "name": "get_weather",
        "description": "Get current weather for a city. Returns temperature in Celsius and conditions.",
        "parameters": {
          "type": "object",
          "properties": {
            "city": {"type": "string", "description": "City name, e.g. 'Tokyo'"},
            "units": {"type": "string", "enum": ["celsius", "fahrenheit"]}
          },
          "required": ["city"]
        }
      }
    }
  ],
  "tool_choice": "auto"  // "auto" | "required" | "none" | {"type":"function","function":{"name":"..."}} 
}

The model's response when it decides to call a tool:

{
  "choices": [{
    "message": {
      "role": "assistant",
      "content": null,          // null when calling tools (no prose)
      "tool_calls": [
        {"id": "call_abc123", "type": "function",
         "function": {"name": "get_weather", "arguments": "{\"city\":\"Tokyo\",\"units\":\"celsius\"}"}},
        {"id": "call_def456", "type": "function",
         "function": {"name": "get_weather", "arguments": "{\"city\":\"London\",\"units\":\"celsius\"}"}}
      ]
    },
    "finish_reason": "tool_calls"  // signals: model wants to call tools before responding
  }]
}

Note that the model requested two tool calls simultaneously — one for Tokyo and one for London. These should be executed in parallel by your code, not sequentially.

The Complete Tool Execution Loop

import json, httpx

def run_agent_loop(user_message: str, tools: list, tool_executor: dict) -> str:
    messages = [{"role": "user", "content": user_message}]

    while True:
        response = call_openai(messages=messages, tools=tools)
        msg = response["choices"][0]["message"]
        finish = response["choices"][0]["finish_reason"]

        messages.append(msg)  # always add assistant message to history

        if finish == "stop":
            return msg["content"]  # model is done, return final response

        if finish == "tool_calls":
            # Execute all tool calls (in parallel for efficiency)
            import concurrent.futures
            with concurrent.futures.ThreadPoolExecutor() as pool:
                futures = {
                    pool.submit(
                        tool_executor[tc["function"]["name"]],
                        **json.loads(tc["function"]["arguments"])
                    ): tc["id"]
                    for tc in msg["tool_calls"]
                }
                for future, call_id in futures.items():
                    result = future.result()
                    messages.append({
                        "role": "tool",
                        "tool_call_id": call_id,
                        "content": json.dumps(result)
                    })
            # Loop continues — model will now see tool results and respond

Tool Description Quality Is Everything

The model decides which tool to call (and with which arguments) based entirely on the description field in your tool definition. This is a natural language field that the model reads as part of its prompt — treat it like a contract:

The "IMPORTANT:" instruction in tool descriptions is read by the model as part of its decision-making context. Safety constraints on dangerous tools belong in the description, not just in your application code.

Structured Outputs vs Function Calling

Function calling and structured outputs are related but distinct:

• Function calling: Model decides if and when to invoke a tool. The output is a tool call JSON, not the final response. Used for agentic workflows.
• Structured outputs (response_format: {type: "json_schema"}): The model's final text response is constrained to match a JSON schema. Used when you always want structured data back — no decision required.

For extracting structured data from text (e.g., parsing a receipt), use structured outputs. For workflows where the model chooses whether to look something up or act, use tool calling.

Handling Tool Errors Gracefully

def safe_tool_call(fn, **kwargs):
    try:
        result = fn(**kwargs)
        return {"success": True, "result": result}
    except Exception as e:
        # Return error as structured tool result — don't raise
        # Model will see the error and can decide to retry or explain
        return {"success": False, "error": str(e), "error_type": type(e).__name__}

Never raise exceptions from tool executor functions. Return structured error objects instead — the model can read error messages and decide to retry with corrected arguments, call a fallback tool, or explain the failure to the user.

token_choice: "required" for Structured Pipelines

By default (tool_choice: "auto"), the model decides whether to call a tool. For pipelines where you always need structured extraction, set tool_choice: {"type": "function", "function": {"name": "extract_data"}} to force the model to always call your extraction function. This makes output format deterministic — the response will always be a tool call, never a prose response.