Secure LLM Tool Calling: Permissions, Validation, and Human Approval Workflows

Overview

Tool calling, also called function calling, allows an LLM to return a structured request for an application function. The application then decides whether to execute that function and sends the result back to the model. This distinction is the foundation of secure LLM tool calling: the model may propose an action, but application code must authorize, validate, execute, and record it.

A safe design separates five responsibilities:

1. Model selection: The model chooses, or suggests, a tool based on the conversation and available schemas.
2. Request validation: The application checks the tool name, argument types, formats, ranges, and business rules.
3. Authorization: A policy engine or service determines whether the current user, agent, and workflow may perform the action.
4. Execution: A controlled service calls the downstream API with its own credentials and limited scope.
5. Observation: Logs, traces, approvals, failures, and outcomes are recorded for review.

Do not place unrestricted credentials in a prompt, expose internal administrative functions as general-purpose tools, or assume that a valid JSON payload is automatically safe. A request can be syntactically correct and still be unauthorized, financially consequential, privacy-sensitive, or operationally dangerous.

For structured response design, pair this workflow with a strict schema and recovery strategy. The guide to structured outputs from LLM APIs provides useful background on schema validation and handling malformed responses.

What to track

1. Tool definitions and risk levels

Maintain an inventory of every tool exposed to an agent. For each entry, record its purpose, owning team, input schema, downstream system, authentication method, data classification, reversibility, and approval requirement. A plain description such as manage account is too broad; prefer a narrowly scoped operation such as create a draft support reply or look up an order by authenticated customer ID.

Risk is determined by impact, not by whether the tool is technically simple. A one-line API call that deletes data deserves more protection than a complex read-only report.

2. Permissions and identity context

Apply least privilege at multiple layers. The user should have permission to request the action, the agent should be allowed to use the tool for that workflow, and the service account should have only the downstream access required to complete it. Pass trusted identity context from the application rather than allowing the model to supply user IDs, roles, or approval status in free text.

A practical permission-policy template can look like this:

tool: create_refund
purpose: prepare or submit a refund for an authenticated order
allowed_roles: support_supervisor
allowed_workflows: returns_assistant
input_constraints:
  order_id: authenticated order owned by the customer or assigned case
  amount: positive value, no greater than the verified eligible amount
  currency: must match the order currency
execution_mode: approval_required
rate_limit: per user, agent, and account
idempotency: required
logging: request, decision, approver, result, correlation_id
fallback: create a review task; do not execute automatically

Keep policy decisions outside the prompt. Instructions can help the model select an appropriate tool, but they should not be the enforcement mechanism.

3. Validation and execution controls

Validate the tool name against an explicit allowlist. Reject unknown tools and unexpected arguments. Enforce types, maximum lengths, enum values, numeric bounds, date windows, resource ownership, and cross-field rules. Normalize inputs before validation where appropriate, but retain the original request for audit purposes.

Authentication belongs in the execution layer. The tool runner should obtain credentials from a secure secret manager or delegated authorization flow, not from model output. Use separate credentials for separate environments, and keep development tools away from production data unless access is deliberately granted.

For actions that may be retried, use idempotency keys and record the downstream result. This prevents a timeout or repeated model response from creating duplicate tickets, orders, messages, or payments. Add timeouts, bounded retries, circuit breakers, and rate limits so a faulty workflow cannot generate uncontrolled activity.

4. Prompt-injection and data-flow defenses

Treat retrieved documents, emails, web pages, uploaded files, and tool results as untrusted data. Text that says to ignore previous instructions or call a different tool is not an authorization signal. Keep data and instructions conceptually separate, label untrusted content, and constrain the tools available at each workflow stage.

For retrieval-based systems, return only the fields needed for the current task and apply access control before content reaches the model. A retrieval pipeline should not become an indirect path to confidential records. The RAG workflow guide for developers is a useful companion when tool calling is combined with search or document retrieval.

5. Approvals, logs, and measurable outcomes

Require human approval when an action is irreversible, affects money or access, communicates externally, changes regulated or sensitive records, or exceeds a defined threshold. Show the approver the intended tool, normalized arguments, affected resource, reason, and expected consequence. Approval should expire, be bound to the exact request, and not be reusable for a materially different action.

Log the model and application versions, tool schema version, user and workflow identifiers, request and validation result, authorization decision, approval record, downstream response, latency, and error category. Avoid logging secrets or unnecessary personal data. Logs should support investigation without creating a second sensitive-data store.

Cadence and checkpoints

Security review should be part of the normal AI development workflow rather than a one-time launch task. Use the following cadence as a starting point and adjust it to the impact of the system.

• Per deployment: Run schema validation, authorization tests, prompt-injection cases, duplicate-request tests, timeout tests, and approval-path tests.
• Monthly: Review tool usage, denied requests, approval rates, unusual argument patterns, error spikes, rate-limit events, and tools that are no longer needed.
• Quarterly: Reassess tool risk, service-account scopes, allowlists, retention rules, incident procedures, and model or provider changes. Reconfirm that each tool has an owner.
• After a provider or SDK release: Recheck tool schemas, strict-mode behavior, response parsing, streaming behavior, retry semantics, and approval integrations. Do not assume that an unchanged application has unchanged execution behavior.
• After an incident or failed call: Preserve the relevant trace, reproduce the scenario safely, identify whether the failure was selection, validation, authorization, execution, or monitoring, and add a regression test.

Keep a change record for tool additions, permission changes, schema revisions, model changes, and downstream API migrations. A small change to an input field can alter the action's risk or break an authorization rule.

How to interpret changes

A rise in tool calls is not automatically an improvement. Compare volume with successful outcomes, user-confirmed value, denial rates, latency, cost, and operational impact. An increase in rejected calls may indicate better defenses, a broken schema, a prompt change, or a new attack pattern. Investigate before relaxing controls.

Group failures by stage:

• Selection failures: The model chooses the wrong tool or attempts a tool that is not relevant.
• Validation failures: Arguments are missing, malformed, out of range, or inconsistent with the trusted application context.
• Authorization failures: The user, workflow, or service lacks permission.
• Execution failures: The downstream API times out, rejects the request, or returns an unexpected result.
• Outcome failures: The call succeeds technically but produces an incorrect, duplicate, incomplete, or harmful result.

Track these categories separately. Improving model tool selection will not fix an over-privileged service account, and increasing retries will not fix an invalid business rule. Review a sample of successful calls as well as failures; unsafe behavior can pass validation and still reveal that the tool is too broad.

When a model or provider changes, compare behavior using a fixed evaluation set. Include benign requests, ambiguous requests, unauthorized requests, prompt-injection attempts, malformed arguments, repeated requests, and requests requiring approval. Store expected decisions rather than relying only on generated text.

When to revisit

Revisit this workflow whenever you add a tool, expose a new data source, change a permission, alter a schema, switch models or providers, update an SDK, change a downstream API, or move from simulation to production execution. Also review it after a failed tool call, unexpected user report, security finding, or material change to the business process.

For a routine review, complete these steps:

1. Export the current tool inventory and confirm an owner, purpose, risk rating, and approval rule for every tool.
2. Compare permissions with actual usage. Remove unused scopes, stale allowlist entries, and tools that no longer have a clear business purpose.
3. Run the test checklist in a non-production environment: valid request, missing field, wrong type, boundary value, unauthorized resource, injected instruction, duplicate request, timeout, downstream error, and approval rejection.
4. Inspect representative traces for sensitive-data exposure, untrusted instructions, excessive arguments, missing correlation IDs, and incomplete audit records.
5. Confirm that high-impact actions remain approval-gated and that approval covers the exact normalized action rather than a broad conversational intent.
6. Record findings, assign owners, set deadlines, and rerun failed tests before expanding access.

The goal is not to prevent an LLM from doing useful work. It is to ensure that every tool call has a narrow purpose, a trusted identity context, an enforceable policy, a visible outcome, and a recovery path. Treat permissions, validation, approvals, and monitoring as versioned parts of the application, and secure LLM tool calling becomes a repeatable engineering practice rather than a prompt-writing exercise.