Skip to content
Help guide the future of agentgateway — Take the survey

For the complete documentation index, see llms.txt. Markdown versions of all docs pages are available by appending .md to any docs URL.

Debug your setup

Verified : Code examples on this page have been automatically tested and verified.
Page as Markdown

Find and fix configuration and runtime problems in a standalone agentgateway using its admin endpoints and agctl.

Inspect and troubleshoot a standalone agentgateway instance through the admin endpoints and the agctl command-line tool.

About

Agentgateway exposes an admin interface on 127.0.0.1:15000 by default. The admin interface is a local debugging endpoint for one running process. It provides the following endpoints for inspection and debugging.

The admin interface is not how you operate agentgateway day to day, and most installations never touch it. You reach it when you are debugging a specific problem on the host that runs the proxy, and you use agctl rather than raw endpoints for most of that.

Caution

The admin interface binds to the loopback interface, so only a client on the same host can reach it. Keep it that way. Endpoints such as /quitquitquit and /config_dump shut down the proxy and dump its full configuration to any caller that can open a connection.

Do not confuse the admin interface with the agentgateway UI, which is the web interface that you use to manage the proxy. The admin interface serves a copy of the UI as a convenience for local use, but the two are different surfaces: the UI belongs on a gateway, where you can authenticate it, and the debugging endpoints in the following table are served only on the admin address. For more information, see The UI and the admin interface are not the same thing.

EndpointDescription
/config_dumpReturns the runtime configuration that agentgateway has loaded, including binds, listeners, routes, backends, workloads, services, and policies.
/debug/traceStreams a JSON-over-SSE trace of the next request that the proxy handles. The agctl proxy trace command consumes this endpoint.
/loggingGet and set the logging level at runtime.
/memoryDump allocator and process memory statistics.
/debug/pprof/profileBuild a CPU profile by using the pprof profiler. Available on Linux builds only. Use ?seconds=N to set the duration (1–300s, default 10s) and ?frequency=N to set the sampling rate in Hz (1–1000, default 100).
/debug/pprof/heapCollect heap profiling data. Contains allocation samples on Linux builds only.
/debug/tasksInspect the live tokio task tree.
/quitquitquitTrigger a graceful shutdown of agentgateway.

You rarely need to move the admin address, and you should not expose it. If you must change it, see Change the admin address.

To inspect the proxy’s configuration and to capture per-request traces, use the agctl command-line tool. agctl wraps the admin endpoints and renders their output in formats that are easier to scan than raw JSON.

Inspect the loaded configuration

To dump the configuration that the running proxy has loaded, capture the JSON from the /config_dump endpoint and pass it to agctl proxy config all.

  1. Save the proxy’s config dump to a file.

    curl -s http://127.0.0.1:15000/config_dump > /tmp/agw-dump.json
  2. Render it with agctl. Use -o yaml for a more readable view.

    agctl proxy config all --file /tmp/agw-dump.json -o yaml

For complete steps, see Inspect agentgateway configuration.

Trace requests

For historical LLM requests, query the request-log database with agctl. For example, agctl standalone logs --since 1h --status 500 lists stored server-error responses. Database request logs complement the live trace below. Startup and configuration reload errors appear in the process logs only.

To capture a per-request trace as agentgateway processes it, use agctl proxy trace. The trace shows you the route that was selected, the policies that were applied, the backend that was chosen, and the response status. Tracing is invaluable for understanding why a request matched (or did not match) a route, why a policy was or was not applied, or why a request returned an unexpected status.

  1. In one terminal, start a watch.

    agctl proxy trace --local
  2. In another terminal, send a request.

    curl http://127.0.0.1:3000/headers

    agctl opens a text-based terminal user interface (TUI) that walks you through the request and response lifecycle. Use --raw to print JSON Lines instead.

For complete steps, including how to inject a request from agctl itself, see Trace requests with agctl.

Enable debug logs

Agentgateway uses the same level syntax as RUST_LOG: error, warn, info, debug, and trace. You can change the level at runtime through the /logging endpoint, or set it in your config file at startup.

Set the log level without restarting agentgateway. If you configured agentgateway to use a different admin address, update the host and port accordingly.

curl -X POST "http://localhost:15000/logging?level=debug"

Example output:

current log level is typespec_client_core::http::policies::logging=warn,hickory_server::server::server_future=off,rmcp=warn,debug

The agentgateway process now writes debug log lines, such as the following.

2026-02-12T16:11:25.493503Z	debug	proxy::httpproxy	request before normalization: Request { method: OPTIONS, uri: /sse?sessionId=...

You can also set fine-grained levels per module by using the same RUST_LOG filter syntax, such as info,proxy::httpproxy=trace.

Debug client-safe error responses

When agentgateway rejects a request or cannot complete it, the response body carries a short, generic message, such as authentication failed or no backends available. The message does not include internal details, such as backend addresses, credential sources, or token validation errors. The error field of the access log and the request traces keep the full cause.

For example, a route uses backendAuth: copilot, but agentgateway finds no Copilot token. The client receives only a generic message.

HTTP/1.1 500 Internal Server Error
content-type: text/plain
content-length: 29

backend authentication failed

The access log line for the same request names the cause.

error	request gateway=default/default listener=default route=default/copilot ... http.status=500 error="backend authentication failed: Copilot token not found; set GH_COPILOT_TOKEN or authenticate with GitHub Copilot/GitHub CLI" reason=Internal

The following table lists common failures and the message that the client receives. The HTTP status code is the same as in earlier versions. Only the message is shorter.

FailureResponse body
JWT, API key, basic, or OIDC authentication fails.authentication failed
An authorization policy denies the request, or the external authorization service cannot be reached.authorization failed
Backend authentication cannot get or use a credential.backend authentication failed
The route has no backend, or the backend has no healthy endpoint.no backends available
A dynamic backend resolves to a loopback or link-local address.backend not allowed
DNS resolution for the backend fails.dns resolution failed
The connection to the backend fails.connection refused, connection reset, connection aborted, connection timed out, upstream connection failed, or upstream call failed
The request times out.request timeout
A policy, such as a CEL expression, fails to run.policy failed
An external processing call fails.external processing failed
An LLM request fails with a 5xx status code.failed to process LLM request or failed to process LLM response
A budget limit is reached.budget exceeded, in the error.message field of a JSON body
Another internal error occurs.internal error

Some responses keep their details or work differently.

  • LLM client errors: An LLM request that fails with a 4xx status code keeps its detail, because the detail describes the client’s own request. For example, failed to process LLM request: unsupported conversion: from Completions to provider custom.
  • MCP client errors: MCP errors that describe the client’s own request also keep their detail, such as session not found or client must accept both application/json and text/event-stream. The message no longer has the mcp: prefix. Other MCP errors carry a generic message, such as failed to send message or service unavailable.
  • MCP authorization: A tool, prompt, or resource that an MCP authorization rule denies returns the JSON-RPC message unknown resource. Clients cannot tell a denied resource from one that does not exist. This message stays the same even when you expose error details.
  • Responses from other services: Agentgateway passes some responses through unchanged. These include a denial response from an HTTP external authorization service, a rejection response that you configure for a guardrail, a response from a remote rate limit service, and an HTTP error from an MCP server.
  • CEL proxy.error.message: The proxy.error.message variable holds the full internal message, which is the same message as in the logs. If a transformation copies this variable onto a response, clients receive the internal detail. To tell clients the type of failure, use proxy.error.reason instead.

Return error details to clients

For local debugging in a trusted environment, you can send the full internal error back to the client. Set the AGENTGATEWAY_UNSAFE_EXPOSE_ERROR_DETAILS environment variable to true. Any other value keeps the details hidden.

Warning

Do not use this setting on a shared or public gateway. Client responses can include backend addresses, credential lookup failures, and other details about your environment.

  1. Start agentgateway with the environment variable.

    AGENTGATEWAY_UNSAFE_EXPOSE_ERROR_DETAILS=true agentgateway -f config.yaml
  2. Send a request that fails. The response now includes the cause, such as for the Copilot example from the previous section.

    HTTP/1.1 500 Internal Server Error
    content-type: text/plain
    content-length: 123
    
    backend authentication failed: Copilot token not found; set GH_COPILOT_TOKEN or authenticate with GitHub Copilot/GitHub CLI
    

    Agentgateway logs a warning the first time that it returns an error with details.

    warn	proxy	AGENTGATEWAY_UNSAFE_EXPOSE_ERROR_DETAILS is enabled; internal error details are sent to clients
    
  3. When you finish debugging, restart agentgateway without the environment variable.

Capture profiles

Agentgateway includes pprof endpoints to help you investigate CPU and memory issues. Use the agctl proxy profile commands to read those endpoints and write the profile to a file.

Important

Profiling data is available only when agentgateway runs on Linux. On macOS and Windows builds, the CPU profile endpoint is not registered, so agctl proxy profile cpu fails with a 404 Not Found error, and agctl proxy profile heap writes a profile that contains no allocation samples.

  1. Optional: If you have not already, download Graphviz to visualize the profiles.

  2. Capture a CPU profile. The default duration is 30 seconds. Send traffic through agentgateway while the profile runs. Otherwise, the profile contains no samples.

    agctl proxy profile cpu --local --seconds 30 -o ./cpu.pprof

    Example output:

    Wrote cpu profile to ./cpu.pprof
  3. Capture a heap profile.

    agctl proxy profile heap --local -o ./heap.pprof

    Example output:

    Wrote heap profile to ./heap.pprof

    If you omit -o, agctl writes the profile to agentgateway-cpu-<timestamp>.pb.gz or agentgateway-heap-<timestamp>.pb.gz in the current directory.

  4. Inspect the profile with go tool pprof.

    CPU profile

    go tool pprof -http=: cpu.pprof

    Heap profile

    go tool pprof -http=: heap.pprof

    Graphviz opens on your web browser to a UI on localhost. Example:

    Heap profile graph
    Heap profile graph

Note

agctl proxy profile assumes that the admin server listens on port 15000. If you changed the admin address, pass the port with -p, such as agctl proxy profile heap --local -p 16000. You can also request the profiles directly from the admin endpoints, such as curl -o cpu.pprof "http://127.0.0.1:15000/debug/pprof/profile?seconds=30". Use the endpoints directly when you want to set the sampling rate with the frequency parameter, which agctl does not expose.

Change the admin address

Most installations leave the admin address alone. Change it only when the default port conflicts with something else on the host, or when you want to turn the admin interface off entirely.

The admin address is localhost:15000 in every installation method, whether or not the UI is also attached to a gateway. A gateway that serves the UI does not change the admin address, and changing the admin address does not change the gateway port. Set adminAddr in the config section of your configuration file to move the admin address, or to turn it off.

The value must use ip:port format, and also accepts unix:/path/to/socket or off. Setting off disables the admin interface altogether, which leaves a gateway in the ui section as the only way to reach the UI.

Tip

To reach the UI from another host, attach the UI to a gateway instead of moving the admin address. A gateway is the only location that you can put an authentication policy on. For more information, see Serve the UI on a gateway.

  1. Add or update the adminAddr field in your configuration file.

    # yaml-language-server: $schema=https://agentgateway.dev/schema/config
    config:
      adminAddr: localhost:9090
  2. Start agentgateway with the updated config. Because adminAddr is in the config section, a running instance keeps the previous address until you restart it.

    agentgateway -f config.yaml

    Example output:

    INFO app  serving UI at http://localhost:9090/ui
  3. Confirm that the admin interface answers on the new address. In this example, the endpoints move to port 9090.

    curl -s -o /dev/null -w "%{http_code}\n" http://localhost:9090/config_dump

    Example output:

    200

Note

If you change adminAddr, update any command that calls the admin interface to use the new address. For example, change curl http://localhost:15000/logging to use the new port.

Reach the UI in a container

The default admin address binds to the container’s own loopback interface, so publishing port 15000 with -p 15000:15000 does not make it reachable from your host. You have two options.

  • Serve the UI on a gateway instead, which is what the generated configuration does. This is the better option, because you can attach authentication policies to the gateway, and because it leaves the admin interface where it belongs. For more information, see Serve the UI on a gateway.

  • Bind the admin address to all interfaces by setting config.adminAddr to 0.0.0.0:15000, then publish that port. Do this only on a host where nothing untrusted can reach the published port, such as your personal workstation.

    Caution

    Binding the admin address to 0.0.0.0 publishes unauthenticated shutdown and configuration-dump endpoints to every network that the host is attached to. Never do this on a shared or production host. Attach the UI to a gateway instead.

    # yaml-language-server: $schema=https://agentgateway.dev/schema/config
    config:
      adminAddr: 0.0.0.0:15000
    gateways:
      default:
        port: 4000
    docker run -d \
      --name agentgateway \
      --user "$(id -u):$(id -g)" \
      -v "$PWD/config.yaml:/config.yaml" \
      -p 4000:4000 -p 15000:15000 \
      cr.agentgateway.dev/agentgateway:v1.7.0 \
      -f /config.yaml
Was this page helpful?
Agentgateway assistant

Ask me anything about agentgateway configuration, features, or usage.

Note: AI-generated content might contain errors; please verify and test all returned information.

Tip: one topic per conversation gives the best results. Use the + button in the chat header to start a new conversation.

Switching topics? Starting a new conversation improves accuracy.
↑↓ navigate ↵ select esc dismiss

What could be improved?

Your feedback helps us improve assistant answers and identify docs gaps we should fix.

Need more help? Join us on Discord: https://discord.gg/y9efgEmppm

Want to use your own agent? Add the Solo MCP server to query our docs directly. Get started here: https://search.solo.io/.