Logo

Search posts

Search blog posts by title, summary, tags, or content.

NVIDIA Nemo Agent Toolkit

NVIDIA Nemo Agent Toolkit

11 min read
Authors
#ai
On this page

Introduction

A local prototype agent is easy enough to build. It takes prompts, returns expected answers, and looks finished. Production is where the other problems start. NVIDIA calls these “day 2” problems: day 1 is building the agent; day 2 is everything around it.

Once several agents and model types share a system, the problems multiply:

  • Architecture: Popular frameworks do not share one way to call tools, store memory, or collect traces. LangChain, LlamaIndex, CrewAI, Semantic Kernel, and Google ADK each make different choices.
  • Repeatability: Results vary across environments and model versions.
  • Code reuse: Teams building similar workflows end up rebuilding the same pieces.
  • Performance: You need to understand the whole system to accelerate it; optimizing one component is not enough.
  • Observability: Without traces, debugging a multi-step agent execution is guesswork.
  • Security: Agents need access to tools, files, and APIs. Unrestricted access is not acceptable in an enterprise deployment.

A production system needs to:

  • Expose agents as APIs with consistent interfaces
  • Monitor what's happening across all agent steps
  • Check for edge cases through systematic evaluation
  • Use feedback to improve the workflow over time
  • Protect private data with configurable guardrails
  • Evaluate changes so regressions surface quickly

NVIDIA NeMo Agent Toolkit (NAT) targets these problems without forcing a rewrite. It works with agents built in LangChain, LangGraph, CrewAI, Semantic Kernel, Google ADK, LlamaIndex, or another framework, then adds the production layer around them.

What is NVIDIA NeMo Agent Toolkit?

NeMo Agent Toolkit is a modular, framework-agnostic Python library for taking agent workflows into production. NAT sits around your existing agents and provides:

  • Framework-agnostic integration: Each framework registers its LLM providers, embedders, and tools through the plugin system. The Builder resolves configuration types to implementation classes at runtime through a centralized registry.
  • YAML-driven configuration: Agents, tools, LLMs, evaluation, and telemetry live in declarative YAML files. Workflows become reproducible, version-controlled, and auditable.
  • Unified CLI: The nat command handles running, serving, evaluating, and profiling workflows.
  • Plugin ecosystem: Optional packages for LangChain (nvidia-nat[langchain]), LlamaIndex, CrewAI, and others install only what you need.

Core architecture

NAT uses a Subject-Observer pattern for telemetry. An IntermediateStepManager publishes workflow events to a reactive stream, and asynchronous exporters process them off the hot path.

Getting Started

Installation

# Core toolkit
pip install nvidia-nat

# With LangChain integration (most common)
pip install "nvidia-nat[langchain]"

# Verify installation
nat --version

Set your NVIDIA API key (free tier available at build.nvidia.com):

export NVIDIA_API_KEY=nvapi-...

Your first workflow

Create a workflow.yml:

functions:
  wikipedia_search:
    _type: wiki_search
    max_results: 2

llms:
  nim_llm:
    _type: nim
    model_name: nvidia/nemotron-3-nano-30b-a3b
    temperature: 0.0

workflow:
  _type: react_agent
  tool_names: [wikipedia_search]
  llm_name: nim_llm
  verbose: true
  parse_agent_response_max_retries: 3

Run it:

nat run --config_file workflow.yml --input "List five subspecies of Aardvarks"

This runs a ReAct agent with a Wikipedia search tool, powered by Nemotron on NVIDIA NIM, with retry logic built in.

Project scaffolding

# Generate a new project structure
nat workflow create my-first-agent

# Structure generated:
# my-first-agent/
#   configs/
#     config.yml        # Main workflow config
#     eval_config.yml   # Evaluation config
#   prompts/            # Prompt templates

Observability: Seeing Inside the Black Box

When a production agent fails, the hard part is finding out what happened. NAT's observability system uses an event-driven architecture with the IntermediateStepManager broadcasting workflow events to registered telemetry exporters.

Built-in exporters

NAT ships with exporters for:

  • LangSmith - Full trace visualization, feedback collection
  • Phoenix (Arize) - Open-source observability with embedding analysis
  • Langfuse - Open-source LLM observability with cost tracking
  • Weave (W&B) - Experiment tracking and evaluation
  • OpenTelemetry - Standard protocol for any OTel-compatible backend

Configuration

general:
  telemetry:
    logging:
      console:
        _type: console
        level: INFO

    tracing:
      langfuse:
        _type: langfuse
        public_key: ${LANGFUSE_PUBLIC_KEY}
        secret_key: ${LANGFUSE_SECRET_KEY}
        host: https://cloud.langfuse.com

      phoenix:
        _type: phoenix
        endpoint: http://localhost:6006

Multiple exporters run concurrently. Each processes the same event stream asynchronously, so adding observability doesn't slow down your agent.

What gets traced

  • LLM calls: Input tokens, output tokens, latency, model name, temperature
  • Tool invocations: Function name, arguments, return values, duration
  • Intermediate steps: Agent reasoning traces (thought/action/observation cycles)
  • Workflow-level metadata: Total duration, error counts, retry attempts

When an agent produces a wrong answer, you can trace the reasoning chain and find where it went wrong.

Evaluation: Catching Regressions Before Users Do

Agents are non-deterministic. The same input can produce different outputs across model versions, temperature changes, or prompt tweaks. Systematic evaluation is the only way to catch regressions.

NAT integrates evaluation directly into the workflow lifecycle. The nat eval command runs your workflow against a dataset and scores the outputs.

Defining an evaluation config

eval:
  general:
    dataset:
      _type: json
      file_path: ./data/eval_questions.json

    evaluators:
      correctness:
        _type: llm_as_judge
        llm_name: eval_llm
        rubric: |
          Score from 1-5:
          1: Completely wrong
          3: Partially correct
          5: Perfectly correct

      latency:
        _type: latency
        max_acceptable_ms: 5000

  profiler:
    compute_llm_metrics: true
    bottleneck_analysis:
      enable_nested_stack: true

Running evaluation

nat eval --config_file configs/eval_config.yml

The output gives you:

  • Per-question scores across all evaluators
  • Aggregate statistics (mean, median, p95)
  • Latency distributions
  • Bottleneck analysis showing which steps consume the most time
  • Token usage breakdowns (input vs output, per tool call)

Continuous evaluation in CI

# GitHub Actions example
- name: Run agent evaluation
  run: |
    nat eval --config_file configs/eval_config.yml
    # Fail if correctness drops below threshold

With evaluation in CI, prompt changes, model swaps, and new tools become measurable changes instead of guesses.

Profiling: Performance at the Tool Level

For latency-sensitive applications, you need to know exactly where time is spent. NAT's profiler instruments workflows down to individual tool calls and LLM invocations.

What the profiler measures

  • LLM metrics: Input/output token counts, tokens per second, time to first token
  • Tool latency: Per-tool execution time with nested call tracking
  • Bottleneck analysis: Identifies steps that dominate total latency
  • Concurrency spike analysis: Detects periods of high parallel tool usage
  • Token uniqueness: Measures token reuse across queries for KV cache optimization

Prediction Trie for Dynamo

The profiler can generate a prediction trie from execution traces. This hierarchical data structure captures statistics for each LLM call position. When deployed with NVIDIA Dynamo, the statistics become routing hints for KV cache management and request scheduling.

llms:
  dynamo_llm:
    _type: dynamo
    model_name: meta/llama-3.1-70b-instruct
    prediction_trie_path: ./profiles/workflow_trie.json

Deployment: From Script to API

Serving as a REST API

nat serve --config_file workflow.yml

This launches a FastAPI server with:

  • POST /run - Execute the workflow synchronously
  • POST /evaluate - Run evaluation against the workflow
  • OpenAPI schema auto-generated from your workflow config

Production deployment

# pyproject.toml dependency declaration
[project]
dependencies = [
    "nvidia-nat[langchain]==1.8.*",
]

FROM python:3.13-slim
COPY . /app
WORKDIR /app
RUN pip install .
EXPOSE 8000
CMD ["nat", "serve", "--config_file", "configs/workflow.yml", "--host", "0.0.0.0", "--port", "8000"]

Integration with existing services

The REST API can sit behind or alongside:

  • API gateways (Kong, NGINX, Traefik) for rate limiting and auth
  • Message queues (Kafka, RabbitMQ) for async agent execution
  • Monitoring stacks (Prometheus, Grafana) via the OpenTelemetry exporter
  • CI/CD pipelines for automated evaluation on every deploy

MCP and A2A: Inter-Agent Communication

Model Context Protocol (MCP)

NAT can act as an MCP client for remote tools and can publish its own tools as an MCP server.

functions:
  remote_tool:
    _type: mcp_client
    server_url: https://my-mcp-server/tools
    tool_name: search_database

Your workflow can then call tools from remote MCP servers, including Claude's MCP ecosystem.

Agent-to-Agent (A2A) Protocol

For distributed agent systems, NAT supports the A2A protocol. You can delegate tasks to remote A2A agents or publish a workflow as a discoverable A2A agent.

workflow:
  _type: a2a_agent
  agent_card:
    name: "Research Assistant"
    description: "Conducts deep research on technical topics"
    capabilities: ["web_search", "document_analysis", "summarization"]

For example, a research agent can delegate summarization and fact-checking to specialized agents, with NAT coordinating the calls.

The Broader NVIDIA Agent Ecosystem

NeMo Agent Toolkit is part of the agent stack NVIDIA announced at GTC 2026.

OpenShell

OpenShell is the secure sandboxed runtime for AI agents. It sits between your agent and your infrastructure, providing:

  • Isolated execution: Every agent runs in a sandbox with Landlock, seccomp, and network namespace isolation. Access is deny-by-default.
  • Privacy router: Routes inference requests based on sensitivity - local models for private data, cloud models for general tasks.
  • Granular permissions: Control what files, network endpoints, and system resources the agent can access.
  • Live policy updates: Change security policies without restarting the agent.
  • Full audit trail: Every action is logged for compliance.
# Create an OpenShell sandbox
openshell sandbox create --remote spark --from openclaw

# Run your NAT workflow inside the sandbox
openshell exec -- nat serve --config_file workflow.yml

NemoClaw

NemoClaw is an open-source reference stack that orchestrates the entire agent lifecycle:

  • Single-command deployment: nemoclaw CLI installs Nemotron models and OpenShell runtime in one step
  • Blueprint management: Versioned blueprints for reproducible sandbox creation
  • Routed inference: Transparent model routing - credentials stay on the host, the agent sees only inference.local
  • Declarative network policy: YAML-defined egress rules with real-time approval workflows
  • Posture profiles: Pre-configured security profiles for different risk levels (development, internal tools, production)
# Deploy an always-on agent with NemoClaw
nemoclaw deploy --agent my-workflow --provider anthropic

# The stack handles:
# - OpenShell sandbox creation
# - Model endpoint routing
# - Network policy enforcement
# - Lifecycle management

The full stack

Building a Complete Agentic Application

A technical research assistant is a useful example. It searches the web, summarizes findings, and exposes a REST API.

Step 1: Define tools

functions:
  web_search:
    _type: tavily_search
    max_results: 5
    include_domains: ["arxiv.org", "github.com", "docs.nvidia.com"]

  summarize:
    _type: llm_function
    llm_name: nim_llm
    system_prompt: |
      Summarize the following content in 3-5 bullet points.
      Focus on technical accuracy and actionable insights.

  save_to_file:
    _type: file_write
    base_path: ./research_outputs/

Step 2: Configure the LLM with thinking mode

llms:
  nim_llm:
    _type: nim
    model_name: nvidia/nemotron-3-nano-30b-a3b
    temperature: 0.0
    chat_template_kwargs:
      enable_thinking: true
    max_tokens: 4096

Step 3: Build the workflow

workflow:
  _type: plan_execute_agent
  planner_llm_name: nim_llm
  executor_llm_name: nim_llm
  tool_names: [web_search, summarize, save_to_file]
  max_steps: 10

Step 4: Add observability

general:
  telemetry:
    tracing:
      langfuse:
        _type: langfuse
        public_key: ${LANGFUSE_PUBLIC_KEY}
        secret_key: ${LANGFUSE_SECRET_KEY}

Step 5: Configure evaluation

eval:
  general:
    dataset:
      _type: json
      file_path: ./eval_questions.json

    evaluators:
      accuracy:
        _type: llm_as_judge
        llm_name: eval_llm
      completeness:
        _type: llm_as_judge
        llm_name: eval_llm
        rubric: "Does the answer cover all aspects of the question?"

Step 6: Deploy

# Development
nat run --config_file workflow.yml --input "Explain NVIDIA Nemotron architecture"

# Evaluation
nat eval --config_file configs/eval_config.yml

# Production
nat serve --config_file workflow.yml --host 0.0.0.0 --port 8000

Step 7: Frontend integration

// React component calling the NAT API
async function researchAgent(query: string) {
  const response = await fetch("http://localhost:8000/run", {
    method: "POST",
    headers: { "Content-Type": "application/json" },
    body: JSON.stringify({ input: query }),
  })
  return response.json()
}

Conclusion

NeMo Agent Toolkit handles the operational work around an agent: tracing, evaluation, profiling, serving, and protocol integrations. You can keep agent code built with LangChain, LangGraph, CrewAI, Semantic Kernel, Google ADK, or LlamaIndex while adding a common configuration and runtime layer.

NAT covers the workflow layer. OpenShell adds sandboxing and policy enforcement, while NemoClaw packages the surrounding deployment stack. Together, they give an agent a path from a local workflow to a controlled production service.

Sources

NeMo Agent Toolkit DocumentationGitHub RepositoryOpenShell DocumentationNemoClaw DocumentationNVIDIA NIM
Next postLinux: The Big PicturePrevious postBlog of February 2026
Discuss on X • View on GitHub