PandaProbe - Open source agent engineering platform

PandaProbe is an open-source agent engineering platform designed for building, testing, and monitoring AI agents in production environments. It focuses on observability and evaluation for agent systems by capturing full execution traces, analyzing long-running behaviors, and providing structured evaluation metrics. The platform is aimed at developers working with LLM-based agents who need deeper insights into tool usage, decision paths, and performance regression over time.

1. Agent tracing and observability

   Captures complete agent trajectories including tool calls, LLM interactions, metadata, and decision branches. Supports one-line instrumentation across major agent frameworks.

2. Evals & metrics for agent behavior

   Provides research-based evaluation metrics to assess uncertainty, drift, and performance across full agent sessions instead of isolated requests. Includes LLM-as-judge scoring with structured feedback.

3. Monitoring and regression detection

   Enables scheduled evaluation runs (hourly, daily, or cron-based) to detect performance regressions and behavioral drift in production systems.

4. Multi-framework and LLM integration

   Works with frameworks like LangChain, LangGraph, CrewAI, Google ADK, Claude Agent SDK, and OpenAI Agents SDK, as well as providers such as OpenAI, Anthropic, Gemini, Mistral, and AWS Bedrock.

5. Developer-friendly CLI and Skills system

   Includes a CLI tool for trace and eval management and a “Skills” integration that allows coding agents (e.g., Cursor, Claude Code) to interact with PandaProbe via natural language workflows.

6. Deployment flexibility (Cloud or self-hosted)

   Supports managed cloud usage as well as full open-source self-hosting under an Apache 2.0 license.

• Install the agent skill integration using:
  • npx skills add chirpz-ai/pandaprobe-skills
• Instrument Google ADK agents at startup by initializing the adapter before creating agents.
• Use CLI commands to manage workflows:
  • pandaprobe traces list
  • pandaprobe evals runs create
  • pandaprobe evals scores list
• Schedule evaluation runs regularly to catch regressions early in production.
• Combine tracing + evaluation instead of using them separately for better debugging coverage.

What is PandaProbe?

PandaProbe is an open-source platform for monitoring and evaluating AI agents, providing tracing, metrics, and performance analysis tools.

What does PandaProbe help me with?

It helps track agent behavior, debug tool usage, evaluate long-running workflows, and detect performance issues before they affect users.

Can I use just tracing without other features?

Yes, tracing can be used independently as the foundational layer for collecting agent execution data.

What deployment options exist?

It supports cloud-hosted usage as well as full self-hosting under an open-source license.

Is self-hosting actually free?

Yes, core features can be self-hosted freely under the Apache 2.0 license.

What frameworks are supported?

It supports LangChain, LangGraph, CrewAI, Google ADK, Claude Agent SDK, OpenAI Agents SDK, and more.

What's the latency impact?

Not explicitly specified in the provided content.

How do I get started?

You can install the CLI/skills package, integrate tracing into your agent framework, and start running evaluations on agent sessions.

How does pricing work?

Pricing is tiered across Hobby ($0/month), Pro ($29/month), Startup ($299/month), Enterprise (custom), and an open-source self-hosted option.

PandaProbe stands out as a specialized observability and evaluation layer for AI agents, especially useful for teams building production-grade LLM workflows. Its strongest advantage is the deep tracing capability combined with structured evaluation metrics, which makes debugging multi-step agents significantly easier.

On the downside, the platform may feel complex for beginners who only need simple logging or basic monitoring. Also, some advanced capabilities depend on proper integration setup and production-scale usage, which can introduce overhead during onboarding.

Overall, it is a strong fit for engineering teams focused on reliability, evaluation discipline, and scaling agent systems beyond prototypes.