Oqoqo - The simplest method to create evaluations and personalized benchmarks for practical agentic tasks.

Oqoqo is a platform designed to facilitate the creation of evaluations and custom benchmarks for real-world agentic tasks. It allows users to run evaluation experiments at scale in realistic environments using fully managed cloud infrastructure. With Oqoqo, users can define custom task sets to build private benchmarks, measure agent performance, and gain insights into various operational metrics.

1. Custom Task Sets

   Users can define their own task sets to create private benchmarks tailored to their specific needs. This feature allows for a more personalized evaluation process.

2. Scalable Experiments

   Oqoqo supports running experiments at scale, enabling users to evaluate multiple agents and models simultaneously in a controlled environment.

3. Managed Cloud Infrastructure

   The platform provides fully managed cloud infrastructure, ensuring that large experiments run quickly and reliably without the need for users to manage the underlying hardware.

4. Detailed Insights

   Users can generate insights on the fly, including metrics such as product interface friction, token inefficiencies, and performance differences, which help in understanding agent behavior.

5. Rich Customizable Environments

   Each task runs on its own sandboxed machine, allowing users to customize dependencies and tooling as needed for their specific experiments.

6. Comprehensive Evaluation Metrics

   Oqoqo captures detailed metrics for each run, including pass rates, tokens used, and frictions encountered, providing a thorough analysis of agent performance.

1. Start by signing in to your Oqoqo account and navigating to the dashboard.
2. Click on "New Experiment" to begin setting up your evaluation.
3. Define your tasks, agents, treatments, and rubrics to customize your experiment.
4. Launch the experiment and monitor the progress of each run in real-time.
5. Review the results to analyze pass/fail rates and identify areas for improvement.

What is Oqoqo for?

Oqoqo is designed for creating evaluations and private benchmarks for real-world agentic tasks, allowing users to assess whether agents can effectively use various products.

How is this different from trying an agent myself?

Oqoqo enables repeated evaluations across agents, models, and treatments, providing comprehensive data on performance rather than a one-time test.

Can I build private benchmarks?

Yes, users can write their own tasks and rubrics in plain language, which are versioned with each experiment for comparability.

What do I see after a run?

After a run, users receive detailed reports on pass or fail outcomes, including the full trajectory of the experiment, metrics, and reasons for failures.

Which agents can I run?

Oqoqo supports various agents, including Claude Code, Codex, Cursor, GitHub Copilot, and more, allowing users to choose models based on their needs.

Price data is not available yet; please visit the official Oqoqo website for the latest pricing information.

1. Oqoqo effectively streamlines the process of evaluating agents and models, making it easier for teams to conduct experiments and analyze results.
2. The platform's ability to run experiments in isolated environments enhances reliability and accuracy in testing.
3. However, the complexity of setting up custom tasks and benchmarks may pose a challenge for new users, requiring a learning curve to fully utilize the platform's capabilities.
4. While Oqoqo offers a robust feature set, ongoing support and documentation could be improved to assist users in navigating the platform more effectively.