Skip to main content
Custom agents are in private beta. Email support@obss.app to enable them for your organization.
To measure how well Ask Obss answers questions about your projects, add it as a custom agent and score its replies with tasks and rubrics, as you would for any agent.

Prerequisites

  • Ask Obss works for you on the Ask Obss page: it has a model under Obss AI features, and you connected it to your projects.
  • The ID of the project that Ask Obss should use, from obss projects current or the dashboard URL (/projects/prj_…).
The API key acts as you in every run of this agent, including runs that a teammate launches. In those runs, Ask Obss can read and change your projects, as it can on the Ask Obss page, and its chats appear in your Ask Obss history.

Steps

1

Create a personal API key

Nobody can sign in through a browser during a run, so the CLI signs in with a personal API key from the OBSS_API_KEY environment variable. In Profile → API keys, create a key and copy it.
2

Find the Linux x64 release of the CLI

Each run downloads the Obss CLI, which serves Ask Obss over the Agent Client Protocol (ACP) with obss acp ask-obss. Runs use its Linux x64 build, obss-linux-x64. To find that build’s URL and SHA-256:
  1. Open https://obss.app/install in a browser. The installer script names the release location of the stable channel.
  2. Open the release manifest, latest.json, at that location.
  3. Copy the url and sha256 of the linux-x64 entry.
The URL names one release, so runs keep that exact build, and the CLI doesn’t update itself while OBSS_API_KEY is set. For a newer build, enter a new url and sha256 in the agent’s settings, then save.
3

Create the custom agent

In Manage → Agents, select Connect custom agent and enter:Under Environment variables, add OBSS_API_KEY with your key as its value, then select Connect agent. Obss saves the download, a binary rather than an archive, as obss and runs it with the arguments.
4

Test the agent

Select Test agent. The test asks Ask Obss for its name and what it can help with, and the status updates on the page. When it answers, the status is Ready and Models lists the models it offers: turn on the ones to compare.If the status is Needs attention, match the error:
5

Write tasks about your projects

Ask Obss calls its tools through your Obss account, not on the run’s machine, so it can’t use the run’s files, services, or a treatment’s tools. Write tasks that ask about your projects instead, such as “How many tasks does this project have?”, with a rubric that judges the reply.
6

Run an experiment

Select New experiment, and choose your tasks, Ask Obss with the models to compare, and the Raw Agent treatment (no tools), because a treatment’s tools don’t reach Ask Obss. Then select Launch.

Interpret the results

On the experiment’s Matrix with the Evals metric, each cell shows the share of trials in which Ask Obss’s reply passed the rubric, for each task and model. Switch the metric to Cost, Tokens, or Duration to compare what each model costs.

From the CLI and the plugin

Only the agent’s API key stays in the dashboard. To create the same agent with the CLI, run:
The result gives the agent key and the dashboard link where you set OBSS_API_KEY. Then test the agent:
See obss project ai apply and obss project ai test-agent. From the plugin, send the same operation to Create custom agents and change agents, models, and Obss AI settings (manage_project_ai), then call Test a custom agent (test_custom_agent).

Other ways to run it

  • Run your own ACP agent. Follow the same steps with your agent’s npm package, Python package, download, or ACP Registry entry. See Custom agents.
  • Track the answers over time. Schedule the experiment to see whether the answers change as your projects do. See Track results over time.

Next steps