Custom agents are in private beta. Email support@obss.app to enable them for your organization.
Prerequisites
- Ask Obss works for you on the Ask Obss page: it has a model under Obss AI features, and you connected it to your projects.
- The ID of the project that Ask Obss should use, from
obss projects currentor the dashboard URL (/projects/prj_…).
Steps
1
Create a personal API key
Nobody can sign in through a browser during a run, so the CLI signs in with a personal API key from the
OBSS_API_KEY environment variable. In Profile → API keys, create a key and copy it.2
Find the Linux x64 release of the CLI
Each run downloads the Obss CLI, which serves Ask Obss over the Agent Client Protocol (ACP) with
obss acp ask-obss. Runs use its Linux x64 build, obss-linux-x64. To find that build’s URL and SHA-256:- Open
https://obss.app/installin a browser. The installer script names the release location of thestablechannel. - Open the release manifest,
latest.json, at that location. - Copy the
urlandsha256of thelinux-x64entry.
OBSS_API_KEY is set. For a newer build, enter a new url and sha256 in the agent’s settings, then save.3
Create the custom agent
In Manage → Agents, select Connect custom agent and enter:
Under Environment variables, add
OBSS_API_KEY with your key as its value, then select Connect agent. Obss saves the download, a binary rather than an archive, as obss and runs it with the arguments.4
Test the agent
Select Test agent. The test asks Ask Obss for its name and what it can help with, and the status updates on the page. When it answers, the status is Ready and Models lists the models it offers: turn on the ones to compare.If the status is Needs attention, match the error:
5
Write tasks about your projects
Ask Obss calls its tools through your Obss account, not on the run’s machine, so it can’t use the run’s files, services, or a treatment’s tools. Write tasks that ask about your projects instead, such as “How many tasks does this project have?”, with a rubric that judges the reply.
6
Run an experiment
Select New experiment, and choose your tasks, Ask Obss with the models to compare, and the Raw Agent treatment (no tools), because a treatment’s tools don’t reach Ask Obss. Then select Launch.
Interpret the results
On the experiment’s Matrix with the Evals metric, each cell shows the share of trials in which Ask Obss’s reply passed the rubric, for each task and model. Switch the metric to Cost, Tokens, or Duration to compare what each model costs.From the CLI and the plugin
Only the agent’s API key stays in the dashboard. To create the same agent with the CLI, run:OBSS_API_KEY. Then test the agent:
obss project ai apply and obss project ai test-agent. From the plugin, send the same operation to Create custom agents and change agents, models, and Obss AI settings (manage_project_ai), then call Test a custom agent (test_custom_agent).
Other ways to run it
- Run your own ACP agent. Follow the same steps with your agent’s npm package, Python package, download, or ACP Registry entry. See Custom agents.
- Track the answers over time. Schedule the experiment to see whether the answers change as your projects do. See Track results over time.
