Skip to main content
Custom agents are in private beta. Email support@obss.app to enable them for your organization.
A custom agent lets you run your own agent, or one Obss doesn’t list, on your tasks next to the built-in agents. It’s any agent that supports the Agent Client Protocol (ACP), the open standard editors use to talk to AI agents. Use one to:
  • Evaluate an ACP agent that isn’t built in.
  • Test your own agent on your tasks before you publish it.
  • Compare an agent’s ACP version with the built-in agent.
Connecting a custom agent is optional step 4 of the setup checklist. Obss installs the agent in each run’s machine, where it works with the task’s files and services and the treatment’s tools. It’s an agent, not a treatment: you choose it at launch, give it treatments, and read its results in the same places as a built-in agent.
An agent that runs its tools on its own computer, such as Ask Obss, can’t use the run’s files, services, or tools, so give it tasks that don’t need them (Run Ask Obss as a custom agent).

Who can see a custom agent

Your organization owns each custom agent, and every project lists it under Manage → Agents with a Custom tag. It’s on in the project where you created it and off elsewhere until you turn it on. Every run uses the same environment variable values, in every project and for every member.

Connect a custom agent

  1. Open Manage → Agents and select Connect custom agent.
  2. Enter a unique name of 120 characters or fewer in Name this agent. Optional: select Add a description.
  3. Under Install, choose ACP Registry, npm package, Python package, Download, or Import JSON (explained below).
  4. Under Environment variables, select Add variable for each value the agent reads.
  5. Select Connect agent, then test the agent on the page that opens.
The Connect custom agent form has the install choices, the ACP Registry search, the environment variables, and Advanced.
Connect custom agent form with the name, the Install choices, the ACP Registry search list, and Environment variables.Connect custom agent form with the name, the Install choices, the ACP Registry search list, and Environment variables.

ACP Registry

Under Agent, search the registry by name, ID, or description. Each row shows the agent’s version and description.
  • Obss pins the agent’s current version when you choose it, and runs use that version until you change it. When the registry has a newer version, the agent page shows Update to. Select it, then select Save changes.
  • The registry can pause an agent with a problem, and paused agents aren’t listed. If you choose one from the CLI or the plugin, the command fails and shows the registry’s reason. Choose another agent, or install it as a package.
  • For agents that Obss recognizes, it adds the variable names the agent usually reads, with empty values. Remove the ones you don’t need. For example, goose adds GOOSE_PROVIDER and GOOSE_MODEL, with a note to set the API key of the provider GOOSE_PROVIDER names, such as ANTHROPIC_API_KEY.
  • Without your own description, Obss uses the registry’s.
  • An Also built in tag means Obss has a built-in version. Add the registry version only to compare the two.

npm package, Python package, or Download

Use these for your own agent or one that isn’t in the registry. The CLI and the plugin call all three the manual source. Then enter the Arguments, separated by spaces, with quotes around any argument that has spaces.
  • Packages need an exact version, which keeps results comparable. If the package or version doesn’t exist, the install fails with the npm or uv error.
  • Obss checks a download against its SHA-256 (64 hexadecimal characters) before running it.
  • A .tar.gz, .tgz, .tar.bz2, or .zip download is an archive: Obss extracts it, and Command is the path to run inside it. Any other download is a binary that Obss saves under the Command name and runs.

Import JSON

Paste a standard ACP Registry agent.json, or select Choose file, to test an entry before you publish it.
  • The entry needs an id, name, version, and a Linux x64 distribution: npx, uvx, or binary with a linux-x86_64 target and its SHA-256.
  • A local command can’t run, because nothing is preinstalled in a run’s machine.

Set environment variables

An ACP agent can’t sign in through a browser during a run, so give it its credentials, such as its model provider’s API key, as environment variables (up to 64).
  • Names use capital letters, digits, and underscores, and start with a letter. Don’t use names Obss sets for the run, such as PATH, HOME, and anything starting with OQO_.
  • A saved name doesn’t change. To rename a variable, remove it and add a new one.
  • Values are secret, and you enter them only in the dashboard. The CLI and the plugin return a dashboard link for each value instead. Every run of the agent can read them, and Obss removes them from each run’s saved recorded work.
The ACP Registry doesn’t say which variables an agent needs, so an empty list doesn’t mean it needs none. Most agents need their model provider’s API key. If the agent can’t sign in, Test agent shows the sign-in methods it reports.

Test agent

Test agent checks that the agent installs, starts, and answers. It uses one run (Uses 1 run) with:
  • the Agent check task, which asks the agent for its name and what it can help with, in one sentence
  • the Agent default model
  • the Raw Agent treatment (no tools)
  • Evals and Insights off
While it runs, the status shows Testing, and Open test run opens the run. The status updates when the run ends. A test is a check, not an experiment: it sends no notice and doesn’t show on Home or in the experiments list, though the CLI and the plugin list it. It has no project files, so if your agent needs them, run an experiment with a task that has them.
On the agent page, under Status, select Test agent.
When Test agent can’t start, the disabled button, the CLI, and the plugin say why:

Statuses

The status shows on the agent page, Manage → Agents, and New experiment. When two apply, the higher one in this table wins. Any finished run with the current settings, in any project, updates the status, except a cancelled run or one where the agent refused the treatment’s MCP servers. Saving a change to the settings or a value resets it to Not tested. Open latest run opens the run that set the status, and the CLI and the plugin also give the agent’s reported version and the kinds of MCP server it accepts.

When the agent needs attention

Status shows the step that failed, the agent’s own error, and the fix, with a link such as Go to Models to the right section or the run. When the agent asks for a sign-in, the page lists its sign-in methods, which often name the variable it reads.

Confirm the agent’s models

Obss learns the agent’s models from each run, so Models has no Refresh models button.
  • Agent default is always first. Obss doesn’t choose a model, so the agent uses its own default. Some agents read the model from a variable, such as GOOSE_MODEL for goose.
  • After a run, the list shows each model the agent offered. Turn on the ones you want to launch with. You can’t add a model that the agent doesn’t offer.
An agent with no model option has only Agent default. Custom agents have no effort setting.

Run a custom agent in an experiment

New experiment lists custom agents after the built-in ones, with the Custom tag and, unless the agent is Ready, its status. Each model you turned on is a choice. Selecting New experiment on the agent’s page opens an experiment with this agent on Agent default beside the built-in agents, for comparison. To run only this agent, select Clear beside the agent search and select it again. Nothing runs until you select Launch.
  • New experiment is the page’s main action once the agent is Ready. Before that, Test agent is the main action.
  • The page has no New experiment while the agent is off in the project (turn on In this project) or has no models turned on (follow Go to Models).
  • An experiment that you open another way doesn’t select a custom agent for you. When you select one, it starts on Agent default.
A launch from the CLI or the plugin must name its agents: find the model IDs with obss catalog list --kind agent --projection full (plugin: inspect_catalog with kinds: ["agent"]). Before each run, Obss copies the treatment’s skills into the agent’s skills folder, ~/.agents/skills by default. If the agent uses another folder, set Advanced → Skills folder on the agent page.

Read a custom agent’s results

Results work as for a built-in agent, with these differences:
  • Models show under the agent’s own names, and Agent default adds the reported model, such as Agent default: opus.
  • An answer that ends early shows a note, such as “Stopped early: the agent reached its token limit.” Evals score it as usual.
  • Tokens and cost show only when the agent reports them. Otherwise the Runs cost cell is blank, because Obss never estimates a custom agent’s cost.
  • When the agent fails before it answers, the run status says “The agent failed before it answered.” A banner shows the failed step, the agent’s error, and the fix, with a link such as Go to Models. Evals don’t score the run, and as an errored run it’s left out of the pass rate and doesn’t use your run balance.

Where a cost comes from

Every cost names its source, beside Cost on a run or on hover in the Runs list. A Matrix cell whose runs mix sources shows Mixed sources, with the mean cost per source rather than one combined value.

Change or remove a custom agent

Change settings on the agent page and select Save changes. You can also change how it installs: another install choice, another ACP Registry agent, or a newer version. Past runs keep their launch settings, and the status resets to Not tested.
  1. Open the agent page.
  2. Open the ⋯ menu and select Remove.
  3. Select Remove agent.
Removing an agent takes it off the Agents list and out of new experiments, and Obss clears its variable values. Past results keep its name. You can’t remove an agent while one of its runs, a test or an experiment, is in progress, so wait for it to finish or cancel it first.

From the CLI and the plugin

The CLI and the plugin offer the same operations, except that variable values are set in the dashboard: a create or update returns a link for each one. An update or removal needs the current revision from obss project ai inspect as expectedRevision. If the revision is stale, the command changes nothing. To change how an agent installs, send update_custom_agent with a new source.

Next steps