# Custom agents - Obss

Set up
Custom agents
Add an Agent Client Protocol (ACP) agent to your organization, test it, and run it in experiments next to the built-in agents.
Custom agents are in private beta. Email
support@obss.app
to enable them for your organization.
A
custom agent
lets you run your own agent, or one Obss doesn’t list, on your tasks next to the built-in agents. It’s any agent that supports the
Agent Client Protocol
(ACP), the open standard editors use to talk to AI agents. Use one to:
Evaluate an ACP agent that isn’t built in.
Test your own agent on your tasks before you publish it.
Compare an agent’s ACP version with the built-in agent.
Connecting a custom agent is optional step 4 of the
setup checklist
.
Obss installs the agent in each run’s machine, where it works with the task’s files and services and the treatment’s tools. It’s an agent, not a treatment: you choose it at launch, give it
treatments
, and read its results in the same places as a built-in agent.
An agent that runs its tools on its own computer, such as Ask Obss, can’t use the run’s files, services, or tools, so give it tasks that don’t need them (
Run Ask Obss as a custom agent
).
Who can see a custom agent
Your organization owns each custom agent, and every project lists it under
Manage → Agents
with a
Custom
tag. It’s on in the project where you created it and off elsewhere until you turn it on. Every run uses the same environment variable values, in every project and for every member.
Connect a custom agent
Dashboard
CLI
Plugin (MCP)
Open
Manage → Agents
and select
Connect custom agent
.
Enter a unique name of 120 characters or fewer in
Name this agent
. Optional: select
Add a description
.
Under
Install
, choose
ACP Registry
,
npm package
,
Python package
,
Download
, or
Import JSON
(explained below).
Under
Environment variables
, select
Add variable
for each value the agent reads.
Select
Connect agent
, then
test the agent
on the page that opens.
Create the agent with its variable names. For example, from the ACP Registry:
obss
project
ai
apply
--operation
'{"action":"create_custom_agent","name":"Gemini CLI","source":{"kind":"registry","registryId":"gemini"},"env":["GEMINI_API_KEY"]}'
--output
json
Or from an npm package:
obss
project
ai
apply
--operation
'{"action":"create_custom_agent","name":"My agent","source":{"kind":"manual","install":{"kind":"npm","package":"my-agent@1.4.2"},"args":"acp --verbose"},"env":["MY_AGENT_API_KEY"]}'
--output
json
The result gives the agent key (
acp-…
), the agent page, and a dashboard link for each value to set (
obss project ai apply
).
Ask your AI assistant to connect the custom agent. It calls
Create custom agents and change agents, models, and Obss AI settings
(
manage_project_ai
) and returns a dashboard link for each value to set.
Add the Gemini CLI from the ACP Registry as a custom agent in Obss.
The
Connect custom agent
form has the install choices, the ACP Registry search, the environment variables, and
Advanced
.
ACP Registry
Under
Agent
, search the registry by name, ID, or description. Each row shows the agent’s version and description.
Obss pins the agent’s current version when you choose it, and runs use that version until you change it. When the registry has a newer version, the agent page shows
Update to
. Select it, then select
Save changes
.
The registry can pause an agent with a problem, and paused agents aren’t listed. If you choose one from the CLI or the plugin, the command fails and shows the registry’s reason. Choose another agent, or install it as a package.
For agents that Obss recognizes, it adds the variable names the agent usually reads, with empty values. Remove the ones you don’t need. For example,
goose
adds
GOOSE_PROVIDER
and
GOOSE_MODEL
, with a note to set the API key of the provider
GOOSE_PROVIDER
names, such as
ANTHROPIC_API_KEY
.
Without your own description, Obss uses the registry’s.
An
Also built in
tag means Obss has a built-in version. Add the registry version only to compare the two.
npm package, Python package, or Download
Use these for your own agent or one that isn’t in the registry. The CLI and the plugin call all three the
manual
source.
Install
What you enter
npm package
The package with its exact version, such as
my-agent@1.4.2
Python package
The package with its exact version, such as
my-agent==1.4.2
Download
The
Download URL
(HTTPS) of a Linux x64 archive or binary, its
SHA-256
, and the
Command
: the file to run in the download
Then enter the
Arguments
, separated by spaces, with quotes around any argument that has spaces.
Packages need an exact version, which keeps results comparable. If the package or version doesn’t exist, the install fails with the npm or uv error.
Obss checks a download against its SHA-256 (64 hexadecimal characters) before running it.
A
.tar.gz
,
.tgz
,
.tar.bz2
, or
.zip
download is an archive: Obss extracts it, and
Command
is the path to run inside it. Any other download is a binary that Obss saves under the
Command
name and runs.
Import JSON
Paste a standard ACP Registry
agent.json
, or select
Choose file
, to test an entry before you publish it.
The entry needs an
id
,
name
,
version
, and a Linux x64 distribution:
npx
,
uvx
, or
binary
with a
linux-x86_64
target and its SHA-256.
A
local
command can’t run, because nothing is preinstalled in a run’s machine.
Set environment variables
An ACP agent can’t sign in through a browser during a run, so give it its credentials, such as its model provider’s API key, as environment variables (up to 64).
Names use capital letters, digits, and underscores, and start with a letter. Don’t use names Obss sets for the run, such as
PATH
,
HOME
, and anything starting with
OQO_
.
A saved name doesn’t change. To rename a variable, remove it and add a new one.
Values are secret, and you enter them only in the dashboard. The CLI and the plugin return a dashboard link for each value instead. Every run of the agent can read them, and Obss removes them from each run’s saved recorded work.
The ACP Registry doesn’t say which variables an agent needs, so an empty list doesn’t mean it needs none. Most agents need their model provider’s API key. If the agent can’t sign in,
Test agent
shows the sign-in methods it reports.
Test agent
Test agent
checks that the agent installs, starts, and answers. It uses one run (
Uses 1 run
) with:
the
Agent check
task, which asks the agent for its name and what it can help with, in one sentence
the
Agent default
model
the
Raw Agent
treatment (no tools)
Evals and Insights off
While it runs, the status shows
Testing
, and
Open test run
opens the run. The status updates when the run ends.
A test is a check, not an experiment: it sends no notice and doesn’t show on Home or in the experiments list, though the CLI and the plugin list it. It has no project files, so if your agent needs them, run an experiment with a task that has them.
Dashboard
CLI
Plugin (MCP)
On the agent page, under
Status
, select
Test agent
.
obss
project
ai
test-agent
acp-0123456789abcdef
obss
experiments
watch
<
launch-i
d
>
The CLI prints an attempt ID. Pass it as
--attempt-id
to retry without a second run (
obss project ai test-agent
).
Ask your AI assistant to test the custom agent. It calls
Test a custom agent
(
test_custom_agent
), reuses the same
launch_attempt_id
to retry without a second run, and polls
Get launch status
(
get_launch_status
). A test uses a run, so turn on tool approval in your AI assistant.
When Test agent can’t start, the disabled button, the CLI, and the plugin say why:
Message
What to do
Save or discard your changes first.
Save the agent page, or discard the changes.
A test with the current settings is running. Open its run.
Select
Open test run
.
Turn on this agent in this project.
On the agent page, turn on
In this project
.
Set
NAME
.
Enter each value under
Environment variables
.
Turn on Agent default. Test agent uses it.
Turn on
Agent default
under
Models
.
Statuses
The status shows on the agent page,
Manage → Agents
, and
New experiment
. When two apply, the higher one in this table wins.
Status
Meaning
Needs values
An environment variable has no value. Set a value for each one.
Testing
A Test agent run with the current settings is in progress.
Needs attention
The agent did not answer its latest run with the current settings.
Not tested
No run with the current settings has finished.
Ready
The agent answered its latest run with the current settings.
Any finished run with the current settings, in any project, updates the status, except a cancelled run or one where the agent refused the treatment’s MCP servers. Saving a change to the settings or a value resets it to
Not tested
.
Open latest run
opens the run that set the status, and the CLI and the plugin also give the agent’s reported version and the kinds of MCP server it accepts.
When the agent needs attention
Status
shows the step that failed, the agent’s own error, and the fix, with a link such as
Go to Models
to the right section or the run. When the agent asks for a sign-in, the page lists its sign-in methods, which often name the variable it reads.
What failed
Fix
The agent did not accept the run’s MCP servers
Run this agent with a treatment that has no MCP servers.
The agent asks for a sign-in
Set the sign-in values this agent reads in its environment variables. Then test the agent again.
The agent did not accept the model
Use a model that the agent lists. Turn this model off in
Models
, or run with
Agent default
.
The download does not match its SHA-256
Check that the download URL names the right file. Then enter that file’s SHA-256.
An npm or Python package agent could not be installed
Check the package name and its exact version. The error shows the message from npm or uv.
An npm or Python package agent did not start
Check the package name and its exact version, and the arguments. The error shows the message from npm or uv.
The agent can’t run on the machine’s Node.js
Change Node.js in the machine’s setup
, or choose another agent.
A
Download
agent could not be installed or did not start
Check the download URL, its SHA-256, and the command.
An
ACP Registry
agent could not be installed
Test the agent again. If it fails again, change its install to its npm package, Python package, or download.
An
ACP Registry
agent did not start or connect
Read the last lines of the agent’s log in the run.
The agent did not start a session or answer the prompt
Read the agent’s error. If it asks you to sign in, set the sign-in values that this agent reads in its environment variables. Then test the agent again.
Confirm the agent’s models
Obss learns the agent’s models from each run, so
Models
has no
Refresh models
button.
Agent default
is always first. Obss doesn’t choose a model, so the agent uses its own default. Some agents read the model from a variable, such as
GOOSE_MODEL
for goose.
After a run, the list shows each model the agent offered. Turn on the ones you want to launch with. You can’t add a model that the agent doesn’t offer.
Model state
Meaning
Current
A run with the current settings listed this model, or it is
Agent default
.
Not confirmed
The agent listed this model with older settings. Test the agent to confirm it.
Unavailable
The agent did not accept this model with the current settings.
An agent with no model option has only
Agent default
. Custom agents have no effort setting.
Run a custom agent in an experiment
New experiment
lists custom agents after the built-in ones, with the
Custom
tag and, unless the agent is
Ready
, its status. Each model you turned on is a choice.
Selecting
New experiment
on the agent’s page opens an experiment with this agent on
Agent default
beside the built-in agents, for comparison. To run only this agent, select
Clear
beside the agent search and select it again. Nothing runs until you select
Launch
.
New experiment
is the page’s main action once the agent is
Ready
. Before that,
Test agent
is the main action.
The page has no
New experiment
while the agent is off in the project (turn on
In this project
) or has no models turned on (follow
Go to Models
).
An experiment that you open another way doesn’t select a custom agent for you. When you select one, it starts on
Agent default
.
Launch state
Cause
What to do
Blocked
A selected custom agent has a variable with no value
Set the value in the dashboard
Blocked
A treatment has an MCP server the agent can’t accept
Remove the agent or the treatment
Allowed, with a notice
The agent is
Not tested
or
Needs attention
(for example, “My agent is not tested yet.” with
Open agent
)
Test the agent before a full experiment
A launch from the CLI or the plugin must name its agents: find the model IDs with
obss catalog list --kind agent --projection full
(plugin:
inspect_catalog
with
kinds: ["agent"]
).
Before each run, Obss copies the treatment’s skills into the agent’s skills folder,
~/.agents/skills
by default. If the agent uses another folder, set
Advanced → Skills folder
on the agent page.
Read a custom agent’s results
Results work as for a built-in agent, with these differences:
Models show under the agent’s own names, and
Agent default
adds the reported model, such as
Agent default: opus
.
An answer that ends early shows a note, such as “Stopped early: the agent reached its token limit.” Evals score it as usual.
Tokens and cost show only when the agent reports them. Otherwise the
Runs
cost cell is blank, because Obss never estimates a custom agent’s cost.
When the agent fails before it answers, the run status says “The agent failed before it answered.” A banner shows the failed step, the agent’s error, and the fix, with a link such as
Go to Models
. Evals don’t score the run, and as an errored run it’s left out of the pass rate and doesn’t use your
run balance
.
Where a cost comes from
Every cost names its source, beside
Cost
on a run or on hover in the Runs list.
Label
Meaning
Reported by the agent
The agent reported its own total.
Estimated from list prices
Obss calculated the cost from token counts and list prices.
No label
The run’s record does not say where the cost came from.
A Matrix cell whose runs mix sources shows
Mixed sources
, with the mean cost per source rather than one combined value.
Change or remove a custom agent
Change settings on the agent page and select
Save changes
. You can also change how it installs: another install choice, another ACP Registry agent, or a newer version. Past runs keep their launch settings, and the status resets to
Not tested
.
Dashboard
CLI
Plugin (MCP)
Open the agent page.
Open the
⋯
menu and select
Remove
.
Select
Remove agent
.
obss
project
ai
delete
--operation
'{"action":"remove_custom_agent","agentKey":"acp-0123456789abcdef","expectedRevision":4}'
Pass the current
revision
from
obss project ai inspect --view agents
. In a terminal, the CLI asks you to confirm. With
--output json
or outside a terminal, pass
--yes
(
obss project ai delete
).
Ask your AI assistant to remove the custom agent. It calls
Remove custom agents or provider connections
(
delete_project_ai
), and it asks you to approve the call.
Removing an agent takes it off the Agents list and out of new experiments, and Obss clears its variable values. Past results keep its name. You can’t remove an agent while one of its runs, a test or an experiment, is in progress, so wait for it to finish or cancel it first.
From the CLI and the plugin
The CLI and the plugin offer the same operations, except that variable values are set in the dashboard: a create or update returns a link for each one.
Task
CLI
Plugin (MCP)
List agents with each custom agent’s status, variables, note about variables (
envNote
), and fix
obss project ai inspect --view agents
inspect_project_ai
with
view=agents
Find an agent in the ACP Registry
obss project ai inspect --view agent_registry --query gemini
inspect_project_ai
with
view=agent_registry
Create or change a custom agent
obss project ai apply
with
create_custom_agent
or
update_custom_agent
manage_project_ai
Test a custom agent
obss project ai test-agent
test_custom_agent
List its models
obss project ai inspect --view models --agent-key
inspect_project_ai
with
view=models
and
agent_key
Turn a model on or off
obss project ai apply
with
set_model_enabled
manage_project_ai
Read a run’s stop reason, reported model, and failure
obss runs trajectory
,
obss runs profile
get_run_trajectory
,
get_run_profile
Remove a custom agent
obss project ai delete
with
remove_custom_agent
delete_project_ai
An update or removal needs the current
revision
from
obss project ai inspect
as
expectedRevision
. If the revision is stale, the command changes nothing. To change how an agent installs, send
update_custom_agent
with a new
source
.
Next steps
Obss AI features
Step 5: choose models for Evals, Insights, and Ask Obss.
Run Ask Obss as a custom agent
Install Ask Obss from a download and give it an API key.
Compare agents and models
Compare your custom agent with the built-in agents.
Launch an experiment
Choose your custom agent at launch.
Agents
Obss AI features
Powered by
This documentation is built and hosted on Mintlify, a developer documentation platform