Agent and automation rules

This page is for developers building scripts, internal tools, or agents on top of CloudEval.

Start with safe defaults

Use these defaults unless you have a good reason not to:

cloudeval capabilities --format json
cloudeval doctor --format json
cloudeval doctor --mcp --format json before MCP client setup
--format json
--non-interactive
--profile <name>
--print-url --no-open
stored cloudeval login --headless auth, or --machine when service-principal credentials are configured
--output <file> when the result must be persisted

These defaults reduce ambiguity and make the CLI easier to compose with other systems.

Stdout and stderr contract

For machine-readable commands:

stdout is the data channel
stderr is for prompts, warnings, auth flow text, and browser-open messages

Do not parse terminal UI output. If you need automation, use explicit subcommands such as setup, config, doctor, status, models, sessions, projects, reports, ask, agents, connections, billing, validate, rules, or open.

CLI profiles for agents

Use named profiles when multiple agents, environments, or workspaces share the same host. A profile can hold default CloudEval service URL, app URL, project, model, and output preferences.

cloudeval setup \
  --non-interactive \
  --profile codex \
  --project <project-id> \
  --model gpt-5-nano \
  --format json

Then pass the same profile to automation commands:

cloudeval ask "Summarize project risk" \
  --profile codex \
  --format json \
  --non-interactive

Explicit flags still override profile defaults, so scripts can pin a project or model for one run without changing the stored profile.

Agent Profiles

Agent Profiles are CloudEval-owned reviewer roles. They help the same project evidence produce different kinds of answers without every client rebuilding its own prompt logic. Use Agent Profiles when the question is not just “what did CloudEval find?” but “who is reading this, and what decision are they trying to make?”

Architecture reviews topology, dependencies, blast radius, availability, and Well-Architected tradeoffs.
Cost reviews spend drivers, waste signals, savings confidence, and cost validation checks.
Triage reviews likely failure domains, impact, containment, and rollback signals.
Remediation turns findings into ordered fixes with owners, dependencies, rollout cautions, and validation checks.

The public profile ids are architecture, cost, triage, and remediation. Architecture includes the Well-Architected review lens, so there is no separate Well-Architected Agent Profile.

Agent Profiles work across the web app, CLI, and MCP. CloudEval applies the profile on the backend, so clients do not need to maintain separate prompt templates or local fallback logic.

cloudeval agents list --format json
cloudeval agents show cost --format json
cloudeval agents run cost \
  --project <project-id> \
  --format json \
  --non-interactive

Expected output from agents run:

{
  "ok": true,
  "command": "agents run",
  "data": {
    "profile": {
      "id": "cost",
      "display_name": "Cost"
    },
    "prompt": "Review this live Azure sync for cost risk...",
    "response": "Top cost signal: ...",
    "threadId": "8d5d...",
    "project": {
      "id": "project-123",
      "name": "Production"
    },
    "frontendUrl": "https://app.cloudeval.com/app/chat?thread=8d5d..."
  }
}

Agent Profiles are distinct from local CLI config profiles:

An Agent Profile is the reviewer role sent to CloudEval: architecture, cost, triage, or remediation.
A CLI config profile is local machine configuration: CloudEval service URL, app URL, default project, model, output format, and local hooks.

How Agent Profiles behave

CloudEval owns the profile catalog. Chat, CLI, and MCP all use the same public profile ids, labels, starter prompts, capability hints, and server-side answer contracts.

Field	What it controls
`id`	Stable machine id used by Chat, CLI, and MCP.
`display_name`	Single-word public label shown in UI and command output.
`description` and `personality`	Catalog copy and launch surfaces.
`starter_prompts.template`	Default prompt for ARM/Bicep template projects.
`starter_prompts.sync`	Default prompt for live sync projects.
`required_capabilities`	Capabilities the client can show before running the profile.
`default_settings`	Backend-owned run defaults such as agent mode, response depth, reasoning effort, and tool limits.
`output_contract`	The shape of the answer CloudEval asks the model to produce for that role.

How profiles shape a run

Agent Profiles do not create a separate product workflow. They shape the normal CloudEval review path:

The user chooses architecture, cost, triage, or remediation.
CloudEval keeps the selected profile separate from selected resources.
CloudEval applies the profile’s focus, starter prompt, response style, output contract, and tool priorities.
The run uses the same project evidence, grounding rules, and safety checks as normal chat.
The final answer is shaped by the selected role without overriding user intent, accuracy, or citation expectations.

No selected Agent Profile means the normal chat path remains in control. CloudEval does not treat Default as a profile id.

Same question, different lens

Use the same plain-English question and choose a different profile when the decision context changes.

cloudeval agents run architecture "What needs attention first?" --project <project-id>
cloudeval agents run cost "What needs attention first?" --project <project-id>
cloudeval agents run triage "What needs attention first?" --project <project-id>
cloudeval agents run remediation "What needs attention first?" --project <project-id>

Expected answer shape:

Profile	What the answer should emphasize	Typical next action
`architecture`	Topology risk, dependency path, blast radius, availability, or Well-Architected tradeoff.	Validate the design boundary or dependency path before changing architecture.
`cost`	Spend exposure, waste, idle capacity, expensive configuration, savings confidence, and period or unit when available.	Check the top spend driver and validate the highest-confidence savings opportunity.
`triage`	Likely failure domain, affected path, containment, verification, and monitor or rollback signal.	Contain impact, verify the failure domain, then monitor or roll back.
`remediation`	Ordered fix, owner expectation, dependency, rollout caution, and validation check.	Turn the highest-confidence finding into a sequenced fix plan.

For example, a generic question such as “Can we trust this readout?” should not produce four copies of the same response:

architecture should explain whether the topology and dependency evidence is enough to trust the architecture conclusion.
cost should explain whether the spend or savings evidence is strong enough to trust the financial conclusion.
triage should explain whether the evidence is enough to act during an incident or investigation.
remediation should explain whether the evidence is strong enough to assign work, sequence fixes, and validate completion.

Web app report prompts

Report prompt chips in the web app can launch chat with a profile already selected. The visible chat message should stay short, such as:

Can we trust this readout?

Report details, evidence references, tab state, and prompt metadata should travel as chat context, not as a large visible message. That keeps the conversation readable while preserving grounding. When a prompt comes from a report, the selected profile should match the report area:

Report area	Recommended profile	Why
Architecture, dependencies, security posture, or trust questions	`architecture`	The answer needs topology, dependency, blast-radius, and Well-Architected context.
Cost and FinOps	`cost`	The answer needs spend, waste, rightsizing, and savings-confidence context.
Active issue investigation or “what broke?” questions	`triage`	The answer needs failure-domain and containment framing.
Action plan or fix sequencing	`remediation`	The answer needs owners, dependencies, rollout cautions, and validation checks.

For report-triggered prompts, prefer one-shot controls: apply the profile, mode, project, and hidden context to that request, then return the chat input to the user’s previous selection.

Starter prompts by project source and mode

When a user launches a profile from the Developer workspace or runs cloudeval agents run <profile-id> without a prompt, CloudEval picks a starter prompt based on the selected project source and selected mode.

Profile	Template ask starter	Template agent starter	Live sync ask starter	Live sync agent starter
`architecture`	One architecture risk and first validation check.	Topology, dependency evidence, Well-Architected tradeoff, design action, and checkpoint.	One live architecture risk and first validation check.	Topology, dependencies, blast radius, Well-Architected evidence, and validation checkpoints.
`cost`	One template cost risk and first savings validation check.	Cost evidence, expensive SKUs, waste risk, savings confidence, and validation checks.	One live cost risk and first savings validation check.	Current spend signals, idle resources, rightsizing actions, savings confidence, and checkpoint.
`triage`	One likely deployment/runtime failure domain and first check.	Failure domain, blast radius, contain step, verify check, and rollback signal.	One likely active failure domain and first check.	Failure path, affected resources, contain step, verify check, and monitor/rollback signal.
`remediation`	One template fix and first validation check.	Ordered edits, prerequisites, owners, dependencies, rollout cautions, and verification.	One live fix and first validation check.	Ordered actions, owners, dependencies, rollout cautions, and validation checks.

CloudEval can expose multiple starter variants for each source/mode pair. The Developer workspace randomizes the visible launch prompt when opening a profile. The CLI stays deterministic so scripts are repeatable and uses the first matching variant when no explicit prompt is passed. Pass an explicit prompt when you want the profile behavior but not the default starter text:

cloudeval agents run architecture \
  "Review only public ingress, private networking, and dependency blast radius." \
  --project <project-id> \
  --format json

MCP tools

MCP-compatible agents can use the same catalog without shelling out to the CLI. The public tools are:

agent_profiles_list
agent_profiles_get
agent_profiles_run

Use MCP when the agent already has a tool-calling loop and needs CloudEval as a live evidence source. Use the CLI when the workflow is a script, CI job, or terminal session.

Common mistakes

Do not send default as an Agent Profile id. Leave the profile selector empty for the normal chat flow.
Do not create a separate Well-Architected profile in integrations. Use architecture.
Do not confuse --profile codex with agents run cost. The first selects local CLI config. The second selects CloudEval reviewer behavior.
Do not paste long report contracts into visible chat messages. Send the short question as the message and attach report details as context.
Do not assume local hooks are CloudEval-hosted automation. Hooks run on the CLI host only and can be bypassed with --no-hooks.

Local CLI hooks

Local hooks are opt-in commands stored in the active CLI config profile. They run on the local machine only; CloudEval does not store, execute, or report hook runs in v1. Supported hook events:

cli.command.before
cli.command.after
cli.command.error
agent_profile.run.before
agent_profile.run.after
agent_profile.run.error

Use --no-hooks on supported commands to bypass local hook execution for one run. Hook output goes to stderr so JSON and NDJSON stdout remain parseable.

MCP server for agents

Use cloudeval mcp serve when your agent framework already supports MCP and you want CloudEval as a live tool server instead of a shell command wrapper. Check local MCP discovery first:

cloudeval mcp status --format json
cloudeval doctor --mcp --format json

Example client configuration:

{
  "mcpServers": {
    "cloudeval": {
      "command": "cloudeval",
      "args": ["mcp", "serve"]
    }
  }
}

CloudEval can also generate setup guidance for common clients:

cloudeval mcp setup codex --dry-run
cloudeval mcp setup claude --dry-run
cloudeval mcp setup cursor --dry-run
cloudeval mcp setup vscode --dry-run
cloudeval mcp setup generic --dry-run --toolset readonly --format json

mcp setup --toolset accepts all, readonly, projects, reports, or billing. Use mcp serve --toolset graph or --toolset validation when the agent needs those surfaces. Use generic for MCP-compatible clients that expect an mcpServers JSON entry. For Ollama-powered agents, configure the MCP host launched by Ollama with that generated CloudEval stdio entry. Use focused toolsets when an agent only needs part of the CloudEval surface:

cloudeval mcp serve --toolset readonly
cloudeval mcp serve --toolset projects
cloudeval mcp serve --toolset reports
cloudeval mcp serve --toolset billing
cloudeval mcp serve --toolset graph
cloudeval mcp serve --toolset validation

Important rules:

The server uses stdio.
Authenticate with stored cloudeval login credentials, stored cloudeval login --headless credentials, or --machine.
Run login before starting mcp serve; stdin is reserved for MCP protocol messages.
Treat MCP tool results as the same CloudEval data contract you would expect from the CLI: stable envelopes, returned IDs, and explicit errors.
Prefer focused MCP toolsets for assistants that should only inspect projects, reports, billing, or read-only data.
MCP clients that support resources and prompts can discover CloudEval capabilities, project context, billing summaries, latest reports, and review-oriented prompt templates.
Agent Profile MCP tools are agent_profiles_list, agent_profiles_get, and agent_profiles_run; they use the same architecture, cost, triage, and remediation ids as the CLI.

Graph and validation automation

Use graph commands for project intelligence that should not require opening the workspace UI:

cloudeval projects graph <project-id> --format json --non-interactive
cloudeval projects graph sync-runs <project-id> --format json --non-interactive
cloudeval projects graph insights <project-id> --focus overview --format json --non-interactive

Use template validation commands before deployment-oriented automation:

cloudeval validate template \
  --template-file ./azuredeploy.json \
  --parameters-file ./azuredeploy.parameters.json \
  --rule <check-id> \
  --details \
  --wait \
  --progress stderr \
  --wait-timeout 600000 \
  --format json \
  --non-interactive

cloudeval validate tests \
  --template-file ./azuredeploy.json \
  --parameters-file ./azuredeploy.parameters.json \
  --wait \
  --progress stderr \
  --wait-timeout 600000 \
  --format json \
  --non-interactive

cloudeval validate parse \
  --template-file ./azuredeploy.json \
  --parameters-file ./azuredeploy.parameters.json \
  --format json \
  --non-interactive

cloudeval rules search "public network" --format json --non-interactive

--parameters-file is optional for both validation and parsing. Agents should pass it when a parameter file is present and omit it when defaults are enough. Use rules search or rules show to resolve check ids, then pass repeatable --rule values when an automation should run only specific checks. Use --wait for deployment gates that need final validation results instead of only a queued job id, and include --wait-timeout so the gate fails instead of hanging indefinitely. Add --progress stderr when a human should see queued/running/completed status while the command waits. Progress is written to stderr; final JSON remains on stdout for automation. Completed progress includes failing check/test details when the backend returns message, recommendation, severity, and location fields. Worker-local temp file paths are replaced with the submitted template filename so automation logs do not point at inaccessible backend files. Do not block automation solely because a parameters file is missing.

Stable JSON envelope

CloudEval uses a stable JSON envelope for machine-readable success and error responses:

{
  "ok": true,
  "command": "projects create",
  "data": {},
  "frontendUrl": "https://cloudeval.ai/app/projects/..."
}

{
  "ok": false,
  "command": "reports run",
  "error": { "message": "Authentication required" }
}

Some commands also include fields such as:

warnings
filesWritten
traceId

When you use ndjson, arrays are emitted one JSON object per line instead of one wrapped array payload.

Ask mode vs agent mode

CloudEval supports two practical usage patterns:

ASK mode is best for one grounded answer, usually through cloudeval ask.
AGENT mode is best for multi-step workflows that may inspect projects, run reports, open deeplinks, or create CloudEval artifacts when explicitly requested.

Important guardrails:

ASK flows should stay read-first and should not silently create or change CloudEval artifacts.
AGENT workflows can be broader, but they still need explicit intent before taking write actions.
Do not claim CloudEval mutates customer cloud infrastructure unless a separately verified feature explicitly supports that behavior.

Session continuity for agents

Successful ask runs create local, profile-scoped session history. Use it when an agent needs to find or continue recent CloudEval work on the same machine.

cloudeval sessions search "cost review" --profile codex --format json
cloudeval sessions rename <thread-id> "Cost review" --profile codex --format json
cloudeval chat --resume "Cost review" --profile codex
cloudeval ask "Continue the same investigation" --thread <thread-id> --profile codex --format json --non-interactive

Rules for session use:

Session history is local to the machine and scoped by profile.
Use sessions search before assuming a thread ID.
Use sessions rename to make important threads easy to find later.
Use ask --thread only when a one-shot follow-up should stay attached to an existing conversation.

Grounding model

CloudEval answers are expected to be grounded in the data the product actually has access to. That can include:

project metadata
connection metadata
ARM or Bicep-derived ARM template content
resource graph and diagram relationships
saved cost reports
saved architecture or Well-Architected reports
report history and trend data where available
pricing and product metadata
chat thread history
local CLI session history for one-shot ask runs

If the evidence is missing, the right behavior is to say what is missing and suggest the next useful command.

Authentication and permissions

cloudeval login uses a browser-based login flow.
cloudeval login --headless uses a device-code flow for headless sessions.
Browser-based CLI login is restricted to loopback redirect targets on the local machine.
Use cloudeval login --headless for SSH, containers, or remote terminals.
Use stored login state or --machine when you run cloudeval mcp serve.
Always use IDs returned by CloudEval responses. Do not guess project, report, connection, or thread IDs.

Practical limits

Azure is the primary supported provider today.
ARM JSON is the strongest current IaC path.
AWS and GCP should not be treated as full-parity live sync or reporting paths unless current capabilities confirm it.
Diagram freshness depends on the latest successful import or sync.
Cost outputs can be estimates, not final billing truth.
Architecture and security findings are evaluations, not compliance attestations.
Some browser workflows are still easier or only available in the web app.

Verification before production use

Before shipping a new integration:

Run cloudeval capabilities --format json.
Run cloudeval doctor --format json for the profile or environment you will use.
Run cloudeval doctor --mcp --format json if an MCP client is part of the workflow.
Test the exact commands you plan to automate with --format json --non-interactive.
Confirm the target project, report, connection, or thread IDs come from CloudEval output.
Check that your workflow handles auth-required, service-unavailable, and not-found failures cleanly.

Next step

Use llms.txt and llms-full.txt for the public context files, or CLI command reference for the exact command groups and flags.

CLI quickstart

Automation and agents

CLI reference

Start with safe defaults

Stdout and stderr contract

CLI profiles for agents

Agent Profiles

How Agent Profiles behave

How profiles shape a run

Same question, different lens

Web app report prompts

Starter prompts by project source and mode

MCP tools

Common mistakes

Local CLI hooks

MCP server for agents

Graph and validation automation

Stable JSON envelope

Ask mode vs agent mode

Session continuity for agents

Grounding model

Authentication and permissions

Practical limits

Verification before production use

Next step

​Start with safe defaults

​Stdout and stderr contract

​CLI profiles for agents

​Agent Profiles

​How Agent Profiles behave

​How profiles shape a run

​Same question, different lens

​Web app report prompts

​Starter prompts by project source and mode

​MCP tools

​Common mistakes

​Local CLI hooks

​MCP server for agents

​Graph and validation automation

​Stable JSON envelope

​Ask mode vs agent mode

​Session continuity for agents

​Grounding model

​Authentication and permissions

​Practical limits

​Verification before production use

​Next step

Start with safe defaults

Stdout and stderr contract

CLI profiles for agents

Agent Profiles

How Agent Profiles behave

How profiles shape a run

Same question, different lens

Web app report prompts

Starter prompts by project source and mode

MCP tools

Common mistakes

Local CLI hooks

MCP server for agents

Graph and validation automation

Stable JSON envelope

Ask mode vs agent mode

Session continuity for agents

Grounding model

Authentication and permissions

Practical limits

Verification before production use

Next step