Agent and automation rules - CloudEval AI Docs

This page is for developers building scripts, internal tools, or agents on top of CloudEval.

Start with safe defaults

Use these defaults unless you have a good reason not to:

cloudeval capabilities --format json
cloudeval doctor --format json
cloudeval doctor --mcp --format json before MCP client setup
--format json
--non-interactive
--profile <name>
--print-url --no-open
stored cloudeval login --headless auth, or --machine when service-principal credentials are configured
--output <file> when the result must be persisted

These defaults reduce ambiguity and make the CLI easier to compose with other systems.

Stdout and stderr contract

For machine-readable commands:

stdout is the data channel
stderr is for prompts, warnings, auth flow text, and browser-open messages

Do not parse terminal UI output. If you need automation, use explicit subcommands such as setup, config, doctor, status, models, sessions, projects, reports, ask, agents, connections, billing, validate, rules, or open.

CLI profiles for agents

Use named profiles when multiple agents, environments, or workspaces share the same host. A profile can hold default CloudEval service URL, app URL, project, model, and output preferences.

cloudeval setup \
  --non-interactive \
  --profile codex \
  --project <project-id> \
  --model gpt-5-nano \
  --format json

Then pass the same profile to automation commands:

cloudeval ask "Summarize project risk" \
  --profile codex \
  --format json \
  --non-interactive

Explicit flags still override profile defaults, so scripts can pin a project or model for one run without changing the stored profile.

Agent Profiles

Agent Profiles are CloudEval-owned reviewer roles that work consistently in the web app, CLI, and MCP. Use single-word public names in UX and automation output: Architecture, Cost, Triage, and Remediation. Architecture includes the Well-Architected review lens, so there is no separate Well-Architected Agent Profile. Each profile also exposes starter prompt variants by project source and mode: template or live sync, ask or agent.

cloudeval agents list --format json
cloudeval agents show cost --format json
cloudeval agents run cost \
  --project <project-id> \
  --format json \
  --non-interactive

Agent Profiles are distinct from local CLI config profiles:

An Agent Profile is the reviewer role sent to CloudEval: architecture, cost, triage, or remediation.
A CLI config profile is local machine configuration: CloudEval service URL, app URL, default project, model, output format, and local hooks.

How Agent Profiles behave

CloudEval owns the profile catalog. Chat, CLI, and MCP all use the same public profile ids, labels, starter prompts, and capability hints.

Field	What it controls
`id`	Stable machine id used by Chat, CLI, and MCP.
`display_name`	Single-word public label shown in UI and command output.
`description` and `personality`	Catalog copy and launch surfaces.
`starter_prompts.template`	Default prompt for ARM/Bicep template projects.
`starter_prompts.sync`	Default prompt for live sync projects.
`required_capabilities`	Capabilities the client can show before running the profile.

How profiles shape a run

Agent Profiles do not create a separate product workflow. They shape the normal CloudEval review path:

The user chooses architecture, cost, triage, or remediation.
CloudEval keeps the selected profile separate from selected resources.
CloudEval applies the profile’s focus, starter prompt, response style, and tool priorities.
The run uses the same project evidence and safety checks as normal chat.
The final answer is biased toward the selected role without overriding user intent, accuracy, or citation expectations.

No selected Agent Profile means the normal chat path remains in control. CloudEval does not treat Default as a profile id.

Starter prompts by project source and mode

When a user launches a profile from the Developer workspace or runs cloudeval agents run <profile-id> without a prompt, CloudEval picks a starter prompt based on the selected project source and selected mode.

Profile	Template ask starter	Template agent starter	Live sync ask starter	Live sync agent starter
`architecture`	One architecture risk and first validation check.	Topology, dependency evidence, Well-Architected tradeoff, design action, and checkpoint.	One live architecture risk and first validation check.	Topology, dependencies, blast radius, Well-Architected evidence, and validation checkpoints.
`cost`	One template cost risk and first savings validation check.	Cost evidence, expensive SKUs, waste risk, savings confidence, and validation checks.	One live cost risk and first savings validation check.	Current spend signals, idle resources, rightsizing actions, savings confidence, and checkpoint.
`triage`	One likely deployment/runtime failure domain and first check.	Failure domain, blast radius, contain step, verify check, and rollback signal.	One likely active failure domain and first check.	Failure path, affected resources, contain step, verify check, and monitor/rollback signal.
`remediation`	One template fix and first validation check.	Ordered edits, prerequisites, owners, dependencies, rollout cautions, and verification.	One live fix and first validation check.	Ordered actions, owners, dependencies, rollout cautions, and validation checks.

CloudEval can expose multiple starter variants for each source/mode pair. The Developer workspace randomizes the visible launch prompt when opening a profile. The CLI stays deterministic so scripts are repeatable and uses the first matching variant when no explicit prompt is passed. Pass an explicit prompt when you want the profile behavior but not the default starter text:

cloudeval agents run architecture \
  "Review only public ingress, private networking, and dependency blast radius." \
  --project <project-id> \
  --format json

Common mistakes

Do not send default as an Agent Profile id. Leave the profile selector empty for the normal chat flow.
Do not create a separate Well-Architected profile in integrations. Use architecture.
Do not confuse --profile codex with agents run cost. The first selects local CLI config. The second selects CloudEval reviewer behavior.
Do not assume local hooks are CloudEval-hosted automation. Hooks run on the CLI host only and can be bypassed with --no-hooks.

Local CLI hooks

Local hooks are opt-in commands stored in the active CLI config profile. They run on the local machine only; CloudEval does not store, execute, or report hook runs in v1. Supported hook events:

cli.command.before
cli.command.after
cli.command.error
agent_profile.run.before
agent_profile.run.after
agent_profile.run.error

Use --no-hooks on supported commands to bypass local hook execution for one run. Hook output goes to stderr so JSON and NDJSON stdout remain parseable.

MCP server for agents

Use cloudeval mcp serve when your agent framework already supports MCP and you want CloudEval as a live tool server instead of a shell command wrapper. Check local MCP discovery first:

cloudeval mcp status --format json
cloudeval doctor --mcp --format json

Example client configuration:

{
  "mcpServers": {
    "cloudeval": {
      "command": "cloudeval",
      "args": ["mcp", "serve"]
    }
  }
}

CloudEval can also generate setup guidance for common clients:

cloudeval mcp setup codex --dry-run
cloudeval mcp setup claude --dry-run
cloudeval mcp setup cursor --dry-run
cloudeval mcp setup vscode --dry-run
cloudeval mcp setup generic --dry-run --toolset readonly --format json

mcp setup --toolset accepts all, readonly, projects, reports, or billing. Use mcp serve --toolset graph or --toolset validation when the agent needs those surfaces. Use generic for MCP-compatible clients that expect an mcpServers JSON entry. For Ollama-powered agents, configure the MCP host launched by Ollama with that generated CloudEval stdio entry. Use focused toolsets when an agent only needs part of the CloudEval surface:

cloudeval mcp serve --toolset readonly
cloudeval mcp serve --toolset projects
cloudeval mcp serve --toolset reports
cloudeval mcp serve --toolset billing
cloudeval mcp serve --toolset graph
cloudeval mcp serve --toolset validation

Important rules:

The server uses stdio.
Authenticate with stored cloudeval login credentials, stored cloudeval login --headless credentials, or --machine.
Run login before starting mcp serve; stdin is reserved for MCP protocol messages.
Treat MCP tool results as the same CloudEval data contract you would expect from the CLI: stable envelopes, returned IDs, and explicit errors.
Prefer focused MCP toolsets for assistants that should only inspect projects, reports, billing, or read-only data.
MCP clients that support resources and prompts can discover CloudEval capabilities, project context, billing summaries, latest reports, and review-oriented prompt templates.
Agent Profile MCP tools are agent_profiles_list, agent_profiles_get, and agent_profiles_run; they use the same architecture, cost, triage, and remediation ids as the CLI.

Graph and validation automation

Use graph commands for project intelligence that should not require opening the workspace UI:

cloudeval projects graph <project-id> --format json --non-interactive
cloudeval projects graph sync-runs <project-id> --format json --non-interactive
cloudeval projects graph insights <project-id> --focus overview --format json --non-interactive

Use template validation commands before deployment-oriented automation:

cloudeval validate template \
  --template-file ./azuredeploy.json \
  --parameters-file ./azuredeploy.parameters.json \
  --rule <check-id> \
  --details \
  --wait \
  --progress stderr \
  --wait-timeout 600000 \
  --format json \
  --non-interactive

cloudeval validate tests \
  --template-file ./azuredeploy.json \
  --parameters-file ./azuredeploy.parameters.json \
  --wait \
  --progress stderr \
  --wait-timeout 600000 \
  --format json \
  --non-interactive

cloudeval validate parse \
  --template-file ./azuredeploy.json \
  --parameters-file ./azuredeploy.parameters.json \
  --format json \
  --non-interactive

cloudeval rules search "public network" --format json --non-interactive

--parameters-file is optional for both validation and parsing. Agents should pass it when a parameter file is present and omit it when defaults are enough. Use rules search or rules show to resolve check ids, then pass repeatable --rule values when an automation should run only specific checks. Use --wait for deployment gates that need final validation results instead of only a queued job id, and include --wait-timeout so the gate fails instead of hanging indefinitely. Add --progress stderr when a human should see queued/running/completed status while the command waits. Progress is written to stderr; final JSON remains on stdout for automation. Completed progress includes failing check/test details when the backend returns message, recommendation, severity, and location fields. Worker-local temp file paths are replaced with the submitted template filename so automation logs do not point at inaccessible backend files. Do not block automation solely because a parameters file is missing.

Stable JSON envelope

CloudEval uses a stable JSON envelope for machine-readable success and error responses:

{
  "ok": true,
  "command": "projects create",
  "data": {},
  "frontendUrl": "https://cloudeval.ai/app/projects/..."
}

{
  "ok": false,
  "command": "reports run",
  "error": { "message": "Authentication required" }
}

Some commands also include fields such as:

warnings
filesWritten
traceId

When you use ndjson, arrays are emitted one JSON object per line instead of one wrapped array payload.

Ask mode vs agent mode

CloudEval supports two practical usage patterns:

ASK mode is best for one grounded answer, usually through cloudeval ask.
AGENT mode is best for multi-step workflows that may inspect projects, run reports, open deeplinks, or create CloudEval artifacts when explicitly requested.

Important guardrails:

ASK flows should stay read-first and should not silently create or change CloudEval artifacts.
AGENT workflows can be broader, but they still need explicit intent before taking write actions.
Do not claim CloudEval mutates customer cloud infrastructure unless a separately verified feature explicitly supports that behavior.

Session continuity for agents

Successful ask runs create local, profile-scoped session history. Use it when an agent needs to find or continue recent CloudEval work on the same machine.

cloudeval sessions search "cost review" --profile codex --format json
cloudeval sessions rename <thread-id> "Cost review" --profile codex --format json
cloudeval chat --resume "Cost review" --profile codex
cloudeval ask "Continue the same investigation" --thread <thread-id> --profile codex --format json --non-interactive

Rules for session use:

Session history is local to the machine and scoped by profile.
Use sessions search before assuming a thread ID.
Use sessions rename to make important threads easy to find later.
Use ask --thread only when a one-shot follow-up should stay attached to an existing conversation.

Grounding model

CloudEval answers are expected to be grounded in the data the product actually has access to. That can include:

project metadata
connection metadata
ARM or Bicep-derived ARM template content
resource graph and diagram relationships
saved cost reports
saved architecture or Well-Architected reports
report history and trend data where available
pricing and product metadata
chat thread history
local CLI session history for one-shot ask runs

If the evidence is missing, the right behavior is to say what is missing and suggest the next useful command.

Authentication and permissions

cloudeval login uses a browser-based login flow.
cloudeval login --headless uses a device-code flow for headless sessions.
Browser-based CLI login is restricted to loopback redirect targets on the local machine.
Use cloudeval login --headless for SSH, containers, or remote terminals.
Use stored login state or --machine when you run cloudeval mcp serve.
Always use IDs returned by CloudEval responses. Do not guess project, report, connection, or thread IDs.

Practical limits

Azure is the primary supported provider today.
ARM JSON is the strongest current IaC path.
AWS and GCP should not be treated as full-parity live sync or reporting paths unless current capabilities confirm it.
Diagram freshness depends on the latest successful import or sync.
Cost outputs can be estimates, not final billing truth.
Architecture and security findings are evaluations, not compliance attestations.
Some browser workflows are still easier or only available in the web app.

Verification before production use

Before shipping a new integration:

Run cloudeval capabilities --format json.
Run cloudeval doctor --format json for the profile or environment you will use.
Run cloudeval doctor --mcp --format json if an MCP client is part of the workflow.
Test the exact commands you plan to automate with --format json --non-interactive.
Confirm the target project, report, connection, or thread IDs come from CloudEval output.
Check that your workflow handles auth-required, service-unavailable, and not-found failures cleanly.

Next step

Use llms.txt and llms-full.txt for the public context files, or CLI command reference for the exact command groups and flags.

​Start with safe defaults

​Stdout and stderr contract

​CLI profiles for agents

​Agent Profiles

​How Agent Profiles behave

​How profiles shape a run

​Starter prompts by project source and mode

​Common mistakes

​Local CLI hooks

​MCP server for agents

​Graph and validation automation

​Stable JSON envelope

​Ask mode vs agent mode

​Session continuity for agents

​Grounding model

​Authentication and permissions

​Practical limits

​Verification before production use

​Next step