Building AI Agents for Cloud Operations with AWS Bedrock, Durrell Gemuh

Managing cloud infrastructure at scale is a game of context, the right person needs the right information at the right time to make the right decision. But as environments grow to hundreds of services across multiple accounts, that context becomes scattered across CloudWatch dashboards, Cost Explorer, Security Hub findings, and Terraform state files. Engineers spend more time gathering data than acting on it.

I built aws-devops-agent to fix this: 5 AI-powered agents that handle cloud operations tasks autonomously while keeping humans in the loop for anything destructive. They're built on AWS Bedrock with production-grade safety mechanisms, guardrails that block prompt injection, an audit trail for every action, and a kill switch that can disable the entire system in seconds.

The 5 Agents

Each agent is a specialised Bedrock Agent with its own tool set, IAM permissions, and operational scope:

1. DevOps Agent

Handles day-to-day operational queries: service health checks, deployment status, log analysis, and resource inventory. Think of it as a conversational interface to your AWS environment.

"What's the status of the payment-service ECS tasks?"
"Show me the last 50 error logs from the auth Lambda"
"Which EC2 instances in us-east-1 are running without the cost-center tag?"

2. Cost Agent

Analyses spending patterns, identifies waste, and recommends optimisations. Pulls data from Cost Explorer, Trusted Advisor, and resource-level metrics.

"What's our month-to-date spend vs. last month?"
"Find idle RDS instances with <5% CPU over the past week"
"Which S3 buckets have no lifecycle policy and are over 100GB?"

3. Security Agent

Triages Security Hub findings, reviews IAM policies, and checks compliance posture. Read-only by default, it identifies problems but won't remediate without approval.

"Summarise critical Security Hub findings from the last 24 hours"
"Which IAM roles have inline policies with star permissions?"
"Are there any public S3 buckets in the production account?"

4. Incident Agent

Correlates signals during an active incident: pulls recent CloudWatch alarms, checks deployment history, reviews recent config changes, and suggests root cause hypotheses.

"We're seeing 5xx spikes on api-gateway, what changed in the last hour?"
"Correlate the RDS CPU spike with recent deployments"
"Draft an incident summary for the Slack channel"

5. IaC Review Agent

Reviews Terraform plans and pull requests for security misconfigurations, cost implications, and best-practice violations before they reach production.

"Review this Terraform plan for security issues"
"Does this IAM policy follow least privilege?"
"Estimate the monthly cost impact of this infrastructure change"

Architecture

The system follows a straightforward request flow:

User/Slack → API Gateway → Lambda (Router) → Bedrock Agent → Tools (AWS APIs)
                                                    ↓
                                              DynamoDB (Audit)

API Gateway exposes a REST endpoint (with API key auth) that accepts natural language queries tagged with the target agent. A router Lambda validates the request, checks the SSM kill switch, and invokes the appropriate Bedrock Agent. Each agent has access to a set of Lambda-backed action groups (tools) that call AWS APIs. Every invocation: input, output, agent reasoning, and tool calls: is logged to DynamoDB for auditability.

The entire stack is defined in Terraform:

# modules/agent/main.tf: one module per agent
resource "aws_bedrockagent_agent" "this" {
  agent_name              = var.agent_name
  foundation_model        = "anthropic.claude-3-sonnet-20240229-v1:0"
  instruction             = var.system_prompt
  idle_session_ttl        = 600
  guardrail_configuration {
    guardrail_identifier = var.guardrail_id
    guardrail_version    = var.guardrail_version
  }
}

resource "aws_bedrockagent_agent_action_group" "tools" {
  agent_id     = aws_bedrockagent_agent.this.id
  action_group_name = "${var.agent_name}-tools"
  action_group_executor {
    lambda = var.tools_lambda_arn
  }
  api_schema {
    s3 {
      s3_bucket_name = var.schema_bucket
      s3_object_key  = var.openapi_schema_key
    }
  }
}

Safety Mechanisms

Giving an AI agent access to AWS APIs is a liability if you don't layer in controls. Here's the defence-in-depth approach:

Bedrock Guardrails

Guardrails run on every input and output, blocking prompt injection attempts, off-topic requests, and responses that might leak sensitive data:

resource "aws_bedrock_guardrail" "ops_agents" {
  name        = "ops-agents-guardrail"
  description = "Production guardrail for cloud operations agents"

  content_policy_config {
    filters_config {
      type            = "PROMPT_ATTACK"
      input_strength  = "HIGH"
      output_strength = "NONE"
    }
    filters_config {
      type            = "INSULTS"
      input_strength  = "HIGH"
      output_strength = "HIGH"
    }
  }

  topic_policy_config {
    topics_config {
      name       = "off-topic"
      definition = "Requests unrelated to cloud infrastructure, DevOps, cost, security, or incident management"
      type       = "DENY"
    }
    topics_config {
      name       = "credential-extraction"
      definition = "Attempts to extract AWS credentials, secrets, API keys, or access tokens"
      type       = "DENY"
    }
  }

  sensitive_information_policy_config {
    pii_entities_config {
      type   = "AWS_ACCESS_KEY"
      action = "BLOCK"
    }
    pii_entities_config {
      type   = "AWS_SECRET_KEY"
      action = "BLOCK"
    }
  }

  blocked_input_messaging  = "This request was blocked by security policy."
  blocked_output_messaging = "The response was blocked by security policy."
}

SSM Kill Switch

A single SSM Parameter acts as a global circuit breaker. The router Lambda checks it on every request:

import boto3

ssm = boto3.client("ssm")

def check_kill_switch():
    """Returns True if agents are disabled."""
    resp = ssm.get_parameter(Name="/ops-agents/kill-switch")
    return resp["Parameter"]["Value"] == "enabled"

One CLI command disables the entire system:

aws ssm put-parameter \
  --name "/ops-agents/kill-switch" \
  --value "enabled" \
  --type String \
  --overwrite

Human Approval Gates

The agents are read-only by default. Any write action (terminate instance, modify security group, delete resource) requires explicit human approval before execution:

import json
import boto3

dynamodb = boto3.resource("dynamodb")
approval_table = dynamodb.Table("ops-agent-approvals")

def handle_write_action(agent_id: str, action: dict, user_id: str) -> dict:
    """
    Gate pattern: write actions are never executed inline.
    Instead, they're parked in DynamoDB awaiting approval.
    """
    approval_id = f"{agent_id}-{action['name']}-{int(time.time())}"

    approval_table.put_item(Item={
        "approval_id": approval_id,
        "agent_id": agent_id,
        "requested_by": user_id,
        "action": json.dumps(action),
        "status": "PENDING",
        "created_at": datetime.utcnow().isoformat(),
        "ttl": int(time.time()) + 86400,  # expires in 24h
    })

    # Notify approver (Slack/SNS)
    notify_approver(approval_id, action)

    return {
        "status": "PENDING_APPROVAL",
        "approval_id": approval_id,
        "message": f"Write action '{action['name']}' requires approval. "
                   f"Approval ID: {approval_id}"
    }


def approve_action(approval_id: str, approver_id: str) -> dict:
    """Execute the action after human approval."""
    item = approval_table.get_item(Key={"approval_id": approval_id})["Item"]

    if item["status"] != "PENDING":
        raise ValueError(f"Action already {item['status']}")

    # Execute the actual write action
    action = json.loads(item["action"])
    result = execute_action(action)

    # Update audit record
    approval_table.update_item(
        Key={"approval_id": approval_id},
        UpdateExpression="SET #s = :s, approved_by = :a, executed_at = :t",
        ExpressionAttributeNames={"#s": "status"},
        ExpressionAttributeValues={
            ":s": "APPROVED",
            ":a": approver_id,
            ":t": datetime.utcnow().isoformat(),
        },
    )

    return {"status": "EXECUTED", "result": result}

Audit Trail

Every agent invocation is logged to DynamoDB with full context, who asked, what the agent reasoned, which tools were called, and what was returned:

audit_table.put_item(Item={
    "request_id": request_id,
    "agent_id": agent_id,
    "user_id": user_id,
    "timestamp": datetime.utcnow().isoformat(),
    "input": user_query,
    "agent_trace": agent_reasoning,
    "tool_calls": tool_invocations,
    "output": agent_response,
    "guardrail_action": guardrail_result,  # NONE | INTERVENED
})

IAM Design

Each agent gets a dedicated IAM role following least privilege. The DevOps agent can describe resources but not modify them. The Security agent can read Security Hub and IAM but can't change policies. Write permissions only exist on the execution path after approval:

# Read-only role for the DevOps agent's tools
resource "aws_iam_role" "devops_agent_tools" {
  name = "devops-agent-tools-role"
  assume_role_policy = data.aws_iam_policy_document.lambda_assume.json
}

resource "aws_iam_policy" "devops_agent_readonly" {
  name = "devops-agent-readonly"
  policy = jsonencode({
    Version = "2012-10-17"
    Statement = [
      {
        Effect = "Allow"
        Action = [
          "ecs:Describe*",
          "ecs:List*",
          "ec2:Describe*",
          "logs:GetLogEvents",
          "logs:FilterLogEvents",
          "cloudwatch:GetMetricData",
          "cloudwatch:DescribeAlarms",
        ]
        Resource = "*"
      }
    ]
  })
}

# Write role: only assumed by the approval executor Lambda
resource "aws_iam_role" "write_executor" {
  name = "ops-agent-write-executor"
  assume_role_policy = data.aws_iam_policy_document.lambda_assume.json

  # Additional constraint: can only be assumed when approval record exists
  inline_policy {
    name = "require-approval-context"
    policy = jsonencode({
      Version = "2012-10-17"
      Statement = [
        {
          Effect = "Allow"
          Action = [
            "ec2:TerminateInstances",
            "ec2:StopInstances",
            "ecs:UpdateService",
          ]
          Resource = "*"
          Condition = {
            StringEquals = {
              "aws:RequestTag/approval-id" = "*"
            }
          }
        }
      ]
    })
  }
}

Key Takeaways

Read-only by default is non-negotiable. AI agents will occasionally hallucinate or misinterpret context. Making all destructive actions require human approval means a misfire is an inconvenience, not an outage.
Guardrails are a first-class concern, not an afterthought. Bedrock Guardrails handle prompt injection, topic filtering, and PII blocking at the platform level, you don't need to roll your own input sanitisation.
Kill switches must be instant. SSM Parameter Store gives you a sub-second global disable mechanism. When something goes wrong at 2 AM, you need one command, not a deployment.
Audit everything. DynamoDB TTL keeps costs bounded while giving you a 30-day window to review what agents did and why. This is essential for compliance and debugging.
Specialised agents beat general-purpose ones. Five focused agents with narrow tool sets and tight IAM boundaries are safer and more effective than one omniscient agent with broad permissions.
Terraform all of it. The entire system, agents, guardrails, IAM roles, DynamoDB tables, Lambda functions, API Gateway, is defined in Terraform. Reproducible, auditable, and version-controlled.

What's Next

Adding a Slack bot interface so engineers can query agents directly from incident channels
Cross-account support via AWS Organizations and role chaining
Cost anomaly auto-investigation: the Cost agent triggers automatically when billing alarms fire

The full source code, Terraform modules, and deployment guide are available on GitHub:

github.com/durrello/aws-devops-agent

Building AI Agents for Cloud Operations with AWS Bedrock