AWS Well-Architected Review Automation: Building Your Own Assessment Tools

By /

AWS Well-Architected Review Automation: Building Your Own Assessment Tools

In the rapidly evolving landscape of cloud computing, maintaining the health, security, and cost-efficiency of your AWS environment is paramount. The AWS Well-Architected Framework (WAF) provides a robust set of best practices, but manually conducting reviews across numerous accounts and workloads quickly becomes a bottleneck. It’s a periodic, labor-intensive process that can miss critical issues between assessments. The true power lies in transforming this into a continuous, proactive, and scalable operation. This is where AWS Well-Architected Review Automation, specifically by building your own assessment tools, becomes indispensable. This approach empowers organizations to embed best practices directly into their cloud operations, fostering a culture of continuous improvement and compliance at scale.

Key Concepts: The Foundation of Automated WAF Checks

At its core, automating AWS Well-Architected Reviews involves translating the framework’s conceptual guidance into measurable, actionable checks within your AWS environment.

Understanding the AWS Well-Architected Framework

The WAF organizes best practices into six pillars:
1. Operational Excellence: Focuses on running and monitoring systems and continually improving processes.
2. Security: Dedicated to protecting information, systems, and assets.
3. Reliability: Ensures systems recover from failures and dynamically meet demand.
4. Performance Efficiency: Emphasizes using computing resources efficiently.
5. Cost Optimization: Aims to avoid unneeded costs and optimize spending.
6. Sustainability: Minimizes environmental impacts of cloud operations.

While the native AWS WAF Tool in the console helps define workloads and generate reports, it’s primarily a manual, point-in-time assessment. It lacks the inherent design for continuous, automated monitoring at scale, especially across vast multi-account enterprise landscapes.

The Imperative for Custom Automation

Building custom assessment tools addresses the limitations of manual reviews and unlocks significant benefits:
* Scalability: Manually reviewing hundreds or thousands of AWS accounts and workloads is simply impractical. Automation enables checks at scale.
* Consistency: Eliminates human error and ensures the same, standardized evaluation criteria are applied universally.
* Frequency & Speed: Shifts from periodic (e.g., quarterly) reviews to continuous or event-driven assessments, identifying issues proactively.
* Proactive Issue Identification (“Shift Left”): Integrate checks directly into CI/CD pipelines to catch non-compliant deployments before they reach production, saving significant remediation effort.
* Policy Enforcement & Guardrails: Automate the enforcement of organizational-specific best practices and regulatory compliance across all resources.
* Cost Savings: Continuously identify over-provisioning, idle resources, and costly misconfigurations.
* Improved Security Posture: Proactive and continuous monitoring for security misconfigurations, open ports, unencrypted data, and other vulnerabilities.
* Demonstrable Compliance: Generate automated, auditable reports for internal governance and external auditors, reducing audit fatigue.

Architecture of Custom Assessment Tools

Building your own WAF assessment tools involves orchestrating various AWS services to act as data sources, processing engines, and reporting mechanisms.

Data Sources (Inputs for Assessment):
  • AWS Config: Critical for resource inventory and compliance. AWS Config Rules (managed and custom Lambda-backed) are your primary mechanism to check resource configurations against WAF best practices.
  • AWS Trusted Advisor: Provides high-level recommendations on cost optimization, security, fault tolerance, performance, and service limits. Its findings can be integrated via API.
  • AWS Security Hub: Aggregates security findings from GuardDuty, Inspector, Macie, Config, and partner solutions. Custom WAF security findings can be pushed here for a unified view.
  • AWS GuardDuty: Threat detection service whose findings directly contribute to the Security pillar.
  • AWS CloudTrail: Logs API activity, crucial for detecting unauthorized changes or ensuring adherence to operational procedures.
  • Amazon CloudWatch Logs & Metrics: Provides operational insights, performance monitoring data, and logging best practices.
  • AWS Organizations: Essential for multi-account governance and centralizing data collection from member accounts.
  • Infrastructure-as-Code (IaC) Tools: Integrate static analysis tools like cfn-lint, Checkov, or Terrascan into CI/CD pipelines to scan CloudFormation or Terraform templates for WAF anti-patterns before deployment.
Processing & Logic (The Assessment Engine):
  • AWS Lambda: Serverless functions are ideal for custom logic, making API calls, transforming data, and triggering actions based on assessment results. They are the workhorse for custom Config Rules.
  • AWS Step Functions: For orchestrating complex, multi-step assessment workflows, such as sequentially checking various resources and aggregating results across pillars.
  • AWS EventBridge (formerly CloudWatch Events): An event-driven architecture that triggers Lambda functions based on Config Rule non-compliance, CloudTrail events, or on a predefined schedule.
  • Custom Scripts (e.g., Python with Boto3): For bespoke checks or integrations not easily handled by Config Rules or Lambda alone.
  • Amazon Athena: For querying large datasets of logs (CloudTrail, VPC Flow Logs) to identify patterns or anomalies related to WAF best practices.
Reporting, Visualization & Remediation (Outputs & Actions):
  • Amazon QuickSight / Grafana: Powerful dashboarding tools for visualizing compliance posture, trends, and specific WAF pillar scores over time.
  • AWS S3: For securely storing raw assessment data, generated reports, and audit trails.
  • AWS SNS / SQS / EventBridge: For sending notifications (email, chat, SMS) or queuing messages to other systems upon detection of non-compliance.
  • AWS Security Hub Custom Insights: Create tailored insights based on aggregated WAF findings for a unified security and compliance dashboard.
  • AWS Systems Manager Automation: To trigger automated remediation runbooks for common issues (e.g., encrypting a non-compliant S3 bucket, rotating keys).
  • Integration with IT Service Management (ITSM): Connect to tools like Jira or ServiceNow to create tickets based on high-risk findings for human intervention.

Implementation Guide: Building a Foundational WAF Check

Let’s walk through implementing a basic, yet powerful, custom WAF check using AWS Config and AWS Lambda. Our goal is to ensure all S3 buckets have server-side encryption enabled – a critical Security pillar best practice.

Step 1: Define Your WAF Rule

WAF Pillar: Security
WAF Best Practice: Data Protection – “Encrypt data at rest.”
Technical Rule: All newly created and existing S3 buckets must have default server-side encryption enabled (SSE-S3 or SSE-KMS).

Step 2: Develop a Lambda Function for Evaluation

This Python Lambda function will be invoked by AWS Config whenever an S3 bucket is created or updated, or periodically. It checks the bucket’s encryption status.

import json
import boto3

def evaluate_compliance(configuration_item):
    """
    Evaluates the compliance of an S3 bucket against the encryption requirement.
    """
    bucket_name = configuration_item["resourceName"]
    s3_client = boto3.client('s3')

    try:
        # Check for default encryption configuration
        encryption_config = s3_client.get_bucket_encryption(Bucket=bucket_name)
        rules = encryption_config['ServerSideEncryptionConfiguration']['Rules']

        # Assume compliant if any rule is present
        # For more strict checks, you'd verify specific encryption types (SSE-S3, SSE-KMS)
        if rules:
            return {
                "compliance_type": "COMPLIANT",
                "annotation": f"S3 bucket '{bucket_name}' has default server-side encryption enabled."
            }
        else:
            return {
                "compliance_type": "NON_COMPLIANT",
                "annotation": f"S3 bucket '{bucket_name}' does not have default server-side encryption enabled."
            }
    except s3_client.exceptions.ClientError as e:
        if e.response['Error']['Code'] == 'ServerSideEncryptionConfigurationNotFoundError':
            return {
                "compliance_type": "NON_COMPLIANT",
                "annotation": f"S3 bucket '{bucket_name}' does not have default server-side encryption enabled."
            }
        else:
            # Handle other potential errors gracefully
            print(f"Error checking encryption for bucket {bucket_name}: {e}")
            return {
                "compliance_type": "NOT_APPLICABLE", # Or consider FAILED
                "annotation": f"Could not determine encryption status for '{bucket_name}' due to error: {e}"
            }

def lambda_handler(event, context):
    invoking_event = json.loads(event['invokingEvent'])
    configuration_item = invoking_event['configurationItem']

    # We only care about S3 buckets
    if configuration_item['resourceType'] != "AWS::S3::Bucket":
        return {
            "compliance_type": "NOT_APPLICABLE",
            "annotation": "This rule applies only to S3 buckets."
        }

    result = evaluate_compliance(configuration_item)

    config = boto3.client('config')
    config.put_evaluations(
        Evaluations=[
            {
                'ComplianceResourceType': configuration_item['resourceType'],
                'ComplianceResourceId': configuration_item['resourceId'],
                'ComplianceType': result['compliance_type'],
                'Annotation': result['annotation'],
                'OrderingTimestamp': configuration_item['configurationItemCaptureTime']
            },
        ],
        ResultToken=event['resultToken']
    )
    return result

Step 3: Create a Custom AWS Config Rule

This CloudFormation template defines an AWS Config custom rule that triggers our Lambda function.

# s3-encryption-config-rule.yaml
AWSTemplateFormatVersion: '2010-09-09'
Description: AWS Config Custom Rule for S3 Bucket Default Encryption

Parameters:
  LambdaFunctionName:
    Type: String
    Description: Name of the Lambda function to invoke for evaluation.
    Default: S3EncryptionCheckLambda

Resources:
  # Lambda Function Role
  S3EncryptionCheckLambdaRole:
    Type: AWS::IAM::Role
    Properties:
      AssumeRolePolicyDocument:
        Version: '2012-10-17'
        Statement:
          - Effect: Allow
            Principal:
              Service: lambda.amazonaws.com
            Action: sts:AssumeRole
      ManagedPolicyArns:
        - arn:aws:iam::aws:policy/service-role/AWSLambdaBasicExecutionRole
      Policies:
        - PolicyName: S3ConfigEvaluationPolicy
          PolicyDocument:
            Version: '2012-10-17'
            Statement:
              - Effect: Allow
                Action:
                  - s3:GetBucketEncryption
                Resource: "*" # Restrict to specific buckets if necessary, but * for all checks
              - Effect: Allow
                Action:
                  - config:PutEvaluations
                Resource: "*" # Config needs to put evaluations

  # Lambda Function
  S3EncryptionCheckLambda:
    Type: AWS::Lambda::Function
    Properties:
      FunctionName: !Ref LambdaFunctionName
      Handler: index.lambda_handler # Assuming your Python file is index.py
      Role: !GetAtt S3EncryptionCheckLambdaRole.Arn
      Runtime: python3.9
      Timeout: 30
      Code:
        ZipFile: |
          # The Python code from Step 2 goes here.
          # In a real deployment, you'd package this in an S3 bucket or use CodeUri
          # For simplicity in this example, we're using ZipFile.
          # A better approach for CI/CD would be:
          # Code:
          #   S3Bucket: your-code-bucket
          #   S3Key: s3_encryption_check_lambda.zip
          import json
          import boto3

          def evaluate_compliance(configuration_item):
              bucket_name = configuration_item["resourceName"]
              s3_client = boto3.client('s3')

              try:
                  encryption_config = s3_client.get_bucket_encryption(Bucket=bucket_name)
                  rules = encryption_config['ServerSideEncryptionConfiguration']['Rules']
                  if rules:
                      return {
                          "compliance_type": "COMPLIANT",
                          "annotation": f"S3 bucket '{bucket_name}' has default server-side encryption enabled."
                      }
                  else:
                      return {
                          "compliance_type": "NON_COMPLIANT",
                          "annotation": f"S3 bucket '{bucket_name}' does not have default server-side encryption enabled."
                      }
              except s3_client.exceptions.ClientError as e:
                  if e.response['Error']['Code'] == 'ServerSideEncryptionConfigurationNotFoundError':
                      return {
                          "compliance_type": "NON_COMPLIANT",
                          "annotation": f"S3 bucket '{bucket_name}' does not have default server-side encryption enabled."
                      }
                  else:
                      print(f"Error checking encryption for bucket {bucket_name}: {e}")
                      return {
                          "compliance_type": "NOT_APPLICABLE",
                          "annotation": f"Could not determine encryption status for '{bucket_name}' due to error: {e}"
                      }

          def lambda_handler(event, context):
              invoking_event = json.loads(event['invokingEvent'])
              configuration_item = invoking_event['configurationItem']

              if configuration_item['resourceType'] != "AWS::S3::Bucket":
                  return {
                      "compliance_type": "NOT_APPLICABLE",
                      "annotation": "This rule applies only to S3 buckets."
                  }

              result = evaluate_compliance(configuration_item)

              config = boto3.client('config')
              config.put_evaluations(
                  Evaluations=[
                      {
                          'ComplianceResourceType': configuration_item['resourceType'],
                          'ComplianceResourceId': configuration_item['resourceId'],
                          'ComplianceType': result['compliance_type'],
                          'Annotation': result['annotation'],
                          'OrderingTimestamp': configuration_item['configurationItemCaptureTime']
                      },
                  ],
                  ResultToken=event['resultToken']
              )
              return result

  # Permission for Config to invoke Lambda
  ConfigPermissionToCallLambda:
    Type: AWS::Lambda::Permission
    Properties:
      FunctionName: !GetAtt S3EncryptionCheckLambda.Arn
      Action: lambda:InvokeFunction
      Principal: config.amazonaws.com
      SourceAccount: !Ref "AWS::AccountId"

  # AWS Config Custom Rule
  S3EncryptionConfigRule:
    Type: AWS::Config::ConfigRule
    Properties:
      ConfigRuleName: S3BucketDefaultEncryptionCheck
      Description: Checks if S3 buckets have default server-side encryption enabled.
      InputParameters: {} # No custom parameters needed for this example
      Scope:
        ComplianceResourceTypes:
          - AWS::S3::Bucket
      Source:
        Owner: CUSTOM_LAMBDA
        SourceIdentifier: !GetAtt S3EncryptionCheckLambda.Arn
        # Trigger the rule whenever a bucket is created/updated, or periodically
        EventSource: AWS::Config
        MessageType: ConfigurationItemChangeNotification # Triggers on resource changes
        MaximumExecutionFrequency: TwentyFour_Hours # Or One_Hour, Six_Hours etc. for periodic checks

Outputs:
  ConfigRuleARN:
    Description: ARN of the S3 Encryption Config Rule
    Value: !GetAtt S3EncryptionConfigRule.Arn
  LambdaFunctionARN:
    Description: ARN of the Lambda function for S3 Encryption Check
    Value: !GetAtt S3EncryptionCheckLambda.Arn

Step 4: Deploy and Monitor

Deploy this CloudFormation stack. Once deployed, AWS Config will start evaluating your S3 buckets. You can then view the compliance status in the AWS Config console under “Rules.” Non-compliant findings will indicate which buckets need attention.

Code Examples

Beyond the previous example, here are two more practical code examples.

Example 1: AWS Config Custom Rule for S3 Bucket Encryption (CloudFormation)

(Provided in the Implementation Guide section above)

Example 2: Python Lambda for RDS Multi-AZ Check (Reliability Pillar)

This Lambda function checks if all Amazon RDS instances in your account are configured for Multi-AZ deployment, a critical best practice for the Reliability pillar.

import json
import boto3

def lambda_handler(event, context):
    rds_client = boto3.client('rds')
    config_client = boto3.client('config')

    # Get all RDS DB instances
    response = rds_client.describe_db_instances()
    db_instances = response.get('DBInstances', [])

    evaluations = []

    for instance in db_instances:
        resource_id = instance['DBInstanceIdentifier']
        resource_type = "AWS::RDS::DBInstance"
        is_multi_az = instance.get('MultiAZ', False)

        compliance_type = "NON_COMPLIANT"
        annotation = f"RDS instance '{resource_id}' is NOT configured for Multi-AZ deployment."

        if is_multi_az:
            compliance_type = "COMPLIANT"
            annotation = f"RDS instance '{resource_id}' is configured for Multi-AZ deployment."

        evaluations.append({
            'ComplianceResourceType': resource_type,
            'ComplianceResourceId': resource_id,
            'ComplianceType': compliance_type,
            'Annotation': annotation,
            'OrderingTimestamp': event['configRuleTriggeredTime'] # Use current time for scheduled checks
        })

    # Publish the evaluation results to AWS Config
    if evaluations:
        config_client.put_evaluations(
            Evaluations=evaluations,
            ResultToken=event['resultToken']
        )

    return {
        'statusCode': 200,
        'body': json.dumps('RDS Multi-AZ Check completed.')
    }

# To trigger this Lambda periodically via EventBridge (CloudWatch Events):
#
# EventBridge Rule (e.g., in CloudFormation or console):
#
# Resources:
#   RdsMultiAzCheckEventRule:
#     Type: AWS::Events::Rule
#     Properties:
#       Description: "Triggers RDS Multi-AZ Lambda daily"
#       ScheduleExpression: "rate(24 hours)"
#       Targets:
#         - Arn: !GetAtt RdsMultiAzCheckLambda.Arn # ARN of your Lambda function
#           Id: "RdsMultiAzCheckTarget"
#
# Lambda Permissions for EventBridge:
#   RdsMultiAzCheckLambdaPermission:
#     Type: AWS::Lambda::Permission
#     Properties:
#       FunctionName: !GetAtt RdsMultiAzCheckLambda.Arn
#       Action: lambda:InvokeFunction
#       Principal: events.amazonaws.com
#       SourceArn: !GetAtt RdsMultiAzCheckEventRule.Arn
#
# And ensure your Lambda's IAM Role has permissions for rds:DescribeDBInstances and config:PutEvaluations.

Real-World Scenario: Enhancing Cloud Governance at GlobalTech Inc.

GlobalTech Inc., a rapidly growing SaaS company, struggled with maintaining consistent cloud best practices across its 50+ AWS accounts. Manual Well-Architected Reviews were conducted annually, but by then, architectural drift, security misconfigurations, and cost inefficiencies had often accumulated significantly. Their main pain points were:

  1. Security Gaps: Developers occasionally deployed resources without adhering to strict security baselines (e.g., public S3 buckets, unencrypted databases).
  2. Cost Overruns: Idle resources, over-provisioned instances, and forgotten EBS volumes were common.
  3. Compliance Reporting: Generating audit reports for SOC 2 and ISO 27001 was a monumental, manual task.

GlobalTech’s DevOps team decided to implement a custom WAF automation solution. They focused on two key pillars initially: Security and Cost Optimization.

Solution Implemented:

  • Centralized Governance Account: They set up a dedicated AWS account to host their automation tools, using AWS Organizations to manage member accounts.
  • Security Pillar Automation:
    • Deployed custom AWS Config Rules (like the S3 encryption example) to check for critical security configurations across all member accounts.
    • Integrated AWS Security Hub, pushing findings from GuardDuty, Macie, Inspector, and their custom Config Rules into a single dashboard.
    • Used AWS EventBridge to trigger Lambda functions in response to specific Config Rule non-compliance events. These Lambdas would then either alert the development team via Slack/Jira or, for critical issues (e.g., publicly accessible S3 bucket), trigger an AWS Systems Manager Automation document for immediate remediation.
  • Cost Optimization Pillar Automation:
    • Implemented a nightly AWS Step Functions workflow. This workflow orchestrated multiple Lambda functions: one to identify idle EC2 instances (based on CloudWatch metrics), another to detect unattached EBS volumes, and a third to flag oversized RDS instances.
    • Findings were aggregated and visualized in Amazon QuickSight dashboards, providing clear insights into potential cost savings. Alerts were sent to relevant team leads.
  • IaC Integration: Pre-deployment checks using cfn-lint and Checkov were integrated into their CI/CD pipelines (AWS CodePipeline), ensuring that CloudFormation templates adhered to WAF best practices before resources were provisioned.

Results:

Within six months, GlobalTech Inc. saw a dramatic improvement:

  • 90% Reduction in Security Incidents: Proactive identification and remediation reduced their security incident volume significantly.
  • 15% Reduction in Monthly AWS Spend: Through continuous cost optimization, they identified and acted on millions of dollars in annual savings.
  • Automated Compliance Reporting: Audit reports could now be generated on demand, showing a continuous compliance posture.
  • Faster Development Cycles: Developers gained confidence in deploying, knowing automated guardrails were in place, reducing fear of introducing non-compliant resources.

Best Practices for Robust Automation

Building your own WAF assessment tools requires careful planning and adherence to best practices:

  • Scope Definition: Clearly define which WAF pillars, services, accounts, and regions your custom tools will cover. Start small and expand incrementally.
  • Rule Granularity: Translate abstract WAF questions into concrete, measurable technical rules. For example, “How do you mitigate against single points of failure?” becomes “Check if RDS instances are Multi-AZ” or “Check if EC2 instances are in Auto Scaling Groups across multiple AZs.”
  • False Positives & Context: Design mechanisms to reduce alert fatigue. Implement justification workflows or allow findings to be marked as “acceptable risk” for specific, validated exceptions.
  • Centralized Deployment & Management: Leverage AWS Organizations, CloudFormation StackSets, or AWS Service Catalog to deploy and manage assessment tools consistently across multiple accounts and regions.
  • Version Control & CI/CD: Store all custom code (Lambda functions, Config Rules, CloudFormation/Terraform templates) in Git repositories (e.g., AWS CodeCommit, GitHub) and integrate them into CI/CD pipelines for automated testing and deployment.
  • Least Privilege IAM: Ensure all assessment tools (Lambda functions, Config roles) operate with the minimum necessary permissions required to perform their checks.
  • Modular & Extensible Architecture: Design components to be independent and easily updated as AWS services evolve or WAF best practices are refined.
  • Feedback Loop: Continuously review the effectiveness of your automated checks, update rules, and adapt to new AWS services or internal requirements. Engage with development teams to refine rules based on their operational context.
  • Leverage AWS WAF Custom Lenses: Complement your custom tools by creating Custom Lenses within the native AWS WAF Tool. This allows you to integrate your organizational-specific standards, questions, and best practices directly into the existing framework.
  • Embrace DevSecOps & Policy as Code: Integrate WAF checks as part of your CI/CD pipelines (Shift Left). Define security and compliance policies in machine-readable formats (e.g., OPA Gatekeeper, Sentinel) for automated enforcement.

Troubleshooting Common Issues

Even with careful planning, you may encounter issues:

  • Permission Errors:
    • Symptom: Lambda function fails to call AWS APIs (e.g., s3:GetBucketEncryption), or Config cannot invoke Lambda or publish evaluations.
    • Solution: Review the IAM role attached to your Lambda function and the permissions granted to AWS Config (e.g., lambda:InvokeFunction). Use CloudWatch Logs for Lambda to see detailed error messages.
  • Config Rule Not Triggering:
    • Symptom: Resources are created/modified, but the Config rule doesn’t run, or evaluation results are old.
    • Solution: Check EventSource and MessageType in your AWS::Config::ConfigRule definition. Ensure ConfigurationItemChangeNotification is set for immediate changes and MaximumExecutionFrequency for periodic checks. Verify AWS Config is enabled for the resource types you’re monitoring.
  • False Positives:
    • Symptom: Resources are flagged as non-compliant, but they are intentionally configured that way or fall under an approved exception.
    • Solution: Refine your Lambda logic to account for valid exceptions. Implement a tagging strategy (e.g., waf-exception:true) and modify your Lambda to check for these tags. Integrate with a waiver management system.
  • API Rate Limiting:
    • Symptom: Lambda functions fail with ThrottlingException errors, especially when processing many resources.
    • Solution: Implement exponential backoff and retry logic in your Lambda code. If processing a large number of resources, consider batching API calls or using AWS Step Functions to manage parallel execution with controlled concurrency.
  • Missing or Incomplete Data:
    • Symptom: Config items are missing for certain resources, or a resource is not being evaluated.
    • Solution: Ensure AWS Config is enabled for all desired resource types in the region. Verify CloudTrail logging is configured correctly for API activity monitoring.

Conclusion

Automating AWS Well-Architected Reviews by building your own assessment tools is not just an operational optimization; it’s a strategic imperative for any enterprise serious about its cloud adoption. It transforms a reactive, manual effort into a proactive, continuous journey of improvement, delivering unparalleled scalability, consistency, and insight. From enhancing your security posture and ensuring robust reliability to meticulously optimizing costs and simplifying compliance, the ROI is undeniable.

As you embark on this journey, remember that well-architected is a continuous state, not a destination. Embrace a DevSecOps mindset, integrate your checks early into the development lifecycle, and leverage the vast array of AWS services as your building blocks. The future of cloud governance is automated, intelligent, and deeply embedded into your operational DNA. Start small, iterate often, and witness your cloud environment evolve into a truly well-architected foundation for innovation.

Discover more from Zechariah's Tech Journal

Subscribe to get the latest posts sent to your email.

Leave a Reply

Scroll to Top