Enabling VPC Flow Logs Analysis

What are VPC Flow Logs?

VPC Flow Logs is an AWS feature that captures information about the IP traffic going to and from network interfaces in your VPC. Each flow log record represents a network flow in your VPC during a specific capture window.

CloudWrangler analyzes these logs to provide powerful traffic analysis capabilities:

• Visualize traffic patterns between resources
• Identify top talkers by bandwidth
• Detect rejected traffic (security issues)
• Troubleshoot connectivity problems
• Analyze database and cache connectivity
• Search historical network traffic

Why Enable VPC Flow Logs?

VPC Flow Logs unlock several advanced CloudWrangler features:

Traffic Flow Visualization See real-time (6-hour delay) traffic flows in the Infrastructure Diagram, helping you understand how your resources communicate.

Security Analysis Identify rejected connections that may indicate misconfigured security groups, attempted intrusions, or application errors.

Bandwidth Monitoring Track which resources are generating the most traffic and optimize accordingly.

Compliance and Auditing Search historical network traffic for compliance audits or security investigations with VPC Flow Logs Search.

Troubleshooting Quickly identify connectivity issues between resources by analyzing actual traffic patterns.

Benefits

Operational Visibility

• Understand your actual network topology (not just configured)
• Identify unexpected communication paths
• Detect orphaned resources

Security Monitoring

• Detect port scanning attempts
• Identify unauthorized access attempts
• Find misconfigured security groups
• Track data exfiltration

Cost Optimization

• Identify unnecessary NAT Gateway traffic
• Find resources that could use VPC endpoints
• Optimize inter-AZ data transfer

Performance Analysis

• Identify network bottlenecks
• Analyze database connection patterns
• Optimize load balancer configurations

Setup Process

CloudWrangler makes it easy to set up VPC Flow Logs with a guided, three-step process.

Step 1: Discovery

CloudWrangler automatically scans your AWS account to detect existing VPC Flow Logs configurations.

What CloudWrangler Checks:

• Existing VPC Flow Logs configurations
• S3 buckets with flow log data
• Flow log format (Parquet vs plain text)
• Athena database configuration
• IAM permissions

Discovery Results:

• Found: Existing flow logs detected, shows configuration details
• Not Found: No flow logs configured, shows setup options
• Partial: Some VPCs have flow logs, others don't

Click Continue to proceed to the template deployment step.

Step 2: Deploy Template

If flow logs aren't configured or need additional permissions, CloudWrangler provides a CloudFormation template.

What the Template Creates:

The CloudFormation template sets up everything needed for VPC Flow Logs analysis:

1. S3 Bucket (if needed)

   • Named: cloudwrangler-flow-logs-{account-id}
   • Configured for Parquet format
   • Lifecycle policies for cost optimization

2. VPC Flow Logs

   • Enabled for all VPCs in the region
   • Parquet format (required by CloudWrangler)
   • 1-minute aggregation interval
   • Captures all traffic (accepted and rejected)

3. IAM Role for CloudWrangler

   • Read access to the flow logs S3 bucket
   • Athena query permissions
   • Glue catalog access for metadata

4. Athena Configuration

   • Glue database: vpc_flow_logs
   • Table definitions for Parquet data
   • Query result bucket

Deployment Options:

Option A: CloudFormation Console (Recommended)

1. Click Launch CloudFormation Stack
2. Review the template in AWS Console
3. Check "I acknowledge that AWS CloudFormation might create IAM resources"
4. Click Create Stack
5. Wait 3-5 minutes for completion
6. Return to CloudWrangler

Option B: AWS CLI

# Download the template
curl -O https://cloudwrangler-public.s3.amazonaws.com/vpc-flow-logs-setup.yaml
 
# Deploy the stack
aws cloudformation create-stack \
    --stack-name CloudWrangler-VPC-Flow-Logs \
    --template-body file://vpc-flow-logs-setup.yaml \
    --parameters \
        ParameterKey=ExternalId,ParameterValue=YOUR-WORKSPACE-ID \
        ParameterKey=CloudWranglerAccountId,ParameterValue=119632503960 \
    --capabilities CAPABILITY_NAMED_IAM \
    --region us-east-1
 
# Wait for completion
aws cloudformation wait stack-create-complete \
    --stack-name CloudWrangler-VPC-Flow-Logs \
    --region us-east-1

Template Customization:

You can customize the template parameters:

• FlowLogBucketName - Use an existing S3 bucket
• RetentionDays - How long to keep flow logs (default: 30)
• VPCIds - Specific VPCs to monitor (default: all)
• TrafficType - ALL, ACCEPT, or REJECT (default: ALL)

Step 3: Verification

After deploying the CloudFormation template, CloudWrangler verifies the setup.

Verification Process:

CloudWrangler performs these checks:

1. IAM Role Access - Verify CloudWrangler can assume the role
2. S3 Bucket Access - Test read permissions on the flow logs bucket
3. Athena Configuration - Check database and tables exist
4. Flow Log Data - Verify Parquet files are being written
5. Query Test - Run a sample Athena query

This takes 30-60 seconds.

Success State:

When verification succeeds, you'll see:

• Green checkmark on all verification steps
• Summary of flow logs configuration
• Link to view traffic in Infrastructure Diagram
• Link to Flow Logs Search

What Happens Next:

CloudWrangler will:

• Query flow logs every 6 hours
• Process and aggregate traffic data
• Enrich logs with resource information
• Make data available in the UI

Cost Considerations

VPC Flow Logs are a paid AWS feature with the following cost components:

• VPC Flow Logs ingestion: AWS charges per GB of flow log data
• S3 storage: Standard S3 storage costs
• Athena queries: CloudWrangler uses Athena to query logs in your account

For accurate and up-to-date pricing information, please refer to AWS VPC Flow Logs Pricing.

Troubleshooting

No Flow Logs Appearing

1. Check flow log status:

   aws ec2 describe-flow-logs --flow-log-ids fl-xxxxx

   Status should be ACTIVE.

2. Wait 10-15 minutes: Flow logs aren't instant

3. Verify bucket permissions: Flow logs service needs write access

4. Check CloudWatch Logs: Errors appear in /aws/vpc/flowlogs

CloudWrangler Can't Read Flow Logs

1. Verify IAM role deployed:

   aws iam get-role --role-name CloudWranglerFlowLogAccess

2. Check file format: MUST be Parquet (not plaintext)

3. Test Athena manually:

   SELECT * FROM cloudwrangler_flow_logs.vpc_flow_logs LIMIT 10

4. Check S3 path: Should be AWSLogs/{account-id}/vpcflowlogs/{region}/...

No Traffic Showing

1. Verify resources have traffic: Ping between EC2 instances
2. Check flow log filter: Should be ALL (not just REJECT)
3. Wait for Athena: First query takes 30-60 seconds
4. Verify resource discovery: Resources must be discovered first

Advanced Configuration

Multi-Region Flow Logs

Enable flow logs in all regions to single bucket:

for REGION in us-east-1 us-west-2 eu-west-1; do
    VPC_IDS=$(aws ec2 describe-vpcs --region $REGION --query 'Vpcs[*].VpcId' --output text)
 
    for VPC_ID in $VPC_IDS; do
        aws ec2 create-flow-logs \
            --region $REGION \
            --resource-type VPC \
            --resource-ids $VPC_ID \
            --traffic-type ALL \
            --log-destination-type s3 \
            --log-destination arn:aws:s3:::central-flow-logs-bucket \
            --destination-options 'FileFormat=parquet,HiveCompatiblePartitions=true'
    done
done

Custom Log Format

Include additional fields:

--log-format '${version} ${account-id} ${interface-id} ${srcaddr} ${dstaddr} ${srcport} ${dstport} ${protocol} ${packets} ${bytes} ${start} ${end} ${action} ${log-status} ${vpc-id} ${subnet-id} ${instance-id}'

Organization-Wide Deployment

Use AWS Organizations and CloudFormation StackSets to deploy flow logs across all accounts.

What You Can Analyze

Once flow logs are enabled:

Traffic Analysis Page

• Top Talkers: Resources generating most traffic
• Rejected Traffic: Blocked by security groups/NACLs
• Protocol Breakdown: TCP, UDP, ICMP traffic
• Port Usage: Most common destination ports

Infrastructure Diagram

• Traffic Flows: Visual lines showing communication
• Bandwidth: Line thickness = traffic volume
• Direction: Arrows show traffic direction
• Bidirectional: Purple lines for two-way traffic

Security Insights

• Identify unauthorized traffic attempts
• Detect port scanning
• Find misconfigured security groups
• Track inbound/outbound patterns

Next Steps