DEV Community

Cover image for The 128-Hour Gap ⏰: Where Your AWS Budget is Leaking
Suhas Mallesh
Suhas Mallesh

Posted on • Edited on

The 128-Hour Gap ⏰: Where Your AWS Budget is Leaking

#cloud #devops #terraform #aws

Non-prod Auto Scaling Groups running at full capacity 24/7 waste 70% of costs. Here's how to add scheduled scaling with 5 lines of Terraform and reclaim those hours.

Reality check: Your development and QA environments run 24/7, but your team works 9-5, Monday-Friday.

The math:

Hours per week: 168 total
Developer hours: 40-50 (weekdays)
Wasted hours: 118-128 (nights + weekends)

Your dev environment is idle 70% of the time.
But you're paying 100% of the cost.
Enter fullscreen mode Exit fullscreen mode

Typical dev environment:

  • 3 EC2 instances (t3.large) = $225/month
  • Needed hours: 40/week
  • Actual usage: 70% waste
  • You could pay: $70/month instead

Let me show you how to add scheduled scaling with literally 5 lines of Terraform and slash your non-prod costs by 65%.

💸 The Always-On Tax

Most dev/staging/QA environments look like this:

Auto Scaling Group config:

Min: 3 instances
Max: 6 instances
Desired: 3 instances
Schedule: None (runs 24/7)
Enter fullscreen mode Exit fullscreen mode

Cost breakdown:

3 × t3.large instances
  - $0.0832/hour × 3 × 730 hours = $182/month
  - EBS volumes: $24/month
  - Load balancer: $16/month
Total: $222/month
Annual: $2,664

Actual needed hours: 50 hours/week = 217 hours/month
Waste: 513 hours/month (70% waste!)
Wasted money: $155/month = $1,860/year
Enter fullscreen mode Exit fullscreen mode

🎯 The Solution: Scheduled Actions

Auto Scaling Groups support scheduled actions - change capacity on a schedule.

Strategy:

  • Business hours (8 AM - 6 PM weekdays): Full capacity
  • Nights and weekends: Scale to zero (or minimal)

Implementation: 5 lines of Terraform per schedule.

🛠️ Terraform Implementation

Basic Weekday Schedule 

# asg-scheduled-scaling.tf

# Scale UP for business hours (8 AM Monday-Friday)
resource "aws_autoscaling_schedule" "scale_up_weekday" {
  scheduled_action_name  = "scale-up-weekday"
  min_size               = 3
  max_size               = 6
  desired_capacity       = 3
  recurrence             = "0 8 * * 1-5"  # 8 AM Mon-Fri (UTC)
  autoscaling_group_name = aws_autoscaling_group.dev.name
}

# Scale DOWN for nights (6 PM Monday-Friday)
resource "aws_autoscaling_schedule" "scale_down_weekday" {
  scheduled_action_name  = "scale-down-weekday"
  min_size               = 1
  max_size               = 2
  desired_capacity       = 1
  recurrence             = "0 18 * * 1-5"  # 6 PM Mon-Fri (UTC)
  autoscaling_group_name = aws_autoscaling_group.dev.name
}

# Scale DOWN for weekends (Friday night)
resource "aws_autoscaling_schedule" "scale_down_weekend" {
  scheduled_action_name  = "scale-down-weekend"
  min_size               = 1
  max_size               = 2
  desired_capacity       = 1
  recurrence             = "0 18 * * 5"  # 6 PM Friday (UTC)
  autoscaling_group_name = aws_autoscaling_group.dev.name
}

# Scale UP Monday morning
resource "aws_autoscaling_schedule" "scale_up_monday" {
  scheduled_action_name  = "scale-up-monday"
  min_size               = 3
  max_size               = 6
  desired_capacity       = 3
  recurrence             = "0 8 * * 1"  # 8 AM Monday (UTC)
  autoscaling_group_name = aws_autoscaling_group.dev.name
}
Enter fullscreen mode Exit fullscreen mode

That's it! 4 scheduled actions, 20 lines of code, done. ✅

Production-Ready Module 

# modules/asg-scheduled-scaling/main.tf

variable "asg_name" {
  description = "Auto Scaling Group name"
  type        = string
}

variable "business_hours" {
  description = "Business hours configuration"
  type = object({
    min_size         = number
    max_size         = number
    desired_capacity = number
    start_hour       = number  # 8 = 8 AM
    end_hour         = number  # 18 = 6 PM
  })
  default = {
    min_size         = 3
    max_size         = 6
    desired_capacity = 3
    start_hour       = 8
    end_hour         = 18
  }
}

variable "off_hours" {
  description = "Off-hours configuration"
  type = object({
    min_size         = number
    max_size         = number
    desired_capacity = number
  })
  default = {
    min_size         = 1
    max_size         = 2
    desired_capacity = 1
  }
}

variable "timezone_offset" {
  description = "Hours offset from UTC (e.g., -5 for EST)"
  type        = number
  default     = 0
}

locals {
  # Adjust hours for timezone
  start_hour_utc = (var.business_hours.start_hour - var.timezone_offset) % 24
  end_hour_utc   = (var.business_hours.end_hour - var.timezone_offset) % 24
}

# Scale up for business hours (weekdays)
resource "aws_autoscaling_schedule" "scale_up" {
  scheduled_action_name  = "${var.asg_name}-scale-up-weekday"
  min_size               = var.business_hours.min_size
  max_size               = var.business_hours.max_size
  desired_capacity       = var.business_hours.desired_capacity
  recurrence             = "${local.start_hour_utc} * * 1-5"
  autoscaling_group_name = var.asg_name
}

# Scale down for nights/weekends
resource "aws_autoscaling_schedule" "scale_down" {
  scheduled_action_name  = "${var.asg_name}-scale-down-weekday"
  min_size               = var.off_hours.min_size
  max_size               = var.off_hours.max_size
  desired_capacity       = var.off_hours.desired_capacity
  recurrence             = "${local.end_hour_utc} * * 1-5"
  autoscaling_group_name = var.asg_name
}

output "schedules_created" {
  value = {
    scale_up   = aws_autoscaling_schedule.scale_up.scheduled_action_name
    scale_down = aws_autoscaling_schedule.scale_down.scheduled_action_name
  }
}
Enter fullscreen mode Exit fullscreen mode

Usage 

# main.tf

# Dev environment - keep 1 instance off-hours
module "dev_scaling" {
  source = "./modules/asg-scheduled-scaling"

  asg_name        = aws_autoscaling_group.dev.name
  timezone_offset = -5  # EST

  business_hours = {
    min_size         = 2
    max_size         = 5
    desired_capacity = 2
    start_hour       = 8   # 8 AM EST
    end_hour         = 18  # 6 PM EST
  }

  off_hours = {
    min_size         = 1
    max_size         = 2
    desired_capacity = 1  # Keep 1 instance for CI/CD
  }
}

# QA environment - keep 1 instance off-hours
module "qa_scaling" {
  source = "./modules/asg-scheduled-scaling"

  asg_name        = aws_autoscaling_group.qa.name
  timezone_offset = -5

  business_hours = {
    min_size         = 3
    max_size         = 6
    desired_capacity = 3
    start_hour       = 7   # 7 AM EST (earlier for pre-work testing)
    end_hour         = 20  # 8 PM EST (later for evening testing)
  }

  off_hours = {
    min_size         = 1   # Keep 1 for overnight tests
    max_size         = 2
    desired_capacity = 1
  }
}

# Staging - keep 1 instance on weekends
module "staging_scaling" {
  source = "./modules/asg-scheduled-scaling"

  asg_name        = aws_autoscaling_group.staging.name
  timezone_offset = -5

  business_hours = {
    min_size         = 2
    max_size         = 4
    desired_capacity = 2
    start_hour       = 6   # 6 AM Monday
    end_hour         = 22  # 10 PM Friday
  }

  off_hours = {
    min_size         = 1
    max_size         = 2
    desired_capacity = 1  # Keep 1 on weekends
  }
}
Enter fullscreen mode Exit fullscreen mode

📊 Cost Savings Breakdown

Dev Environment (Keep 1 Instance Off-Hours)

Before:

3 × t3.large running 24/7
  - 3 × $0.0832/hour × 730 hours = $182/month
  - Annual: $2,184
Enter fullscreen mode Exit fullscreen mode

After (3 instances business hours, 1 instance off-hours):

Business hours (50 hrs/week = 217 hrs/month): 3 instances
Off-hours (118 hrs/week = 513 hrs/month): 1 instance

3 × $0.0832 × 217 + 1 × $0.0832 × 513
= $54 + $43 = $97/month
Annual: $1,164

Savings: $85/month = $1,020/year (47% reduction!) 🎉
Enter fullscreen mode Exit fullscreen mode

QA Environment (1 Instance Off-Hours)

Before:

4 × t3.xlarge running 24/7 = $485/month
Enter fullscreen mode Exit fullscreen mode

After:

Business hours (60 hrs/week): 4 instances
Off-hours (108 hrs/week): 1 instance

4 × $0.1664/hour × 260 hrs + 1 × $0.1664/hour × 470 hrs
= $173 + $78 = $251/month

Savings: $234/month = $2,808/year (48% reduction!)
Enter fullscreen mode Exit fullscreen mode

Typical Organization (5 Non-Prod Environments) 

2 Dev + 2 QA + 1 Staging = 5 environments
Average cost before: $300/month each = $1,500/month

After scheduled scaling:
  - Dev (2): $97/month each = $194
  - QA (2): $150/month each = $300
  - Staging: $120/month

Total after: $614/month
Savings: $886/month = $10,632/year 💰
Enter fullscreen mode Exit fullscreen mode

💡 Pro Tips

1. Account for Timezone

Scheduled actions use UTC time. Convert from your timezone:

# EST (UTC-5)
start_hour = 8   # 8 AM EST
utc_hour = 8 + 5 = 13  # 1 PM UTC

recurrence = "0 13 * * 1-5"  # Use UTC hour in cron
Enter fullscreen mode Exit fullscreen mode

Or use the module which handles conversion automatically!

2. Keep Minimal Capacity for Tests

Don't scale to absolute zero if you have:

  • Overnight CI/CD jobs
  • Automated testing suites
  • Health checks that need to pass 
off_hours = {
  min_size         = 1  # Keep 1 instance
  desired_capacity = 1
}
Enter fullscreen mode Exit fullscreen mode

3. Stagger Scale-Up Times

Avoid resource contention by staggering start times:

# Dev starts at 8 AM
recurrence = "0 8 * * 1-5"

# QA starts at 8:15 AM
recurrence = "15 8 * * 1-5"

# Staging starts at 8:30 AM
recurrence = "30 8 * * 1-5"
Enter fullscreen mode Exit fullscreen mode

4. Monitor Scale Events

Set up CloudWatch alarms:

resource "aws_cloudwatch_metric_alarm" "scale_up_success" {
  alarm_name          = "dev-scale-up-verification"
  comparison_operator = "LessThanThreshold"
  evaluation_periods  = 1
  metric_name         = "GroupDesiredCapacity"
  namespace           = "AWS/AutoScaling"
  period              = 300
  statistic           = "Average"
  threshold           = 3
  alarm_description   = "Dev ASG failed to scale up at 8 AM"

  dimensions = {
    AutoScalingGroupName = aws_autoscaling_group.dev.name
  }
}
Enter fullscreen mode Exit fullscreen mode

5. Add Holidays

Extend weekend shutdown for holidays:

# Thanksgiving week shutdown
resource "aws_autoscaling_schedule" "thanksgiving" {
  scheduled_action_name  = "thanksgiving-shutdown"
  min_size               = 0
  max_size               = 1
  desired_capacity       = 0
  start_time             = "2024-11-27T00:00:00Z"
  end_time               = "2024-12-02T08:00:00Z"
  autoscaling_group_name = aws_autoscaling_group.dev.name
}
Enter fullscreen mode Exit fullscreen mode

⚡ Quick Start 

# 1. Add scheduled actions to existing ASG
terraform apply

# 2. Verify schedules created
aws autoscaling describe-scheduled-actions \
  --auto-scaling-group-name dev-asg

# 3. Test scale-down manually (optional)
aws autoscaling set-desired-capacity \
  --auto-scaling-group-name dev-asg \
  --desired-capacity 0

# 4. Wait and verify instances terminate
aws ec2 describe-instances \
  --filters "Name=tag:aws:autoscaling:groupName,Values=dev-asg" \
  --query 'Reservations[].Instances[].State.Name'

# 5. Watch your bill drop next month 🎉
Enter fullscreen mode Exit fullscreen mode

⚠️ Common Gotchas

1. Cron Syntax is UTC, Not Local Time 

# WRONG - This is NOT 8 AM EST
recurrence = "0 8 * * 1-5"  # This is 8 AM UTC = 3 AM EST

# RIGHT - 8 AM EST = 1 PM UTC
recurrence = "0 13 * * 1-5"
Enter fullscreen mode Exit fullscreen mode

2. Min/Max/Desired Must Be Consistent 

# WRONG - Desired > Max
min_size         = 0
max_size         = 2
desired_capacity = 3  # Error!

# RIGHT
min_size         = 0
max_size         = 3
desired_capacity = 3
Enter fullscreen mode Exit fullscreen mode

3. Conflicting Schedules 

# WRONG - Both run at 8 AM Monday
recurrence = "0 8 * * 1-5"  # Weekday scale up
recurrence = "0 8 * * 1"    # Monday scale up (conflicts!)

# RIGHT - Monday schedule is redundant, remove it
recurrence = "0 8 * * 1-5"  # Covers Monday too
Enter fullscreen mode Exit fullscreen mode

4. Stateful Applications

If your instances store state (databases, sessions):

  • Don't scale to 0
  • Use lifecycle hooks to drain connections
  • Or better: make instances stateless!

🎯 Decision Matrix

Environment Recommended Strategy Expected Savings
Dev Keep 1 instance off-hours 40-50%
QA Keep 1 instance off-hours 50-60%
Staging Keep 1 instance on weekends 30-40%
Demo Scale to 1 except during demos 60-70%
Training Keep 1 instance, scale up for sessions 70-80%

📈 Real-World Example

Company: SaaS startup with 3 non-prod environments

Before scheduled scaling:

Dev:     3 × t3.large × 24/7    = $182/month
QA:      4 × t3.xlarge × 24/7   = $485/month
Staging: 2 × t3.large × 24/7    = $121/month

Total: $788/month
Annual: $9,456
Enter fullscreen mode Exit fullscreen mode

After scheduled scaling:

Dev:     3 inst. business hours, 1 off-hours = $97/month
QA:      60 hrs/week, 1 inst off-hours       = $251/month
Staging: Weekdays 2 inst., weekends 1 inst.  = $120/month

Total: $468/month
Annual: $5,616

Savings: $320/month = $3,840/year (41% reduction!)
Enter fullscreen mode Exit fullscreen mode

Implementation time: 1 hour

Lines of Terraform: ~40

Ongoing maintenance: Zero

🎯 Summary

The Problem:

  • Non-prod environments run 24/7
  • Developers work 40-50 hours/week
  • 70% of capacity is wasted
  • Costs accumulate unnecessarily

The Solution:

  • Auto Scaling scheduled actions
  • 5 lines of Terraform per schedule
  • Scale down nights and weekends
  • Zero application changes needed

The Result:

  • Typical savings: 40-60% on non-prod
  • One-time setup, runs forever
  • No operational overhead
  • Works with existing ASGs
  • Keeps minimal capacity for CI/CD

Stop running dev environments at full capacity 24/7 when your team works 9-5. Add scheduled scaling and reclaim those wasted hours. ⏰

Implemented scheduled scaling? How much did you save? Share in the comments! 💬

Follow for more AWS cost optimization with Terraform! ⚡

Top comments (0)