AWS elasticbeanstalk fun

I ran into a couple of odd issues with Elasticbeanstalk today. My EB AutoScalingGroup was unable to create any instances after some application deployments.

The problem

I had just created a new EB docker environment and tried deploying a test image with a “Hello World” Expressjs app. What would happen was EB would create a new ASG, kill the previous EC2 instance and then fail to create a new EC2 instance. The environment would be stuck in a state where it could not create any compute resources.

Simplified service architecture

Investigation

I was able to confirm that adding this .ebextensions triggered the issue.

option_settings:
  aws:autoscaling:launchconfiguration:
    BlockDeviceMappings: /dev/sdh=snap-07abdg5dd5s56875a

I was trying to mount a 3TiB volume (full of lidar data) to the instance.

The EB event logs were useless, but after some rubber duck debugging I got the idea to check the ASG activity log; this is what I found:

Cancelled
Launching a new EC2 instance: i-0431hfc8gaehbf8fe
2018 December 25 03:23:24 UTC-5
2018 December 25 03:23:55 UTC-5
Description:DescriptionLaunching a new EC2 instance: i-0431hfc8gaehbf8fe. Status Reason: Instance became unhealthy while waiting for instance to be in InService state. Termination Reason: Client.InvalidParameterCombination: Could not create volume with size 3072GiB from snapshot 'snap-1e161adfdfb4hhh6s'
Cause:CauseAt 2018-12-25T08:23:21Z an instance was started in response to a difference between desired and actual capacity, increasing the capacity from 0 to 1.

There were many of these log entries, it looks like the ASG kept retrying to create the desired number of EC2 instances after every failure. On a side note I opened a feature request with AWS to have the ASG failures reported back to EB.

After speaking to AWS support I realized the snapshot couldn’t exceed 1TB. So I created a new snapshot and updated the .ebextensions and re-deployed the app successfully.

But why didn’t EB support snapshots that were greater than 1TiB?

EBS volumes default to standard (Max size 1TiB) drives in the following regions when created using the API:

us-east-1, eu-west-1, eu-central-1, us-west-2, us-west-1, sa-east-1, ap-northeast-1, ap-northeast-2, ap-southeast-1, ap-southeast-2, ap-south-1, us-gov-west-1, cn-north-1

While all other regions default to gp2 (Max size 16 TiB)

There are currently no documented examples on how to specify a snapshot’s volume type, but AWS support was able to provide me with a way of doing so:

option_settings:
aws:autoscaling:launchconfiguration:
BlockDeviceMappings: /dev/sdh=snap-0d1agaaddgb4f5s4e:::gp2