Does any kind of change to an EC2 auto-scaling group require a new AMI?

[marijakstrazdas]

Does any kind of change to an EC2 auto-scaling group require a new AMI?

If we run a set of EC2-instances in an Auto-Scaling Group, then rolling out any kind of change, either a new application version or, say, a config file change, requires a new AMI, which takes considerable time. Is this correct? For a production deploy, these kinds of turnarounds are fine, but for deploying a recent change to a dev/test environment, this would be a major impediment to our workflow. Do you have modules to support any quicker (to deploy) ways of making changes, e.g. CodeDeploy for new application versions, or any guidance on how customers typically solve that problem?

[yorinasub17]

If you want to avoid building a new AMI for code rollouts, then the best method is to bake the app installation and boot process into the user-data boot script. For example, you can have a pipeline that publishes your app artifacts to an artifactory (or S3 if you don't have one), and then in the user-data script of the instance it can download the artifact and start the app as the instances are booted up. When you need to do a deployment, you can update the launch configuration with the new user data script, which should be significantly faster than updating the AMI.

In this way, the AMI only contains the core baseline for the service (e.g., security baselines like fail2ban), which should change less often than their app.

Note that it is useful to be aware of why we recommend baking the app into the AMI. The main reason for doing this is to avoid configuration drift. If you use alternative methods of setting up the app (e.g., using ansible to deploy the app on the server), then you risk having mismatched servers when an instance fails and is replaced by the ASG without the operator knowing. The AMIs guarantee that every instance is exactly the same as it is replaced over time.

[yorinasub17]

From @brikis98 :

Basic options:

• Immutable infrastructure with EC2. You build a new AMI each time you make a change and deploy new EC2 instances with that AMI.
  • Advantages: super easy to use and understand, EC2 is proven and solid, you always know exactly what is deployed (which AMI), which makes debugging / repro / rollback easy.
  • Drawbacks: AMIs take 5 - 25 min to build, depending on what you’re doing and whether you encrypt them.
• Mutable infrastructure with EC2. You use a tool like Ansible or Chef to deploy updates to a known set of EC2 instances.
  • Advantages: EC2 is proven and solid, deploys are a bit faster (1 - 10 min).
  • Drawbacks: slightly more complicated to use, mutable infra means you don’t know exactly what’s deployed, so debugging / repro / rollback is a bit harder.
• Immutable infrastructure with Docker. You build a new Docker image each time you make a change and deploy the Docker images into a cluster (e.g., EKS/K8S cluster, ECS cluster, Fargate).
  • Advantages: deploys are much faster (1 - 5 min), you always know exactly what is deployed (which Docker image), which makes debugging / repro / rollback easy.
  • Drawbacks: considerably more complicated to use (esp EKS/K8S), Docker cluster tech isn’t nearly as proven as EC2.