Dynamic DNS hostnames with Route53 and EC2

I keep a lot of one-off EC2 instances for specialty tasks that I do once in a while. I have a FreeBSD instance, a phpMyAdmin instance, a forensic analysis instance, and so on. It's a good practice with AWS EC2 to stop these instances when they aren't needed. It means that I don't pay for instance hours that I'm not using. I start them, do what I need to do, and then stop them when I'm done. I assign public IP addresses to these instances so that I can reach them, but the instance gets a new IP public address each time it starts up. Rather than looking up the new public IP address each time the instance starts, I want to use a DNS name to reach it. I need an A record in DNS to be updated each time the instance boots, so that I can reach it by name, instead of looking up its new public IP address every time I start it.

In this blog post I provide a script that runs at boot time to update DNS each time an instance boots with a public IP. The script figures out the DNS record that needs to be created by looking at the instance metadata, and then it updates records in the right zone in Route53. Because the script pulls all the important information from the instance metadata, there's no unique code or configuration that has to be installed on the instance. I just install this one script and it will figure out the rest from EC2 metadata. If you launch an instance from an AMI, and you tag that instance correctly and you put the instance into the right role, this will work without any changes specific to the individual instance itself.

Assumptions

To make this work, you need a few things before you start. I am not going to describe how to do them.

1. You need a public hosted zone in Route53
2. You need python and boto3 installed in your instance
3. You need to create your own AMI, or your own way of customising instances after launch.

Public hosted zones are described in the Route53 documentation, and installing python and boto3 are described in the boto3 documentation.

Overview

The process looks like this.

1. Create a dynamic subzone for these host names
2. Create a policy that allows editing the subzone
3. Create a role and attach the policy to it (or attach the policy to an existing role)
4. Load the script onto the instance and enable it at boot time
5. Launch an instance with the role and correct tags

Create a Hosted DNS Zone

In the interest of security, we want a dedicated zone for these dynamic host names. We don't want to grant EC2 instances the privilege of writing into our top-level zone. For example, if your organisation is “example.com”, you don't want EC2 instances launching that have rights to modify DNS records in the example.com domain. In this example, I'll use a subzone of example.com, ec2.example.com, for all my dynamic host names. You can imagine uses like dev.example.com, test.example.com, sandbox.example.com, and so on. We can grant permissions to modify one specific hosted domain without worrying about unauthorised changes to example.com itself. This also works if example.com is not running in Route53, but we want ec2.example.com to be dynamic and hosted in Route53. It requires cooperation with the example.com domain owner, but just once.

Create the Subzone

In the AWS Console, create a zone called ec2.example.com. This is not a record set in example.com whose name is “ec2” (we'll create one of those in a second). First you create a new hosted zone.

Capture Some Details

When you create the hosted zone, you immediately see some NS records. It's a list of DNS server names. Copy these NS records. You will also see a Hosted Zone ID. It looks like this: ZZ123ABCD123. Make note of that ID for future use.

Create the Glue Records

Go to your example.com hosted zone and create a record set.

1. For the name, choose whatever subzone you're creating. In my case, I type ec2 so that I am creating a record set ec2.example.com.
2. For the record set type, choose “NS – Name Server”.
3. In the Value box, paste the name servers that you just copied. This is called creating a “glue record” in DNS. It indicates that ec2.example.com is a separate DNS zone (as opposed to an individual host) and it designates the name servers who can answer queries about that zone.

Create an IAM Policy to Allow Changing the Zone

Before our host can dynamically update a zone, we have to grant it permissions. It is important to grant the instance just enough privileges to do this, and nothing more. It's also poor practice to embed AWS IAM access keys or secrets in code. To avoid coding secrets, we are going to use an EC2 Instance Role. That role will need a policy, so that's why we create the policy first.

To create a policy¹ that allows an entity to edit records in your ec2.example.com subzone, go to IAM and choose Create Policy. Choose “Create Your Own Policy”. Give it a descriptive title. I chose r53-edit-ec2-example-com to indicate that it allows editing the ec2.example.com zone in Route53. Enter the following policy. Note that where it says [ZONEID] that's where you put enter the subzone Hosted Zone ID that you recorded earlier.

This policy is in the dns-update-policy.json file here in the repository. It grants the minimum privileges needed by the script. If any of these are restricted, the script will not be able to do its work.

¹ This part of the solution was inspired by this blog post at cantina.co.

Attaching the Policy to an EC2 Instance Role

If you already have a role that you apply to your EC2 Instances, you don't need to create another role. Skip this next step. If you don't have an instance role, then you should create one so that you can attach policies to it.

Create the Role

Go to IAM and create a new role. Give it a descriptive name. I chose dynamic-ec2 to express the idea that these instances are dynamic and in my ec2 DNS zone. Under “AWS Service Roles” click “Select” next to “Amazon EC2”. From the policy list, find the r53-edit-ec2-example-com policy and attach it to the role you're creating. Click “Create Role” and the role will be created.

Attach the Policy

If you already have an instance role that your instances use, find that role and edit it. Attach this policy to that instance role. Now those instances will also have the privilege to edit this Route53 zone, in addition to whatever privileges their existing role grants them.

Launch Instances with the Role

Note that you can't assign a role to an existing instance; you can only specify a role when you launch a new instance. Be sure to specify add this role in Step 3: Configure Instance Details.

From a command line or launching instances via API calls, it is a two-step process. First you create an instance profile for your instances. Then you attach the role to the instance profile. Finally, you launch instances with that instance profile.

See the documentation on Working with Roles for information about attaching roles to instance profiles. For more information about instance profiles, see the documentation About Instance Profiles.

Create The Tag

You need a tag called dnsname on the instance. It should be the full hostname, including domain, that you want for the instance. If you set dnsname to be webdev1.ec2.example.com , then this script will attempt to create an A record for webdev1.ec2.example.com when it runs. It will look for ec2.example.com in the hosted zones of Route53, and then will try to create an A record into that zone.

Install the Script

You need to install this script and make it run at boot time somehow. This might mean creating a script in /etc/rc3.d on a Linux instance. In my case, I just saved the script in /home/ec2-user/auto-update-r53.py and then I added this line to /etc/rc.local:

  /usr/bin/python /home/ec2-user/auto-update-r53.py

The script uses an “UPSERT” call to Route53. That is, if the A record already exists, it is updated. If the A record does not exist, it is inserted. Update + insert = UPSERT.

The script does a few things:

1. It reads the dnsname tag from the instance metadata.
2. It reads the public IP address from the instance metadata.
3. It searches for the domain in the list of hosted zones at Route53.
4. It sends an UPSERT request to create the record in the zone.

If all goes well, it will print a single line of output like this, which will be visible in the instance log.

setting webdev1.ec2.example.com to 54.123.123.123

Security Considerations

All hosts that have this privilege can create, modify, and change any record in the zone. Nothing restricts these hosts from modifying important records like NS records, CNAMES, or any other legitimate record in the zone. This is probably appropriate for a sandbox or dev environment and not suitable for a production environment.

Domain Names and the Same Origin Policy

It is dangerous to allow development code to run using an important top level domain. If www.example.com is the web address for an online retail web site, it would be a bad idea to allow developers to create hosts with names like www.ec2.example.com. Various web browser security mechanisms (e.g., the Same Origin Policy) will tend to trust subzones the same as the root zone. Web browsers with cookies for production use at www.example.com would very likely send those same cookies to www.ec2.example.com (depending on how the application is written). And this could expose security tokens from production to developer web applications who should not see them.

A simple solution to this trust problem is to buy a separate domain that is never used in production and use it for these dynamic hosts. If www.example.com is our online commerce site, we could use one of the other top level domains for our development environments. We could register something like example.tech or example.build for our development work, and then use webdev1.ec2.example.tech as our dynamic host name, while www.example.com remains the name of the production web site.

Policy and Role Complexity

This example used one policy per zone. If you have hundreds of zones that you manage this way, that approach might become cumbersome with so many policies. Instead of having individual policies that control access to a single domain, you could potentially create a single policy that grants privileges to lots of domains (e.g., all the development and test subzones). This will make it a bit easier to manage, but it also makes the security less fine-grained. The right balance depends on the number of domains you manage and how complex the security roles are in your operations teams.

Potential Extensions

This technique can be extended deeper to subzones. If we want to have additional zones that indicate AZ or region, we can. This could easily be extended to create hostnames like webdev1.az1a.eu-west-1.ec2.example.com. It requires creating additional hosted zones and glue records. It also requires creating more IAM policies that grant control to the different zones. But the script that is launched on each instance is identical. It will determine its region, public IP address, hostname, and zone from the instance metadata and it will make the correct API calls.

This example uses UPSERT to create an A record if it doesn't exist, or to update a record if it already does exist. That's an easy way to work but it has a potential side-effect. If you don't manage tags carefully, you can have two hosts that both create the same hostname in DNS. There will be no error and it will successfully overwrite the existing DNS name. This might not be the behaviour you want. You might end up with instances that are still running, but are no longer reachable via the DNS name that you meant for them to have. If two instances are launched with webdev1.ec2.example.com as their dnsname tag, both will attempt to set the A record in the ec2.example.com zone. Both will succeed. The second one will be reachable by that DNS name, and the first one will have no DNS name associated with it. One solution to this problem is to simply be meticulous with tagging. The other possible solution is to change UPSERT to CREATE in the Python script. The CREATE command will fail if an A record already exists with the same name. Using CREATE instead of UPSERT requires deleting A records manually, though (or creating a different automated process that deletes them).

Conclusion

Using this technique, instances can be stop and started at will, but they will use a consistent DNS name. Even though their IP address changes each time they start, the hostname will be the same. There's no need to use an elastic IP address for short-lived instances that have public IPs.