This post covers how to grant access from EC2 instances and docker containers on EC2 instances to other AWS services without the need for embedded credentials. It uses access to ECR as an example but the process is the same for access to any AWS service, e.g. ECS, S3 or RDS. Simply use a policy, as described below, to include whatever access you want to grant.

There are two major steps here:

Setup the policies and roles to allow an EC2 instance to access an ECR repo.
Setup the ECR credential helper

Once these are done, any process on the instance which has access to docker can access images without supplying credentials.

Policies and roles to access ECR

The first thing to do is setup up IAM polices and roles to allow the EC2 instance to access an ECR repo without stored credentials.

I'm going to use OpenTofu code examples here. You shouldn't need to understand OpenTofu/Terraform to get this gist of it. If you have any familiarity with AWS IAM and EC2 this should point you in the right direction. If not, I strongly suggest coming up to speed on these services before cutting and pasting a solution from the Internet.

IAM Policy

The key takeaway here is the policy itself which allows the retrieval of an auth token and read access to the ECR repo.

The value of Resource in this example will be supplied by OpenTofu. For those of you using another mechanism to manage the policy, the actual value follows the format "arn:aws:ecr:region:account-id:repository/repository-name"

Read/pull only access

resource "aws_iam_policy" "ml-ecr-build-ro" {
  name        = "ML-ECR-build-ro"
  description = "Read only access to build container images"
  policy = jsonencode({
    Version = "2012-10-17"
    Statement = [
      { Effect   = "Allow"
        Action   = ["ecr:GetAuthorizationToken"]
        Resource = ["*"]
      },
      { Effect = "Allow"
        Action = ["ecr:BatchCheckLayerAvailability",
          "ecr:BatchGetImage",
          "ecr:GetDownloadUrlForLayer"
        ]
        Resource = [aws_ecr_repository.ml_repos["build_containers"].arn]
      }
    ]
  })
}

Read/write pull/push access

If you want to push as well as pull, the ecr action list is:

        Action = [
          "ecr:BatchGetImage",
          "ecr:BatchCheckLayerAvailability",
          "ecr:CompleteLayerUpload",
          "ecr:GetDownloadUrlForLayer",
          "ecr:InitiateLayerUpload",
          "ecr:PutImage",
          "ecr:UploadLayerPart"
        ]

IAM Role

Once you have the policy you can:

Create a role
Add the policy to the role
Create an instance profile to allow an instance to assume that role.

The following is OpenTofu code to do this.

resource "aws_iam_role" "ec2_build_instance" {
  name        = "ML-EC2-BuildRunner"
  description = "Perms required by an EC2 build runner"
  assume_role_policy = jsonencode({
    Version = "2012-10-17"
    Statement = [
      {
        Action = "sts:AssumeRole"
        Effect = "Allow"
        Sid    = ""
        Principal = {
          Service = "ec2.amazonaws.com"
        }
      },
    ]
  })
}

# Policies attached here a specified in other files
resource "aws_iam_role_policy_attachment" "ml_ec2_build_instance_ecr" {
  role       = aws_iam_role.ec2_build_instance.name
  policy_arn = aws_iam_policy.ml-ecr-build-ro.arn
}

<other policies redacted>

# Instance profile to go along with role
resource "aws_iam_instance_profile" "ml_ec2_build_instance" {
  name = "ML-EC2-BuildInstance"
  role = aws_iam_role.ec2_build_instance.name
}

Associate the role with the instance(s)

You'll need to associate the role with the instance which should be granted access. How you that depends on how you manage your infrastructure. The related AWS docs are at Using an IAM role to grant permissions to applications running on Amazon EC2 instances

The ECR credential helper

The process uses the Amazon ECR Docker Credential Helper discussed in the Seamless access to AWS ECR post. However, there are two changes. Please see that post then note the following.

Executable

In that post, the helper binary, docker-credential-ecr-login, is placed below the users home directory. For this process, I would suggest putting it in a common, default location like /usr/local/bin.

AWS (lack of) credentials file

By associating a policy/role with the EC2 instance, processes running on that instance can automatically retrieve temporary credentials to access the ECR repo. This means you don't need to embed any credentials in a file. However, you do need to set some default values like region. The ~/.aws/credentials file looks like this:

[default]
region = us-east-2
output = json

Note there are no credentials here. Of course, you'll need to use the proper region for your setup.

Docker config.json

For completeness, this is the ~/.docker/config.json file. As above, modify to use your account account specifics.

{
  "auths": {},
  "credHelpers": {
    "012345678910.dkr.ecr-fips.us-east-2.amazonaws.com": "ecr-login",
    "012345678910.dkr.ecr.us-east-2.amazonaws.com": "ecr-login"
  }
}

Test it

You can test the config with any user account that is permitted to run docker. Simply enusre the user's ~/.aws/credentials and ~/.docker/config.json files are configured per above. Then, confirm that user can pull a docker image from the ECR repo without first using docker login.

$ docker image pull 012345678910.dkr.ecr-fips.us-east-2.amazonaws.com/build_containers:core_lint
core_lint: Pulling from build_containers
Digest: sha256:7ecf28d540e4dc28b1f3880ba250a7698b26c18903b8fdf8e689c93daf3e52f5
Status: Image is up to date for 012345678910.dkr.ecr-fips.us-east-2.amazonaws.com/build_containers:core_lint
012345678910.dkr.ecr-fips.us-east-2.amazonaws.com/build_containers:core_lint

Doing the above but from within a docker container

In theory:

Processes on the EC2 instance can now access ECR without embedded creds
Processes in docker on the instance are just processes on instance
Therefore:
- Processes "in" docker on that instance an can now access ECR without embedded creds

Right? The answer is "Not yet." I spent an inordinate amount of time tracking down why this wasn't working for me. Hopefully, the following saves someone else the same aggravation.

AWS IMDSv2

Among other things, AWS's Instance Metadata Service Version 2 is part of the layer that abstracts away all the underlying authentication necessary for the above EC2 role/policy access to "just work."

The problem

For security reasons, the default maximum hop count for IMDSv2 is 1. This means that any calls to IMDS have to come directly from the EC2 instance. However, docker is an abstraction layer. When the docker-credential-ecr-login helper runs in the container it is an extra network hop away from IMDS, i.e. 2 hops. As a result, IMDS refuses talk to it and the whole process fails.

The fix

You can simply increase the hop count to 2. This is a trivial security impact and resolves the issue. This is an example of how to do it for an existing instance using the AWS CLI.

> aws ec2 modify-instance-metadata-options --instance-id i-0123456789abcdef1 --http-put-response-hop-limit 2 --http-endpoint enabled
{
    "InstanceId": "i-0123456789abcdef1",
    "InstanceMetadataOptions": {
        "State": "pending",
        "HttpTokens": "required",
        "HttpPutResponseHopLimit": 2,
        "HttpEndpoint": "enabled",
        "HttpProtocolIpv6": "disabled",
        "InstanceMetadataTags": "disabled"
    }
}

See the AWS docs for Where to configure instance metadata options for both new and existing instances.