Using a private AWS ECR repository with GitLab CI/CD build runners

We maintain our own EC2 based GitLab Runners to support our container based CI/CD builds. It's not efficient to modify a public image for every build step and it's not polite to the services we hit for those packages. Our solution is to generate container images for our build processes in advance and store them in our AWS ECR repo.

We also don't want to embed credentials in files on an EC2 instance, GitLab CI/CD variables or anywhere else if we can avoid it. We use the mechanism discussed in EC2 and Docker access to AWS services without embedded credentials so we don't have to.

The GitLab runner

Setting up and registering the runner is a fairly straightforward process. See GitLab Runner Documentation for details.

GOTCHA

As simple as it should be, it took me a while to figure out is why our runner couldn't find the docker ECR helper information as discussed in EC2 and Docker access to AWS services without embedded credentials. The reason is documented but easy to miss. From the GitLab docs, Access an image from a private container registry

To define which option should be used, the runner process reads the
configuration in this order:
...
A config.json file in $HOME/.docker directory of the user running
the process. If the --user flag is provided to run the child processes
as unprivileged user, the home directory of the main runner process
user is used.

In short, if you're using the --user flag to have the daemon drop privileges after start you need to put the various config files in /root/ not that user's home directory.

Once I put the content in the correct location it all worked as expected.

Container based builds

Again, per the process described in the other post, we include the ecr credential helper and the associated config in into our build container images and it all "just works".