AWS ECR Container Scans across AWS Organization accounts

AWS ECR Container Scans across AWS Organization accounts

July 10th, 2020

Problem statement

Currently Elastic Container Registry (ECR) offers container image scanning for the major Linux distributions and the packages from their package repositories, for free of charge. Such offering is great, but know its current limitations:

  1. Only major Linux distros supported i.e., Amazon Linux, Amazon Linux 2, Debian, Ubuntu, CentOS, Oracle Linux, Alpine, and RHEL. At the moment if you're using docker scratch or busybox, no ECR scans are available for you and any scan attempts will UnsupportedImageError as a response.

  2. Only operating system packages for the these Linux distros are targeted by the scans. Any application libraries (e.g., Node and Python modules, Java jars) are also not targeted.

  1. One can enable "scan on push" to trigger scans upon image push event. However, that's the only time the container image is scanned automatically. As new vulnerabilities are discovered daily, your images needs to be scanned periodically. It's higly likely that freshy baked container image is free from critical vulnerabilities, but not so anymore after a few months.

  2. There's no dashboard to easily visualize the vulnerabilities from your container images. One can build pipelines with CloudWatch events to alert whenever scans complete with alerts, which quickly becomes source of too many alerts and leads to alert fatigue.

  1. AWS ECR does not offer any means to integrate its vulnerability scanning feature with other AWS security tools like AWS Security Hub or AWS Guard Duty, which could offer some aggregation over these vulnerability findings across multiple AWS accounts in the AWS Organization.

There's a great project and blog post AWS ECR Vulnerabilities Notifier how to create alerting pipelines resolving the issues related to periodic scans and getting notified about new vulnerabilities. I also strongly believe, that the integration with other security tooling will become better by time. However, especially the lack of orgnization wide visibility to vulnerability findings is critical for any security automation and actions to prevent malicious actors from exploiting these.

Due to these reasons, I ended up creating a set of own tooling to bridge the current gaps. While still not able to scan than Linux images and operating system packages, the solution can serve as a low-cost, first stage of implementation for workflows on top AWS. Rather than integrating third-party tooling with your cloud infrastructure, the solution relies only on standard AWS APIs and accounts in your AWS Organization with AWS IAM role based access.

The solution in brief

  1. Preparations: Provision IAM roles. Master account should have role allowing ListAccounts action on the AWS Organizations API. All accounts in the organization should have an role associated, allowing the ECR API calls to start new ECR scans, listing and describing repositories, and to describe the scan findings. All these roles needs to be assumeable by role used to run the lambda functions created in the following steps.
  2. First step is to create a AWS Lambda function to query all accounts from your organization's master account, after which the lambda function triggers ECR scans for any image on those accounts (on any configured region). This lambda is triggered by CloudWatch Events once per day (which is also the service limit for ECR scan per image).
  3. The second step is to create a similar lambda functions as we created on the first step, but instead of triggering scans it will collect the scan findings from the repositories in the accounts. Any findings are written into DynamoDB and also into a JSON file, which is finally uploaded to S3. This function is triggered once every three hour.
  4. The third step uses DynamoDB stream, which will trigger a lambda function, that computes the changeset between the old DynamoDB entry and the new entry. If there are new high or critical vulnerabilities for the image, the lambda will post HTTP message to our Hubot that will notify the team on chat with a findings summary.
  5. Simple Vue.js web application serves as a dashboard to findings across different AWS accounts and regions. The web resources are served by a Nginx container running on EKS (or anyplace similar allowing you run containers and connecting them to request from your network). Nginx will also proxy any requests to the findings JSON file to S3, and this file is then used to populate the contents on web application. The access to JSON file in S3 in this example is limited to the VPC network used by the Nginx server.

ECR Findings

The code for the web dashboard as well as for the lambdas and Terraform to provision those is available on GitHub.