AWS Elastic Registry Scanner

You can use the registry scanner for AWS Elastic registries.

AWS ECR Single Account

Use the single account AWS Elastic Container Registry (ECR) installation to target one registry per installation.
To onboard an entire organization containing several member accounts, check the AWS ECR Organizational setup

Ensure you update the Sysdig Helm chart before installation. This lets you access the latest features and fixes.

Use one of the following installation methods based on where the Kubernetes cluster is located:

Same AWS Account

If Elastic Kubernetes Service (EKS) and the registry are in the same account, launch the installation in that account with an AWS EKS. No specific credential setup is required, since EKS already grants AmazonEC2ContainerRegistryReadOnly policy to the default nodes-eks-node-group-* role.

$ helm upgrade --install registry-scanner sysdig/registry-scanner --version=1 \
--set config.secureBaseURL=<SYSDIG_SECURE_URL> \
--set config.secureAPIToken=<SYSDIG_SECURE_API_TOKEN> \
--set config.registryType=ecr \
--set config.registryURL=<AWS_REGISTRY_URL> \
--set config.aws.region=<AWS_REGION>
  • <AWS_REGION>: Registry AWS region
  • <AWS_REGISTRY_URL>: Registry URL, such as 123456789012.dkr.ecr.us-east-1.amazonaws.com.

On Red Hat OpenShift Service on AWS (ROSA), the Registry Scanner pod cannot use the worker node role. Follow Red Hat OpenShift Service on AWS (ROSA) instead.

Different Account or Kubernetes Not in AWS

Credentials are required in this case:

$ helm upgrade --install registry-scanner sysdig/registry-scanner --version=1 \
--set config.secureBaseURL=<SYSDIG_SECURE_URL> \
--set config.secureAPIToken=<SYSDIG_SECURE_API_TOKEN> \
--set config.registryType=ecr \
--set config.registryURL=<AWS_REGISTRY_URL> \
--set config.aws.region=<AWS_REGION> \
--set config.aws.accessKeyId=<AWS_ACCESS_KEY_ID> \
--set config.aws.secretAccessKey=<AWS_SECRET_ACCESS_KEY>
  • <AWS_REGION>: Registry AWS region

  • <AWS_REGISTRY_URL>: Registry URL, such as 123456789012.dkr.ecr.us-east-1.amazonaws.com.

  • <AWS_ACCESS_KEY_ID>, <AWS_SECRET_ACCESS_KEY> from an AWS user with the following permissions for the given registry to be scanned:

    ecr:GetAuthorizationToken
    ecr:BatchCheckLayerAvailability
    ecr:GetDownloadUrlForLayer
    ecr:GetRepositoryPolicy
    ecr:DescribeRepositories
    ecr:ListImages
    ecr:DescribeImages
    ecr:BatchGetImage
    ecr:GetLifecyclePolicy
    ecr:GetLifecyclePolicyPreview
    ecr:ListTagsForResource
    ecr:DescribeImageScanFindings
    

Limitations

  • The scanner for single-account will target a single registry; in other words, the registry of one region. To scan several regions, either use the AWS Organizational scanner or set up one scanner per region.
  • If the AWS Kubernetes cluster and the registry are in different accounts but under the same organization, use the AWS Organizational setup and allow listing a specific account.

Red Hat OpenShift Service on AWS (ROSA)

On Red Hat OpenShift Service on AWS (ROSA) with hosted control planes (HCP), the Registry Scanner reads ECR with an IAM role bound to its Kubernetes service account. You don’t store AWS access keys in the cluster.

The ROSA worker node role grants read access to ECR, but pods on ROSA cannot use it. The Registry Scanner needs its own IAM role, as described in this section, or access keys.

Prerequisites

  • A ROSA with HCP cluster that uses AWS Security Token Service (STS).
  • An ECR registry in the AWS account of the cluster. For a registry in another account, see Registry in a Different AWS Account.
  • The Registry Scanner Helm chart 1.12.0 or later.
  • Registry Scanner v0.14.0 or later.
  • The rosa, aws, oc, helm, and jq command-line tools, with permission to create IAM roles and policies in the AWS account.

Find the Cluster Values

A ROSA cluster doesn’t present the same identity values as an EKS cluster. Read them from your cluster:

  1. Get the OIDC issuer of the cluster:

    OIDC_ISSUER=$(rosa describe cluster --cluster <CLUSTER_NAME> --output json \
      | jq -r .aws.sts.oidc_endpoint_url | sed 's|^https://||')
    echo "$OIDC_ISSUER"
    

    The output looks like oidc.op1.openshiftapps.com/<OIDC_CONFIG_ID>.

  2. Verify that the IAM OIDC provider for that issuer exists in the cluster account:

    aws iam get-open-id-connect-provider \
      --open-id-connect-provider-arn arn:aws:iam::<AWS_ACCOUNT_ID>:oidc-provider/$OIDC_ISSUER \
      --query ClientIDList
    

    ROSA creates this provider when you create the OIDC configuration of the cluster, so you don’t need to create it as you would on EKS.

Create the IAM Role

  1. Create the trust policy. Replace <AWS_ACCOUNT_ID> with the ID of the AWS account where you create the role, and <NAMESPACE> with the namespace where you install the Registry Scanner:

    cat > registry-scanner-trust-policy.json <<EOF
    {
      "Version": "2012-10-17",
      "Statement": [
        {
          "Effect": "Allow",
          "Principal": { "Federated": "arn:aws:iam::<AWS_ACCOUNT_ID>:oidc-provider/${OIDC_ISSUER}" },
          "Action": "sts:AssumeRoleWithWebIdentity",
          "Condition": {
            "StringEquals": {
              "${OIDC_ISSUER}:sub": "system:serviceaccount:<NAMESPACE>:registry-scanner",
              "${OIDC_ISSUER}:aud": "openshift"
            }
          }
        }
      ]
    }
    EOF
    

ROSA issues service account tokens with the openshift audience. If the trust policy expects sts.amazonaws.com, as on EKS, the Registry Scanner cannot assume the role.

  1. Create the permissions policy. Replace <AWS_ACCOUNT_ID> with the ID of the AWS account of the registry, and <AWS_REGION> with the registry region:

    cat > registry-scanner-ecr-policy.json <<EOF
    {
      "Version": "2012-10-17",
      "Statement": [
        {
          "Effect": "Allow",
          "Action": ["ecr:GetAuthorizationToken"],
          "Resource": "*"
        },
        {
          "Effect": "Allow",
          "Action": [
            "ecr:DescribeRepositories",
            "ecr:ListImages",
            "ecr:DescribeImages",
            "ecr:BatchGetImage",
            "ecr:GetDownloadUrlForLayer"
          ],
          "Resource": "arn:aws:ecr:<AWS_REGION>:<AWS_ACCOUNT_ID>:repository/*"
        }
      ]
    }
    EOF
    

    ecr:GetAuthorizationToken cannot be scoped to a resource. To scan only some repositories, keep ecr:DescribeRepositories on repository/* so that the Registry Scanner can discover them, and scope the other actions to those repositories. Repositories outside that scope log AccessDenied errors for ecr:ListImages during discovery.

  2. Create the role and attach the policy:

    aws iam create-role --role-name <ROLE_NAME> \
      --assume-role-policy-document file://registry-scanner-trust-policy.json
    aws iam put-role-policy --role-name <ROLE_NAME> \
      --policy-name registry-scanner-ecr-read \
      --policy-document file://registry-scanner-ecr-policy.json
    

Install the Registry Scanner on ROSA

Log in to the cluster with oc login, then install the Registry Scanner in the namespace that you used in the trust policy:

helm upgrade --install registry-scanner sysdig/registry-scanner --version=1 \
--namespace <NAMESPACE> --create-namespace \
--set config.secureBaseURL=<SYSDIG_SECURE_URL> \
--set config.secureAPIToken=<SYSDIG_SECURE_API_TOKEN> \
--set config.registryType=ecr \
--set config.registryURL=<AWS_REGISTRY_URL> \
--set config.aws.region=<AWS_REGION> \
--set serviceAccount.annotations."eks\.amazonaws\.com/role-arn"=arn:aws:iam::<AWS_ACCOUNT_ID>:role/<ROLE_NAME>
  • <AWS_REGISTRY_URL>: Registry URL, such as 123456789012.dkr.ecr.us-east-1.amazonaws.com.
  • <AWS_REGION>: Registry AWS region.

When the Helm release is named registry-scanner, the chart names the service account registry-scanner, which matches the sub condition of the trust policy. The ROSA pod identity webhook reads the role from the service account annotation and gives the Registry Scanner pods a web identity token for it.

Set the role through serviceAccount.annotations, as shown. Don’t set config.aws.accessKeyId or config.aws.secretAccessKey.

The Registry Scanner pods run under the default restricted-v2 security context constraint. You don’t need to grant an additional one.

Verify the Installation on ROSA

  1. Start a scan instead of waiting for the schedule:

    oc create job --from=cronjob/registry-scanner registry-scanner-manual -n <NAMESPACE>
    
  2. Check the result:

    oc logs -n <NAMESPACE> job/registry-scanner-manual | grep -E "credentials provider|Total selected"
    

    A successful scan logs credentials provider type: 'ecr' and a summary such as Total selected images to be scanned: 3 = Success: 3 + Failed: 0 + Skipped: 0.

The results appear in Sysdig Secure under Vulnerabilities > Registry a few minutes after the scan.

Troubleshoot ROSA

SymptomCauseResolution
InvalidIdentityToken or AccessDenied for sts:AssumeRoleWithWebIdentityThe trust policy doesn’t match the tokenVerify that the federated principal and the condition keys use the cluster OIDC issuer, that aud is openshift, and that sub matches the namespace and the service account name
using AWS ECR default credentials followed by dial tcp 169.254.169.254:80: connect: connection refusedThe service account has no eks.amazonaws.com/role-arn annotation, so the Registry Scanner falls back to the node role, which pods on ROSA cannot reachSet the annotation through serviceAccount.annotations and run the scan again
AWS calls to STS or ECR time outThe VPC routes STS or ECR through interface endpoints whose security group doesn’t allow HTTPS from the worker nodesAllow inbound TCP 443 from the worker nodes on the endpoint security group
AccessDenied for ecr:ListImages on some repositoriesThe permissions policy scopes the read actions to other repositoriesExpected for repositories outside the scope. Extend the policy to scan them

Registry in a Different AWS Account

If the registry is in another AWS account than the cluster, create the IAM role in the registry account, and register the cluster OIDC issuer in that account first:

aws iam create-open-id-connect-provider \
  --url "https://${OIDC_ISSUER}" \
  --client-id-list openshift

Then follow Create the IAM Role with the registry account ID as <AWS_ACCOUNT_ID>.

To scan the registries of an AWS organization from ROSA, use this role as the service account role in AWS ECR Organizational.

ROSA Limitations

  • Validated on ROSA with HCP clusters that use STS. ROSA classic clusters and clusters without STS are not validated.

AWS ECR Organizational

AWS organizational accounts have a management account with multiple member accounts and multiple registries or regions. This option allows scanning of all registries on any member account or the region of an AWS Organization. The Kubernetes cluster that the registry will scan may be inside or outside of AWS.

Process Overview

You will:

  1. Set up roles and permissions in the AWS console to be used for registry scanning.
    • Optionally, validate the setup with a Sysdig test script
    • Optionally, add a configuration to the Helm chart to limit the member accounts that will be checked for scanning.
  2. Install with the Helm chart in one of two ways:
    • Option 1: Access Key/Secret Key With an AWS Identity Access Management (IAM) user security credentials, aimed for quick testing or a more simple setup.
    • Option 2: Service Account With AWS Kubernetes Service Account, aimed for a more productive scenario.

Set Up Permissions in AWS

In the AWS console, set up authentication roles to allow discoverability.

You will:

  • Create one role in the management account (ORGANIZATION_MANAGEMENT_ROLE_ARN).
  • A recommended step is to create a role name to be assigned to every targeted member account (ORGANIZATION_MEMBER_ACCOUNTS_ROLE_NAME). Otherwise, the default AWS member role name OrganizationAccountAccessRole will be used, which includes admin privileges.
    • Tip: To narrow the list of member accounts that will be checked during scanning, you can set the aws.AllowListMemberAccountIDs attribute option in the Registry Scanner Helm chart. This eliminates noisy error messages.
  • Optionally, validate the setup using Sysdig’s test script.
Create Roles
  • Management role: ORGANIZATION_MANAGEMENT_ROLE_ARN will be assumed and used to discover member accounts.
  • Member role name: ORGANIZATION_MEMBER_ACCOUNTS_ROLE_NAME will be assumed on the member accounts to discover and scan their registries.
Providing Credentials to the Registry Scanner

The registry scanner requires three types of credentials to discover and read Amazon Elastic Container Registry (ECR) images from all member accounts and all regions within an AWS organization.

Kubernetes Job/Pod AWS Credentials

You need to provide either an Access Key/Secret Key or a service account that has the necessary permissions.

  • Access Key/Secret Key

    If you choose to provide an Access Key/Secret Key, use the <AWS_ACCESS_KEY_ID>, <AWS_SECRET_ACCESS_KEY> format. Then, you need to set the policy permissions required to allow these credentials to impersonate the ORGANIZATION_MANAGEMENT_ROLE_ARN.

    Use the following policy to allow the impersonation:

    {
      "Version": "2012-10-17",
      "Statement": [
        {
          "Sid": "AllowAssumeManagementRole",
          "Effect": "Allow",
          "Action": "sts:AssumeRole",
          "Resource": "<ORGANIZATION_MANAGEMENT_ROLE_ARN>"
        }
      ]
    }
    
  • Service Account

    For Amazon Elastic Kubernetes Service (EKS) clusters on AWS, you can configure the necessary permissions at the service account level.

    Important points:

    1. Create an IAM OpenID Connect (OIDC) provider for your EKS cluster. Note: If you deployed the cluster with the EKS Terraform module, it was created automatically (check oidc_provider output).

    2. In the role associated with the service account (SERVICE_ACCOUNT_IAM_ROLE_ARN), create a policy to enable assuming the role of the management account.

      {
        "Statement": [
          {
            "Action": "sts:AssumeRole",
            "Effect": "Allow",
            "Resource": "<ORGANIZATION_MANAGEMENT_ROLE_ARN>",
            "Sid": ""
          }
        ],
        "Version": "2012-10-17"
      }
      
    3. In the role associated with the service account (SERVICE_ACCOUNT_IAM_ROLE_ARN), configure a trust relationship to allow the service account in Kubernetes to assume the role.

      • Default SERVICE_ACCOUNT_NAME will be registry-scanner, derived from the helm installation release name.
      {
          "Version": "2012-10-17",
          "Statement": [
              {
                  "Effect": "Allow",
                  "Principal": {
                    "Federated": "arn:aws:iam::<MANAGEMENT_ACCOUNT>:oidc-provider/<OIDC_PROVIDER>"
                  },
                  "Action": "sts:AssumeRoleWithWebIdentity",
                  "Condition": {
                    "StringEquals": {
                      "<OIDC_PROVIDER_URL>:sub": "system:serviceaccount:<NAMESPACE>:<SERVICE_ACCOUNT_NAME>",
                      "<OIDC_PROVIDER_URL>:aud": "sts.amazonaws.com"
                    }
                  }
              }
          ]
      }
      
    4. Save the role <SERVICE_ACCOUNT_IAM_ROLE_ARN> value associated with the Kubernetes service-account and use it during the installation.

Credentials for the Management Role

Provide credentials for the <ORGANIZATION_MANAGEMENT_ROLE_ARN> to discover registries for all member accounts and all regions. You must assign policy permissions and trust relationships to this role.

Assign Policy Permissions

Assign one of the following policy permissions to the management role:

  • AWSOrganizationsReadOnlyAccess: AWS managed policy which Provides read-only access to AWS Organizations.

  • Inline policy to allow the management role to assume member account roles:

    {
      "Version": "2012-10-17",
      "Statement": [
        {
          "Sid": "AllowManagementRoleToAssumeMember",
          "Effect": "Allow",
          "Action": "sts:AssumeRole",
          "Resource": "arn:aws:iam::*:role/<ORGANIZATION_MEMBER_ACCOUNTS_ROLE_NAME>"
        }
      ]
    }
    

Assign the Trust Relationship

You must assign the trust relationship to allow this role to be assumed by the principal of the AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY / service account configured in the previous step.

  • Access Key/Secret Key
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "AWS": "<AWS_ACCESS_USER_ARN>"
      },
      "Action": "sts:AssumeRole"
    }
  ]
}
  • Service Account

    {
        "Version": "2012-10-17",
        "Statement": [
            {
                "Effect": "Allow",
                "Principal": {
                  "AWS": "<SERVICE_ACCOUNT_IAM_ROLE_ARN>"
                },
                "Action": "sts:AssumeRole"
            }
        ]
    }
    
Credentials for the Member Role Name

You need to provide credentials for the <ORGANIZATION_MEMBER_ACCOUNTS_ROLE_NAME> that is available in all scoped member accounts of the organization with access to discover and read registries within. This role will be assumed by the ORGANIZATION_MANAGEMENT_ROLE_ARN to impersonate each member account.

By default, the scanner uses the AWS-created administration role OrganizationAccountAccessRole. Using this role will work fine; however, if you need it, you can create a lower-permission role.

(Optional) Least-Privilege-Access Permissions

Create a role in all accounts and assign the following least-privilege-access permissions to it.

The permissions must be assigned to all roles in all accounts that will be scanned.

  • AmazonEC2ContainerRegistryReadOnly: AWS managed policy that Provides read-only access to Amazon EC2 Container Registry repositories. Assign the following least-privilege-access permissions to the member role name:

  • Custom permissions to allow region discovery:

    {
      "Version": "2012-10-17",
      "Statement": [
        {
          "Sid": "CustomPolicyAllowDescribeRegions",
          "Effect": "Allow",
          "Action": "EC2:DescribeRegions",
          "Resource": "*"
        }
      ]
    }
    
  • The member role must be assumable by the ORGANIZATION_MANAGEMENT_ROLE_ARN, so you must assign the following trust relationship to the new role:

    {
      "Version": "2012-10-17",
      "Statement": [
        {
          "Effect": "Allow",
          "Principal": {
            "AWS": "<ORGANIZATION_MANAGEMENT_ROLE_ARN>"
          },
          "Action": "sts:AssumeRole"
        }
      ]
    }
    

Also, the ORGANIZATION_MANAGEMENT_ROLE_ARN must be configured to allow assuming the ORGANIZATION_MEMBER_ACCOUNTS_ROLE_NAME you just created in all accounts. You must assign the following policy permission to the management role, which will allow it to assume the member role in all accounts:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "AllowManagementRoleToAssumeMember",
      "Effect": "Allow",
      "Action": "sts:AssumeRole",
      "Resource": "arn:aws:iam::*:role/<ORGANIZATION_MEMBER_ACCOUNTS_ROLE_NAME>"
    }
  ]
}

Optionally, if you want to test the role with the validation script of the next section you will need to add the following permissions:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "CustomPolicyAllowDescribeRegions",
      "Effect": "Allow",
      "Action": "iam:SimulatePrincipalPolicy",
      "Resource": "*"
    }
  ]
}
(Optional) Validate Setup with Test Script

Sysdig provides the test script aws-org-test-credentials.sh to check the created roles and permissions, along with validating the discoverables registries throughout the organization.

Launch the script from a shell. Use the AWS authentication credentials for the member account where the Registry Scanner will be deployed.

Note: If you’re running it with some other credentials from the member account, remember that a trust relationship must exist with the Management Account Role. See Assign the Trust Relationship under Credentials for the Management Role.

./aws-org-test-credentials.sh <ORGANIZATION_MANAGEMENT_ROLE_ARN> <ORGANIZATION_MEMBER_ACCOUNTS_ROLE_NAME>

Sample output:

> using *** as member accountID
> Organization accountID: *** with discovered member account IDs
***
-------------------------
> Starting registry discovery and permission checkup

> Account ***
Checking AWS registry permissions
Discovering registries
...
-------------------------
> Discovered Registries
***.dkr.ecr.eu-central-1.amazonaws.com
***.dkr.ecr.eu-west-1.amazonaws.com
...

Install with Helm Chart

Use one of the following installation options depending on the credential format used:

Access Key/Secret Key

Suitable for testing or if your Kubernetes cluster is outside AWS.

<AWS_ACCESS_KEY_ID>, <AWS_SECRET_ACCESS_KEY> from the AWS user

$ helm upgrade --install registry-scanner sysdig/registry-scanner --version=1 \
--set config.secureBaseURL=<SYSDIG_SECURE_URL> \
--set config.secureAPIToken=<SYSDIG_SECURE_API_TOKEN> \
--set config.registryType=ecr \
--set config.aws.accessKeyId=<AWS_ACCESS_KEY_ID> \
--set config.aws.secretAccessKey=<AWS_SECRET_ACCESS_KEY> \
--set config.aws.managementAccountRoleARN=<ORGANIZATION_MANAGEMENT_ROLE_ARN> \
--set config.aws.memberAccountsRoleName=<ORGANIZATION_MEMBER_ACCOUNTS_ROLE_NAME>
Service Account

If you’re using AWS Elastic Kubernetes Service (EKS), you must assign an IAM role (SERVICE_ACCOUNT_IAM_ROLE_ARN) to a Service Account, with which the scanner will run.

$ helm upgrade --install registry-scanner sysdig/registry-scanner --version=1 \
--set config.secureBaseURL=<SYSDIG_SECURE_URL> \
--set config.secureAPIToken=<SYSDIG_SECURE_API_TOKEN> \
--set config.registryType=ecr \
--set serviceAccount.annotations.eks\\.amazonaws\\.com/role-arn=<SERVICE_ACCOUNT_IAM_ROLE_ARN> \
--set config.aws.managementAccountRoleARN=<ORGANIZATION_MANAGEMENT_ROLE_ARN> \
--set config.aws.memberAccountsRoleName=<ORGANIZATION_MEMBER_ACCOUNTS_ROLE_NAME>

Additional Helm Configuration

Limit Registry Accounts to Scan

Optionally, limit the scoped registry accounts using the allowListMemberAccountIDs Helm chart attribute. This approach is preferred for initial testing, when the ORGANIZATION_MEMBER_ACCOUNTS_ROLE_NAME may be provisioned only on select accounts and having the scanner check the rest of the member accounts in the organization would generate noisy error messages.

Set the attributes:

--set config.aws.allowListMemberAccountIDs[0]='<ORG_MEMBER_ACCOUNT_ID_1>' \
--set config.aws.allowListMemberAccountIDs[1]='<ORG_MEMBER_ACCOUNT_ID_2>' \

Limitations

AWS GovCloud

To install the registry Scanner in GovCloud Accounts (both single and organizational), follow the same instructions as above and make sure to specify the AWS GovCloud region with --set config.aws.region="us-gov-west-1" and to set the right AWS GobCloud ARN format.