Optional Services

Running Guardium Insights with QSPM on EKS

Export the following vars.

These should be relevant to your environment, therefore clustername should be your clustername and same with your hostname. Below this is an example.

export clustername=gi-east
export region=us-east-1
export NAMESPACE=openshift-marketplace
export HOSTNAME=apps.gi.thinkforward.work

(Optional) Install the ALB ingress controller

Unlike NGINX Ingress, ALB generates a load balancer when a external service is created in the cluster. So DNS is not necessary. See example here.

Download an IAM policy for the AWS Load Balancer Controller that allows it to make calls to AWS APIs on your behalf.

As of this writing, the latest version of AWS Load Balancer Controller is v2.9.0

curl -O https://raw.githubusercontent.com/kubernetes-sigs/aws-load-balancer-controller/v2.9.0/docs/install/iam_policy.json

Create an IAM policy using the policy downloaded in the previous step.

aws iam create-policy \
    --policy-name AWSLoadBalancerControllerIAMPolicy \
    --tags '{"Key": "Product","Value": "Guardium"}' \
    --policy-document file://iam_policy.json

Should return

{
    "Policy": {
        "PolicyName": "AWSLoadBalancerControllerIAMPolicy",
        "PolicyId": "ANPA3WENOYSATHNEI5OIR",
        "Arn": "arn:aws:iam::<ACCOUNT ID>:policy/AWSLoadBalancerControllerIAMPolicy",
        "Path": "/",
        "DefaultVersionId": "v1",
        "AttachmentCount": 0,
        "PermissionsBoundaryUsageCount": 0,
        "IsAttachable": true,
        "CreateDate": "2024-09-30T21:08:03+00:00",
        "UpdateDate": "2024-09-30T21:08:03+00:00",
        "Tags": [
            {
                "Key": "Product",
                "Value": "Guardium"
            }
        ]
    }
}

Let’s export that policy arn as another env var

export alb_policy_arn=$(aws iam list-policies --query 'Policies[?PolicyName==`AWSLoadBalancerControllerIAMPolicy`].Arn' --output text)

Export the role name as a env var. We’re going to append the cluster name to the role name to help identify it in AWS and in case we have multiple clusters in the same account.

export alb_role_name=AWSLoadBalancerControllerRole-${clustername}

Create a Kubernetes service account named aws-load-balancer-controller in the kube-system namespace for the AWS Load Balancer Controller and annotate the Kubernetes service account with the name of the IAM role.

eksctl create iamserviceaccount \
    --cluster ${clustername} \
    --namespace kube-system \
    --name aws-load-balancer-controller \
    --role-name ${alb_role_name} \
    --attach-policy-arn ${alb_policy_arn} \
    --tags "Product=Guardium" \
    --approve \
    --region ${region}

Now let’s use helm to install the AWS Load Balancer Controller

Install the helm repo

helm repo add eks https://aws.github.io/eks-charts
helm repo update

Now install the ALB controller

helm install aws-load-balancer-controller eks/aws-load-balancer-controller \
  -n kube-system \
  --set clusterName=${clustername} \
  --set serviceAccount.create=false \
  --set serviceAccount.name=aws-load-balancer-controller 

Should return something like this

NAME: aws-load-balancer-controller
LAST DEPLOYED: Mon Sep 30 17:21:43 2024
NAMESPACE: kube-system
STATUS: deployed
REVISION: 1
TEST SUITE: None
NOTES:
AWS Load Balancer controller installed!

Verify the ingress class has been created

kubectl get ingressclass

should return

tsx {2} NAME CONTROLLER PARAMETERS AGE alb ingress.k8s.aws/alb <none> 79s nginx k8s.io/ingress-nginx <none> 35d

{/*

Use AWS CA (ACM) for certs (optional)

If you plan to use a private CA via AWS ACM instead of something like LetsEncrypt by default, follow these steps

The AWS PrivateCA Issuer plugin acts as an addon (see external cert configuration) to cert-manager that signs certificate requests using ACM Private CA.

Create the namespace

kubectl create namespace aws-pca-issuer

Add the helm repo

helm repo add awspca https://cert-manager.github.io/aws-privateca-issuer
helm repo update
helm install awspca/aws-privateca-issuer --generate-name --namespace aws-pca-issuer

Create the private CA in ACM

This is using an example from this helpful AWS link

Generating and installing the CA cert here

In our example we are using the subdomain apps.gi.thinkforward.work. This will not be the domain you will likely use.

Create the following file and populate it with appropriate values. This file is an EXAMPLE:

cat <<EOF > ca_config.txt
{
   "KeyAlgorithm":"RSA_2048",
   "SigningAlgorithm":"SHA256WITHRSA",
   "Subject":{
      "Country":"US",
      "Organization":"IBM",
      "OrganizationalUnit":"Client Engineering",
      "State":"MA",
      "Locality":"Cambridge",
      "CommonName":"$HOSTNAME"
   }
}
EOF

We’re going to use the default OCSP settings. Now create a revocation file

cat <<EOF > revoke_config.txt
{
   "OcspConfiguration":{
      "Enabled":true
   }
}
EOF

Finally, create the CA

aws acm-pca create-certificate-authority \
     --certificate-authority-configuration file://ca_config.txt \
     --revocation-configuration file://revoke_config.txt \
     --certificate-authority-type "ROOT" \
     --idempotency-token 01234567 \
     --region $region \
     --tags Key=Product,Value=Guardium

This should return the new CA’s arn.

{
    "CertificateAuthorityArn": "arn:aws:acm-pca:us-east-1:748107796891:certificate-authority/00c9d3eb-c837-4cef-917a-fd87ec608a04"
}

Let’s collect that arn into an env var

export ca_arn="arn:aws:acm-pca:us-east-1:748107796891:certificate-authority/00c9d3eb-c837-4cef-917a-fd87ec608a04"

Generate and install the CA cert

Generate a certificate signing request (CSR).

aws acm-pca get-certificate-authority-csr \
     --certificate-authority-arn $ca_arn \
     --output text \
     --region $region > ca.csr

Using the CSR from the previous step as the argument for the –csr parameter, issue the root certificate and export the arn as another env var.

We’re going to set the validity to 999 days.

export root_cert_arn=$(aws acm-pca issue-certificate \
     --certificate-authority-arn $ca_arn \
     --csr fileb://ca.csr \
     --signing-algorithm SHA256WITHRSA \
     --region $region \
     --template-arn arn:aws:acm-pca:::template/RootCACertificate/V1 \
     --validity Value=999,Type=DAYS \
     --output text)

Retrieve the cert and let’s stick it in cert.pem

aws acm-pca get-certificate \
    --certificate-authority-arn $ca_arn \
    --certificate-arn $root_cert_arn \
  --region $region \
    --output text > cert.pem

Finally import it into the CA

aws acm-pca import-certificate-authority-certificate \
    --region $region \
    --certificate-authority-arn $ca_arn \
    --certificate fileb://cert.pem

Verify the status of the CA

aws acm-pca describe-certificate-authority \
  --region $region \
    --certificate-authority-arn $ca_arn \
  --query "CertificateAuthority.Status" \
    --output text

This should return ACTIVE.

Create an ACM Private CA

Download the CA certificate using the following command.

aws acm-pca get-certificate-authority-certificate \
--certificate-authority-arn $ca_arn \
--region $region \
--output text > cacert.pem

Set EKS node permission for ACM Private CA

Add the IAM policy to your EKS NodeInstanceRole

Create the following file

cat <<EOF > EKS-CA-NodePerms-policy.json
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "awspcaissuer",
            "Action": [
                "acm-pca:DescribeCertificateAuthority",
                "acm-pca:GetCertificate",
                "acm-pca:IssueCertificate"
            ],
            "Effect": "Allow",
            "Resource": "$ca_arn"
        }
    ]
}
EOF

Create the policy in IAM. This needs to be specific for your cluster using the CA we created.

Export the following var:

export awscapolicy="AWSCAPolicy-${clustername}"

aws iam create-policy \
--policy-name ${awscapolicy} \
--policy-document file://EKS-CA-NodePerms-policy.json

Let’s retrieve the policy arn as an env var

export ca_policy_arn=$(aws iam list-policies --query 'Policies[?PolicyName==`'${awscapolicy}'`].Arn' --output text)

Let’s gather our NodeInstance role name

export node_instance_role=$(aws iam list-roles --region $region --query 'Roles[?starts_with(RoleName, `eksctl-'${clustername}'`)]|[?contains(RoleName, `NodeInstanceRole`)].RoleName' --output text)

Now let’s attach the policy to our EKS node role. In our case the role name is gi-eks-nodes-role. This may be different for you.

aws iam attach-role-policy \
--role-name "${node_instance_role}" \
--policy-arn "${ca_policy_arn}" \
--region "${region}"

Create an issuer

We’re going to create a namespace scoped issuer for now.

Create a yaml file for the issuer

cat <<EOF > awpca-issuer.yaml
apiVersion: awspca.cert-manager.io/v1beta1
kind: AWSPCAIssuer
metadata:
  name: acm-issuer
  namespace: $NAMESPACE
spec:
  arn: $ca_arn
  region: $region
EOF

Deploy the AWSPCAIssuer:

kubectl apply -f awpca-issuer.yaml

*/}

Community Cert Manager (optional)

Installing community Cert Manager

Install the helm repo

helm repo add jetstack https://charts.jetstack.io

Create the namespace

kubectl create ns cert-manager

Now install the Community Cert Manager

helm install cert-manager jetstack/cert-manager \
--namespace cert-manager \
--set installCRDs=true \
--version v1.16.1

Configure Security Constraints (Optional)

For some environments there might be security constraints applied to the EKS cluster. Follow these instructions to apply security constraints using OPA gatekeeper.

If the operating environment is a production or pre-production enterprise environment, then you can skip this section as it’s meant for development environments.

Install OPA Gatekeeper

Deploy OPA Gatekeeper using prebuilt docker images

By default, the violations audit runs every 60 seconds and will report a maximum of 20 audits in the constraint yaml. If you want to increase the number of violations shown on the constraint yaml to 500 (not recommended to increase further), update the gatekeeper.yaml below before applying and add the following argument to the audit deployment.

yaml {17} template: metadata: labels: control-plane: audit-controller gatekeeper.sh/operation: audit gatekeeper.sh/system: "yes" spec: automountServiceAccountToken: true containers: - args: - --operation=audit - --operation=status - --operation=mutation-status - --logtostderr - --disable-opa-builtin={http.send} - --disable-cert-rotation - --constraint-violations-limit=500

kubectl apply -f https://raw.githubusercontent.com/open-policy-agent/gatekeeper/release-3.17/deploy/gatekeeper.yaml

Check the pods in gatekeeper-system namespace

kubectl get pods -n gatekeeper-system

The output will be similar to:

NAME                                             READY   STATUS    RESTARTS   AGE
gatekeeper-audit-5bc9b59c57-9d9hc                1/1     Running   0          25s
gatekeeper-controller-manager-744cdc8556-hxf2n   1/1     Running   0          25s
gatekeeper-controller-manager-744cdc8556-jn42m   1/1     Running   0          25s
gatekeeper-controller-manager-744cdc8556-wwrb6   1/1     Running   0          25s

Install kustomize

To ease in the deployment of the templates and constraints, install kustomize.

The following script detects your OS and downloads the appropriate kustomize binary to your current working directory.

curl -s "https://raw.githubusercontent.com/kubernetes-sigs/kustomize/master/hack/install_kustomize.sh"  | bash

Apply Templates

There are 2 parts to OPA gatekeeper policy enforcment, the first part is the ConstraintTemplate. This is where the logic of the policy exists. The templates must be deployed prior to deploying the constraints.

If you haven’t yet, clone this github repository.

git clone https://github.com/ibm-client-engineering/engineering-journal-quantum-safe.git

Go to the opa library templates directory.

cd engineering-journal-quantum-safe/opa/library/templates

Apply the templates using kustomize

kustomize build . | kubectl apply -f -

Sample output

constrainttemplate.templates.gatekeeper.sh/k8sallowedrepos created
constrainttemplate.templates.gatekeeper.sh/k8sblockendpointeditdefaultrole created
constrainttemplate.templates.gatekeeper.sh/k8sblockloadbalancer created
constrainttemplate.templates.gatekeeper.sh/k8sblocknodeport created
constrainttemplate.templates.gatekeeper.sh/k8sblockwildcardingress created
constrainttemplate.templates.gatekeeper.sh/k8sdisallowanonymous created
constrainttemplate.templates.gatekeeper.sh/k8sdisallowedrepos created
constrainttemplate.templates.gatekeeper.sh/k8sdisallowedtags created
constrainttemplate.templates.gatekeeper.sh/k8sdisallowinteractivetty created
constrainttemplate.templates.gatekeeper.sh/k8sexternalips created
constrainttemplate.templates.gatekeeper.sh/k8shttpsonly created
constrainttemplate.templates.gatekeeper.sh/k8simagedigests created
constrainttemplate.templates.gatekeeper.sh/k8spoddisruptionbudget created
constrainttemplate.templates.gatekeeper.sh/k8spspautomountserviceaccounttokenpod created
constrainttemplate.templates.gatekeeper.sh/k8srequiredresources created
constrainttemplate.templates.gatekeeper.sh/k8sstorageclass created
constrainttemplate.templates.gatekeeper.sh/k8suniqueingresshost created
constrainttemplate.templates.gatekeeper.sh/k8suniqueserviceselector created

Apply Constraints

The 2nd part to OPA gatekeeper policy enforcment is the Constraint. The constraints are a specific application of a template. This instructs the OPA gatekeeper to constrain a resource according to specific parameters and applying one of the templates. For example, a constraint can specify that every namespace must have a specific label applied.

Part of a constraint definition is the enforcement action. By default, OPA gatekeeper will enforce constraints by denying the action. For the constraints here, we are using dryrun, which does not deny the resources that violate constraints, but rather just logs the violation. This allows us to see what resources are violating the constraints without the resources being denied.

Go to the opa library constraints directory.

cd engineering-journal-quantum-safe/opa/library/constraints

Apply the templates using kustomize

kustomize build . | kubectl apply -f -

Sample output

k8sallowedrepos.constraints.gatekeeper.sh/repo-is-amazonaws created
k8sallowedrepos.constraints.gatekeeper.sh/repo-is-openpolicyagent created
k8sblockendpointeditdefaultrole.constraints.gatekeeper.sh/block-endpoint-edit-default-role created
k8sblockloadbalancer.constraints.gatekeeper.sh/block-load-balancer created
k8sblocknodeport.constraints.gatekeeper.sh/block-node-port created
k8sblockwildcardingress.constraints.gatekeeper.sh/block-wildcard-ingress created
k8sdisallowanonymous.constraints.gatekeeper.sh/no-anonymous-no-authenticated created
k8sdisallowinteractivetty.constraints.gatekeeper.sh/no-interactive-tty-containers created
k8sdisallowedrepos.constraints.gatekeeper.sh/repo-must-not-be-k8s-gcr-io created
k8sdisallowedtags.constraints.gatekeeper.sh/container-image-must-not-have-latest-tag created
k8sexternalips.constraints.gatekeeper.sh/external-ips created
k8shttpsonly.constraints.gatekeeper.sh/ingress-https-only created
k8simagedigests.constraints.gatekeeper.sh/container-image-must-have-digest created
k8spspautomountserviceaccounttokenpod.constraints.gatekeeper.sh/psp-automount-serviceaccount-token-pod created
k8spoddisruptionbudget.constraints.gatekeeper.sh/pod-distruption-budget created
k8srequiredresources.constraints.gatekeeper.sh/container-must-have-limits-and-requests created
k8suniqueingresshost.constraints.gatekeeper.sh/unique-ingress-host created

List Violations

Use the following command to list the violations.

kubectl get constraints

Sample output

NAME                                                                                     ENFORCEMENT-ACTION   TOTAL-VIOLATIONS
k8srequiredresources.constraints.gatekeeper.sh/container-must-have-limits-and-requests   dryrun               4

NAME                                                                 ENFORCEMENT-ACTION   TOTAL-VIOLATIONS
k8suniqueingresshost.constraints.gatekeeper.sh/unique-ingress-host   dryrun               0

NAME                                                                                ENFORCEMENT-ACTION   TOTAL-VIOLATIONS
k8sdisallowinteractivetty.constraints.gatekeeper.sh/no-interactive-tty-containers   dryrun               0

NAME                                                        ENFORCEMENT-ACTION   TOTAL-VIOLATIONS
k8shttpsonly.constraints.gatekeeper.sh/ingress-https-only   dryrun               0

NAME                                                                                         ENFORCEMENT-ACTION   TOTAL-VIOLATIONS
k8sblockendpointeditdefaultrole.constraints.gatekeeper.sh/block-endpoint-edit-default-role   dryrun               0

NAME                                                                       ENFORCEMENT-ACTION   TOTAL-VIOLATIONS
k8sdisallowedrepos.constraints.gatekeeper.sh/repo-must-not-be-k8s-gcr-io   dryrun               0

NAME                                                                                   ENFORCEMENT-ACTION   TOTAL-VIOLATIONS
k8sdisallowedtags.constraints.gatekeeper.sh/container-image-must-not-have-latest-tag   dryrun               0

NAME                                                         ENFORCEMENT-ACTION   TOTAL-VIOLATIONS
k8sblocknodeport.constraints.gatekeeper.sh/block-node-port   dryrun               0

Violation Details

To see the details of the violations, you can look at the constraint yaml. For example, to see the container-must-have-limits-and-requests constraint violations, run the following command.

kubectl get k8srequiredresources container-must-have-limits-and-requests -o yaml

Near the end of the yaml output, you will see violations listed.

  violations:
  - enforcementAction: dryrun
    group: ""
    kind: Pod
    message: container <manager> does not have <{"cpu"}> limits defined
    name: gatekeeper-controller-manager-865cc64485-kgkkj
    namespace: gatekeeper-system
    version: v1
  - enforcementAction: dryrun
    group: ""
    kind: Pod
    message: container <manager> does not have <{"cpu"}> limits defined
    name: gatekeeper-controller-manager-865cc64485-h2x7d
    namespace: gatekeeper-system
    version: v1
  - enforcementAction: dryrun
    group: ""
    kind: Pod
    message: container <manager> does not have <{"cpu"}> limits defined
    name: gatekeeper-controller-manager-865cc64485-g4b72
    namespace: gatekeeper-system
    version: v1
  - enforcementAction: dryrun
    group: ""
    kind: Pod
    message: container <manager> does not have <{"cpu"}> limits defined
    name: gatekeeper-audit-c794d5f69-s2nmd
    namespace: gatekeeper-system
    version: v1

If you want a CSV of all of the violations (up to the constraints violation limit set at install), run the following command. Note that the header will be repeated.

kubectl get constraints -o json | yq -o=json eval '.items[].status.violations' | jq -r '(.[0] | keys_unsorted) as $keys | $keys, map([.[ $keys[] ]])[] | @csv' > violations.csv 2>/dev/null

Configure OPA Mutations

If you are deploying to an EKS environment that is airgapped and using a private registry, the following mutations would need to be configured for the environment. This is not applicable to an Openshift environment as that can use an ImageContentSourcePolicy

These mutations essentially work like an Openshift ImageContentSourcePolicy and rewrite the domain paths in pods.

Export the registry domain. This should be the url of your private registry. my.registry.io is an example.

export myprivatereg=my.registry.io

Apply the mutations

cat <<EOF | kubectl apply -f -
apiVersion: mutations.gatekeeper.sh/v1alpha1
kind: AssignImage
metadata:
  name: assign-container-domain
spec:
  applyTo:
  - groups: [ "" ]
    kinds: [ "Pod" ]
    versions: [ "v1" ]
  location: "spec.containers[name:*].image"
  parameters:
    assignDomain: "${myprivatereg}"
  match:
    source: "All"
    scope: Namespaced
    namespaces:
      - openshift-marketplace
      - ibm-cert-manager
    kinds:
    - apiGroups: [ "*" ]
      kinds: [ "Pod" ]
---
apiVersion: mutations.gatekeeper.sh/v1alpha1
kind: AssignImage
metadata:
  name: assign-initcontainer-domain
spec:
  applyTo:
  - groups: [ "" ]
    kinds: [ "Pod" ]
    versions: [ "v1" ]
  location: "spec.initContainers[name:*].image"
  parameters:
    assignDomain: "${myprivatereg}"
  match:
    source: "All"
    scope: Namespaced
    namespaces:
      - openshift-marketplace
      - ibm-cert-manager
    kinds:
    - apiGroups: [ "*" ]
      kinds: [ "Pod" ]
---
apiVersion: mutations.gatekeeper.sh/v1alpha1
kind: AssignImage
metadata:
  name: assign-ephemeralcontainer-domain
spec:
  applyTo:
  - groups: [ "" ]
    kinds: [ "Pod" ]
    versions: [ "v1" ]
  location: "spec.ephemeralContainers[name:*].image"
  parameters:
    assignDomain: "${myprivatereg}"
  match:
    source: "All"
    scope: Namespaced
    namespaces:
      - openshift-marketplace
      - ibm-cert-manager
    kinds:
    - apiGroups: [ "*" ]
      kinds: [ "Pod" ]
EOF