Ingest EKS Fargate metrics to AWS Container Insights using ADOT with Helm

The following link shows how to deploy the ADOT Collector to EKS Fargate that enables the collection of Container Insights metrics from EKS Fargate workloads and send them to CloudWatch.

This is deployed via a couple YAML files that require some editing to work for your cluster. This article will go through deploying the same ADOT collector via a Helm chart, as well some of the prerequisites required for ADOT to run.

Container Insights EKS Fargate | AWS Open Distro for OpenTelemetry

Prerequisites

It is assumed you have an EKS Fargate Cluster running.

The following resources are created with Terraform as an example, but they could be done through CLI or another method.

Cert Manager

You will need to install Cert Manager for ADOT to work. More info on why this is required.

Change the webhook port for it to work on Fargate, as the Kubelet listens on port 10250 by default and clashes with Cert Manager.

resource "helm_release" "cert_manager" {
  depends_on = [
    aws_eks_addon.coredns
  ]

  name             = "cert-manager"
  chart            = "cert-manager"
  create_namespace = true
  repository       = "https://charts.jetstack.io"
  version          = "1.11.0"
  namespace        = "cert-manager"

  set {
    name  = "startupapicheck.timeout"
    value = "5m"
  }
  set {
    name  = "installCRDs"
    value = "true"
  }
  set {
    name  = "webhook.securePort"
    value = "10260"
  }
}

AWS IAM Role for ADOT Service Account

You’ll also need to set up an AWS IAM role for the ADOT Service Account to use with the following:

• a trust policy that provides federated access to the AWS EKS cluster through OIDC (I have used the “terraform-aws-modules/eks/aws” Terraform module and am able to get the OIDC provider arn via module.eks.oidc_provider_arn)
• The managed policy CloudWatchAgentServerPolicy so that ADOT has permissions to send Container Insights to CloudWatch.

variable "service_account_namespace" {
  default = "fargate-container-insights"
}

variable "service_account_name" {
  default = "adot-collector"
}

data "aws_iam_policy_document" "adot_trust_policy" {
  statement {
    actions = ["sts:AssumeRoleWithWebIdentity"]
    principals {
      type        = "Federated"
      identifiers = [module.eks.oidc_provider_arn]
    }
    condition {
      variable = "${module.eks.oidc_provider}:sub"
      test     = "StringEquals"
      values   = ["system:serviceaccount:${var.service_account_namespace}:${var.service_account_name}"]
    }
  }
}

resource "aws_iam_role" "adot_service_acc" {

  name = "${var.env_name}-adot-role"

  assume_role_policy = data.aws_iam_policy_document.adot_trust_policy.json

  managed_policy_arns = [
    "arn:aws:iam::aws:policy/CloudWatchAgentServerPolicy",
  ]
}

output "adot_iam_role_arn" {
  value = aws_iam_role.adot_service_acc.arn
}

resource "kubernetes_namespace" "fargate_container_insights" {
  metadata {
    name = var.service_account_namespace
  }
}

The output adot_iam_role_arn will output the ARN of the IAM role which will be used in the ADOT Helm chart.

Installation of the ADOT Fargate Helm chart

Add the AWS Distro for OpenTelemetry Helm repository to Helm:

helm repo add adot-fargate https://github.com/cjh-cloud/adot-fargate  
helm repo update

Install the AWS Distro for OpenTelemetry Helm chart:
Change the cluster name, region to your own EKS cluster and IAM role ARN to the value of the adot_iam_role_arn Terraform output above.

helm install adot-fargate cjh-cloud/adot-fargate \
  --set clusterName=my-cluster \
  --set region=ap-southeast-2 \
  --set serviceAccount.iamRoleArn=arn:aws:iam::[ACCOUNT ID]:role/adot-role

OR Install via repo:

helm upgrade --install aws-distro-open-telemetry ./ \
  --set clusterName=my-cluster \
  --set region=ap-southeast-2 \
  --set serviceAccount.iamRoleArn=arn:aws:iam::[ACCOUNT ID]:role/adot-role

Verify that the AWS Distro for OpenTelemetry deployment is running:

kubectl get pods -n fargate-container-insights

The output should show a pod with the name adot-collector-0.
View "Container Insights" under AWS CloudWatch to view EKS Cluster metrics.

Performance Monitoring

Container Map

Thanks for reading, I hope you found this useful!