Certificate Management with cert-manager

Complete guide for setting up and managing SSL/TLS certificates using cert-manager and Let’s Encrypt.

Table of Contents

• Overview
• Prerequisites
• Installation
• AWS Route53 Setup
• Certificate Issuers
• Managing Certificates
• Troubleshooting

Overview

This guide covers:

• Installing cert-manager on Kubernetes
• Configuring AWS Route53 for DNS validation
• Creating staging and production certificate issuers
• Generating wildcard certificates for your domain
• Certificate rotation procedures

Prerequisites

Software Requirements

Component	Version	Purpose	Installation Guide
Kubernetes	1.16+	Container orchestration platform	TKG Setup
kubectl	1.16+	Kubernetes command-line tool	kubectl install
Helm	3.x	Kubernetes package manager	Helm install

AWS Requirements

Resource	Purpose	Configuration Required
AWS Account	Route53 DNS management	Valid billing account
Route53 Hosted Zone	Domain DNS management	Your domain must be hosted in Route53
IAM User	API access for DNS validation	Policy with Route53 permissions
AWS Access Keys	Programmatic access	Access Key ID and Secret Access Key

Required IAM Permissions:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "route53:GetChange",
        "route53:ChangeResourceRecordSets",
        "route53:ListResourceRecordSets",
        "route53:ListHostedZonesByName"
      ],
      "Resource": "*"
    }
  ]
}

Environment Variables

Variable	Description	Example	Required
AWS_ACCESS_KEY_ID	AWS access key for Route53	AKIAIOSFODNN7EXAMPLE	Yes
ROUTE53_SECRET_ACCESS_KEY	Route53 secret key	wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY	Yes
LETSENCRYPT_EMAIL	Email for Let’s Encrypt notifications	admin@yourdomain.com	Yes

Network Requirements

Port	Protocol	Purpose	Source	Destination
443	HTTPS	ACME certificate validation	Let’s Encrypt	Kubernetes cluster
53	DNS	Route53 API access	Kubernetes cluster	AWS Route53
80	HTTP	ACME HTTP-01 challenge (optional)	Let’s Encrypt	Kubernetes cluster

Domain Requirements

• Domain Ownership: You must own and control the domain
• Route53 Hosting: Domain must be hosted in AWS Route53
• DNS Resolution: Domain must resolve correctly from public internet
• Wildcard Support: For wildcard certificates, ensure subdomain delegation works

Verification Checklist

Before proceeding, verify:

• Kubernetes cluster is healthy: kubectl get nodes
• AWS credentials work: aws route53 list-hosted-zones
• Domain resolves: nslookup yourdomain.com
• Environment variables set: echo $LETSENCRYPT_EMAIL
• Internet connectivity from cluster to Let’s Encrypt
• cert-manager CRDs can be installed (cluster admin access)

Installation

Deploy cert-manager

Install cert-manager using the official manifests:

kubectl apply -f https://github.com/jetstack/cert-manager/releases/download/v1.5.3/cert-manager.yaml

Verify installation:

kubectl get pods --namespace cert-manager
kubectl get crd | grep cert-manager

AWS Route53 Setup

IAM Role Configuration

Create an IAM policy with the following permissions:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": "route53:GetChange",
      "Resource": "arn:aws:route53:::change/*"
    },
    {
      "Effect": "Allow",
      "Action": [
        "route53:ChangeResourceRecordSets",
        "route53:ListResourceRecordSets"
      ],
      "Resource": "arn:aws:route53:::hostedzone/*"
    },
    {
      "Effect": "Allow",
      "Action": "route53:ListHostedZonesByName",
      "Resource": "*"
    }
  ]
}

Configure Credentials

Create the Route53 secret:

# Load environment variables first: source .envrc or direnv allow
kubectl create secret generic route53-secret \
  --from-literal=secret-access-key="${ROUTE53_SECRET_ACCESS_KEY}" \
  --namespace cert-manager

Verify secret creation:

kubectl get secret route53-secret --namespace cert-manager -ojsonpath={.data.secret-access-key} | base64 -d

Certificate Issuers

Staging Issuer (Testing)

Create a staging issuer for testing certificate generation:

apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: letsencrypt-staging
spec:
  acme:
    server: https://acme-staging-v02.api.letsencrypt.org/directory
    email: ${LETSENCRYPT_EMAIL}
    privateKeySecretRef:
      name: letsencrypt-staging
    solvers:
    - dns01:
        route53:
          region: us-east-1
          accessKeyID: ${AWS_ACCESS_KEY_ID}
          secretAccessKeySecretRef:
            name: route53-secret
            key: secret-access-key

Production Issuer

Create the production issuer for real certificates:

apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: letsencrypt-prod
spec:
  acme:
    server: https://acme-v02.api.letsencrypt.org/directory
    email: ${LETSENCRYPT_EMAIL}
    privateKeySecretRef:
      name: letsencrypt-prod
    solvers:
    - dns01:
        route53:
          region: us-east-1
          accessKeyID: ${AWS_ACCESS_KEY_ID}
          secretAccessKeySecretRef:
            name: route53-secret
            key: secret-access-key

Managing Certificates

Create Wildcard Certificate

Generate a wildcard certificate for your domain:

apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: prod-wildcard-certs
  namespace: cert-manager
spec:
  secretName: prod-wildcard-certs
  issuerRef:
    name: letsencrypt-prod
    kind: ClusterIssuer
  dnsNames:
  - "*.markalston.net"
  - "*.tkg.markalston.net"

Export Certificates

Extract the generated certificates:

# Export private key
kubectl get secrets prod-wildcard-certs -n cert-manager \
  -ojsonpath={.data.'tls\.key'} | base64 -d > prod.key

# Export certificate
kubectl get secrets prod-wildcard-certs -n cert-manager \
  -ojsonpath={.data.'tls\.crt'} | base64 -d > prod.crt

Certificate Rotation

To rotate certificates:

1. Delete the existing certificate secret
2. cert-manager will automatically regenerate it

kubectl delete secret prod-wildcard-certs -n cert-manager
# cert-manager will recreate the certificate

Verification

Check Certificate Status

kubectl get certificates -n cert-manager
kubectl describe certificate prod-wildcard-certs -n cert-manager

Test Certificate

openssl x509 -in prod.crt -text -noout

Troubleshooting

Common Issues

Issue: Certificate stuck in “Pending”

Check cert-manager logs:

kubectl logs -n cert-manager deployment/cert-manager

Issue: DNS validation failing

Verify Route53 permissions:

kubectl describe challenge -n cert-manager

Debug Commands

# Check all cert-manager resources
kubectl get Issuers,ClusterIssuers,Certificates,CertificateRequests,Orders,Challenges --all-namespaces

# View cert-manager events
kubectl get events -n cert-manager --sort-by='.lastTimestamp'

Best Practices

1. Always test with staging issuer first
2. Use wildcard certificates to minimize rate limits
3. Monitor certificate expiration dates
4. Automate certificate distribution to services

References

• cert-manager Documentation
• Let’s Encrypt Rate Limits
• AWS Route53 Documentation

---

Last Updated: 2024 Part of the Homelab Documentation Series