Certificate Management with cert-manager

Complete guide for setting up and managing SSL/TLS certificates using cert-manager and Let’s Encrypt.

Overview
Prerequisites
Installation
AWS Route53 Setup
Certificate Issuers
Managing Certificates
Troubleshooting

Overview

This guide covers:

Installing cert-manager on Kubernetes
Configuring AWS Route53 for DNS validation
Creating staging and production certificate issuers
Generating wildcard certificates for your domain
Certificate rotation procedures

Prerequisites

Software Requirements

AWS Requirements

Required IAM Permissions:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "route53:GetChange",
        "route53:ChangeResourceRecordSets",
        "route53:ListResourceRecordSets",
        "route53:ListHostedZonesByName"
      ],
      "Resource": "*"
    }
  ]
}

Environment Variables

Network Requirements

Domain Requirements

Domain Ownership: You must own and control the domain
Route53 Hosting: Domain must be hosted in AWS Route53
DNS Resolution: Domain must resolve correctly from public internet
Wildcard Support: For wildcard certificates, ensure subdomain delegation works

Verification Checklist

Before proceeding, verify:

Kubernetes cluster is healthy: kubectl get nodes
AWS credentials work: aws route53 list-hosted-zones
Domain resolves: nslookup yourdomain.com
Environment variables set: echo $LETSENCRYPT_EMAIL
Internet connectivity from cluster to Let’s Encrypt
cert-manager CRDs can be installed (cluster admin access)

Installation

Deploy cert-manager

Install cert-manager using the official manifests:

kubectl apply -f https://github.com/jetstack/cert-manager/releases/download/v1.5.3/cert-manager.yaml

Verify installation:

kubectl get pods --namespace cert-manager
kubectl get crd | grep cert-manager

AWS Route53 Setup

IAM Role Configuration

Create an IAM policy with the following permissions:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": "route53:GetChange",
      "Resource": "arn:aws:route53:::change/*"
    },
    {
      "Effect": "Allow",
      "Action": [
        "route53:ChangeResourceRecordSets",
        "route53:ListResourceRecordSets"
      ],
      "Resource": "arn:aws:route53:::hostedzone/*"
    },
    {
      "Effect": "Allow",
      "Action": "route53:ListHostedZonesByName",
      "Resource": "*"
    }
  ]
}

Configure Credentials

Create the Route53 secret:

# Load environment variables first: source .envrc or direnv allow
kubectl create secret generic route53-secret \
  --from-literal=secret-access-key="${ROUTE53_SECRET_ACCESS_KEY}" \
  --namespace cert-manager

Verify secret creation:

kubectl get secret route53-secret --namespace cert-manager -ojsonpath={.data.secret-access-key} | base64 -d

Certificate Issuers

Staging Issuer (Testing)

Create a staging issuer for testing certificate generation:

apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: letsencrypt-staging
spec:
  acme:
    server: https://acme-staging-v02.api.letsencrypt.org/directory
    email: ${LETSENCRYPT_EMAIL}
    privateKeySecretRef:
      name: letsencrypt-staging
    solvers:
    - dns01:
        route53:
          region: us-east-1
          accessKeyID: ${AWS_ACCESS_KEY_ID}
          secretAccessKeySecretRef:
            name: route53-secret
            key: secret-access-key

Production Issuer

Create the production issuer for real certificates:

apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: letsencrypt-prod
spec:
  acme:
    server: https://acme-v02.api.letsencrypt.org/directory
    email: ${LETSENCRYPT_EMAIL}
    privateKeySecretRef:
      name: letsencrypt-prod
    solvers:
    - dns01:
        route53:
          region: us-east-1
          accessKeyID: ${AWS_ACCESS_KEY_ID}
          secretAccessKeySecretRef:
            name: route53-secret
            key: secret-access-key

Managing Certificates

Create Wildcard Certificate

Generate a wildcard certificate for your domain:

apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: prod-wildcard-certs
  namespace: cert-manager
spec:
  secretName: prod-wildcard-certs
  issuerRef:
    name: letsencrypt-prod
    kind: ClusterIssuer
  dnsNames:
  - "*.markalston.net"
  - "*.tkg.markalston.net"

Export Certificates

Extract the generated certificates:

# Export private key
kubectl get secrets prod-wildcard-certs -n cert-manager \
  -ojsonpath={.data.'tls\.key'} | base64 -d > prod.key

# Export certificate
kubectl get secrets prod-wildcard-certs -n cert-manager \
  -ojsonpath={.data.'tls\.crt'} | base64 -d > prod.crt

Certificate Rotation

To rotate certificates:

Delete the existing certificate secret
cert-manager will automatically regenerate it

kubectl delete secret prod-wildcard-certs -n cert-manager
# cert-manager will recreate the certificate

Verification

Check Certificate Status

kubectl get certificates -n cert-manager
kubectl describe certificate prod-wildcard-certs -n cert-manager

Test Certificate

openssl x509 -in prod.crt -text -noout

Troubleshooting

Common Issues

Issue: Certificate stuck in “Pending”

Check cert-manager logs:

kubectl logs -n cert-manager deployment/cert-manager

Issue: DNS validation failing

Verify Route53 permissions:

kubectl describe challenge -n cert-manager

Debug Commands

# Check all cert-manager resources
kubectl get Issuers,ClusterIssuers,Certificates,CertificateRequests,Orders,Challenges --all-namespaces

# View cert-manager events
kubectl get events -n cert-manager --sort-by='.lastTimestamp'

Best Practices

Always test with staging issuer first
Use wildcard certificates to minimize rate limits
Monitor certificate expiration dates
Automate certificate distribution to services

References

Last Updated: 2024 Part of the Homelab Documentation Series