Kubernetes Cluster Setup

This guide covers the initial setup and configuration of Tanzu Kubernetes clusters in the homelab environment.

Table of Contents

Overview

This document describes how to set up a Tanzu Kubernetes Grid (TKG) cluster on vSphere with Tanzu, including:

  • Configuring trust for Harbor registry
  • Creating development clusters
  • Setting up RBAC and Pod Security Policies
  • Testing registry connectivity

Prerequisites

Environment Requirements

  • vSphere 7.0+ with Tanzu enabled
  • Supervisor cluster configured
  • kubectl CLI installed
  • Harbor registry deployed

Required Credentials

  • VCENTER_USERNAME: vSphere administrator account
  • VCENTER_PASSWORD: vSphere administrator password
  • Harbor CA certificate available

Cluster Configuration

Step 1: Configure Harbor CA Trust

Why This Step? Harbor uses a self-signed certificate in lab environments. Kubernetes clusters need to trust this CA to pull images from Harbor registry. Without this trust relationship, image pulls will fail with certificate errors.

Update the supervisor cluster to trust the Harbor CA certificate:

# This configures all TKG clusters to trust the Harbor registry
cat <<EOF | kubectl apply -f -
apiVersion: run.tanzu.vmware.com/v1alpha1
kind: TkgServiceConfiguration
metadata:
  name: tkg-service-configuration
spec:
  defaultCNI: antrea  # Why Antrea? Provides NSX-T integration and network policies
  trust:
    additionalTrustedCAs:
      - name: harbor-ca
        data: $(base64 ./ca.crt)  # Base64 encoding required for YAML embedding
EOF

What This Does:

  • Adds Harbor CA to all TKG cluster trust stores
  • Enables secure image pulls from Harbor registry
  • Applies to all future clusters created by this supervisor

Step 2: Create Development Cluster

Why a Separate Dev Cluster?

  • Isolation: Development workloads separated from production infrastructure
  • Resource Control: Dedicated resource allocation and limits
  • Security: Different RBAC policies and network policies
  • Flexibility: Ability to experiment without affecting other workloads

Apply the cluster configuration:

# Creates a new TKG workload cluster in the development namespace
kubectl apply -f tanzu/dev-cluster.yaml

What This Does:

  • Provisions Kubernetes control plane VMs
  • Creates worker node VMs with specified resources
  • Configures networking and storage classes
  • Sets up monitoring and logging integration

Step 3: Login to vSphere

Why This Login Process? vSphere with Tanzu uses a unique authentication model where you authenticate to the supervisor cluster, then access workload clusters through context switching.

Connect to the Tanzu cluster:

kubectl vsphere login --server ${VCENTER_SERVER} \
  --insecure-skip-tls-verify \
  --vsphere-username "${VCENTER_USERNAME}" \
  --tanzu-kubernetes-cluster-name dev-cluster \
  --tanzu-kubernetes-cluster-namespace development

What This Does:

  • Authenticates against vCenter SSO
  • Downloads kubeconfig for the workload cluster
  • Sets up kubectl context for cluster access
  • Enables namespace-scoped cluster management

Step 4: Switch Context

Why Context Switching? kubectl can manage multiple clusters simultaneously. Context switching ensures commands target the correct cluster, preventing accidental operations on the wrong environment.

Set the kubectl context to the new cluster:

kubectl config use-context dev-cluster

Verify the Context:

kubectl config current-context
kubectl cluster-info

RBAC Setup

Configure Pod Security Policies

Why Pod Security Policies? Pod Security Policies (PSPs) control security-sensitive aspects of pod specifications. They define what security contexts pods can run with, preventing privilege escalation attacks and ensuring compliance with security standards.

VMware Security Policies:

  • vmware-system-privileged: Allows privileged containers (for system workloads)
  • vmware-system-restricted: Enforces security restrictions (for application workloads)

Create necessary role bindings for Pod Security Policies:

# Default namespace privileged access
# Why: Development environments often need privileged containers for testing
kubectl create rolebinding rolebinding-default-privileged-sa-ns_default \
    --namespace=default \
    --clusterrole=psp:vmware-system-privileged \
    --group=system:serviceaccounts

# Cluster-wide privileged access for service accounts
# Why: System components (Contour, cert-manager) need privileged access
kubectl create clusterrolebinding clusterrolebinding-privileged-sa \
    --clusterrole=psp:vmware-system-privileged \
    --group=system:serviceaccounts

# Restricted access for service accounts
# Why: Application workloads should run with minimal privileges
kubectl create rolebinding psp:serviceaccounts \
    --clusterrole=psp:vmware-system-restricted \
    --group=system:serviceaccounts

Security Note: In production, use more granular RBAC with specific service accounts rather than broad group assignments.

Verification

Test Registry Access

Deploy a test pod to verify Harbor registry access:

kubectl run test-pod --image=harbor.tkg.markalston.net/library/nginx:latest
kubectl get pods
kubectl delete pod test-pod

Check Cluster Status

Verify cluster health:

kubectl get nodes
kubectl get pods -A

Troubleshooting

Common Issues

Issue: Cannot login to vSphere

Solution: Verify vCenter credentials and network connectivity:

ping 192.168.24.66
# Check credentials are set
echo $VCENTER_USERNAME

Issue: Harbor images fail to pull

Solution: Verify CA trust configuration:

kubectl get tkgserviceconfiguration -o yaml
# Check if Harbor CA is properly configured

Debug Commands

Enable verbose logging:

kubectl vsphere login --server 192.168.24.66 --verbose=9

References

Last Updated: 2024 Part of the Homelab Documentation Series

This project is for educational and home lab purposes.