10G Network Upgrade Strategy

Comprehensive plan for implementing 10G networking infrastructure in the homelab environment.

Table of Contents

• Overview
• Current Network Analysis
• 10G Infrastructure Design
• Shopping List
• Implementation Plan
• Configuration Guide
• Performance Optimization
• Troubleshooting
• References

Overview

This document outlines the strategy for upgrading the homelab network infrastructure to support 10G connectivity for high-performance workloads including Tanzu Kubernetes Grid (TKG) and Tanzu Application Services (TAS).

Key Objectives

• 10G backbone: High-speed connectivity between compute nodes
• Storage acceleration: 10G connection to Synology NAS
• Future-ready: Support for expanded compute infrastructure
• Cost-effective: Focus investment where it provides maximum benefit

Current Network Analysis

Existing Infrastructure

Internet → Fiber → Garage (UXG-Lite + US-8) → 1G Uplink → Office Infrastructure
                                                              ↓
                                                    USW-16-PoE + US-8
                                                              ↓
                                                    3x NUCs + Synology NAS

Current Bottlenecks

• 1G switching: Limits inter-node communication
• Storage bandwidth: NAS limited to 1G throughput
• Multiple hops: Daisy-chained switches create latency
• No VLAN segmentation: Single broadcast domain

Traffic Analysis

What needs 10G:

• VM-to-VM communication (East-West traffic)
• vMotion operations
• Storage access from compute nodes
• Container image transfers
• NSX-T overlay traffic

What can stay 1G:

• Internet access (uplink to garage)
• Management traffic
• WiFi device connectivity
• External service access

10G Infrastructure Design

Recommended Architecture

Office Lab Infrastructure:
┌─────────────────────────────────────┐
│              USW-Aggregation        │ (10G Core)
│         8x SFP+ 10G Ports           │
├─────────────────────────────────────┤
│ Port 1: MS-A2 #1    │ Port 5: Spare │
│ Port 2: MS-A2 #2    │ Port 6: Spare │
│ Port 3: MS-A2 #3    │ Port 7: Spare │
│ Port 4: Synology    │ Port 8: 1G Up │
└─────────────────────────────────────┘
         │                      │
         │ 10G DAC            │ 1G to Garage
         ▼                      ▼
   [Compute Nodes]        [Existing Network]

VLAN Design for Tanzu

VLAN	Network	Purpose	Bandwidth
10	192.168.10.0/24	Management	1G
20	192.168.20.0/24	vMotion/Transport	10G
30	192.168.30.0/24	Storage	10G
100	192.168.100.0/24	TKG Workload	10G
110	192.168.110.0/24	TKG Frontend	1G

Benefits of This Design

• Local 10G traffic: 95% of lab traffic stays within 10G infrastructure
• 1G uplink sufficient: Internet and management traffic handled efficiently
• Cost optimization: No need for expensive fiber run to garage
• Scalable: Easy to add more 10G devices

Shopping List

Core 10G Switch

Ubiquiti USW-Aggregation

• Specs: 8x 10G SFP+ ports, fanless design
• Price: $300-400
• Vendor: Amazon, CDW

10G Connectivity Cables

Direct Attach Copper (DAC) Cables

• 3x 0.5m (1.6ft): For MS-A2 connections - $20 each
• 1x 1m (3.3ft): For NAS connection - $25
• 1x 2m (6.6ft): Spare/future use - $30
• Total: ~$115
• Vendor: 10GTek on Amazon

1G Uplink Module

1000Base-T SFP Module

• Purpose: Connect to existing 1G infrastructure
• Price: $20-30
• Vendor: FS.com SFP-1000Base-T

Synology 10G Upgrade Options

Option 1: SFP+ to RJ45 Adapter (Budget)

• 10GTek ASF-10G-T: SFP+ to RJ45 module - $60-80
• Benefits: Use existing NAS without modification
• Limitation: NAS internal bandwidth still 1G

Option 2: Synology 10G Card (Premium)

• Synology E10G18-T1: Native 10G for DS918+ - $120-150
• Benefits: True 10G at NAS level
• Requirements: SFP+ to RJ45 adapter in switch ($60)

Total Investment

Budget Build: $425

• USW-Aggregation: $350
• 3x DAC cables: $60
• 1G SFP module: $25

Premium Build: $750

• USW-Aggregation: $350
• 5x DAC cables: $115
• Synology 10G card: $130
• 10G-T adapter: $80
• 1G SFP module: $25
• Spare modules: $50

Implementation Plan

Phase 1: Infrastructure Deployment

1. Install USW-Aggregation in rack
2. Configure basic settings in UniFi Controller
3. Test connectivity with existing equipment

Phase 2: 10G Connections

1. Connect first MS-A2 via DAC cable
2. Validate 10G speeds between nodes
3. Connect remaining devices incrementally

Phase 3: Network Optimization

1. Configure VLANs for workload separation
2. Set up trunk ports for multi-VLAN devices
3. Optimize switch settings for low latency

Migration Steps

1. Document current network configuration
2. Plan maintenance window for switch replacement
3. Configure new switch offline first
4. Swap connections during maintenance window
5. Verify all services operational

Configuration Guide

UniFi Controller Setup

Initial Switch Configuration

1. Adopt USW-Aggregation in UniFi Controller
2. Update firmware to latest version
3. Configure switch name and location

VLAN Configuration

# Create VLANs in UniFi Controller
Settings → Networks → Create New

VLAN 10 - Management:
- Name: "Management"
- VLAN ID: 10
- Network: 192.168.10.0/24
- Gateway: 192.168.10.1

VLAN 20 - vMotion:
- Name: "vMotion"
- VLAN ID: 20
- Network: 192.168.20.0/24
- Gateway: 192.168.20.1

VLAN 30 - Storage:
- Name: "Storage"
- VLAN ID: 30
- Network: 192.168.30.0/24
- Gateway: 192.168.30.1

Port Profile Configuration

# Create port profiles for different device types

MS-A2 Profile (Trunk):
- Native VLAN: 10 (Management)
- Tagged VLANs: 20, 30, 100
- Profile Name: "MS-A2-Trunk"

Storage Profile:
- Native VLAN: 30 (Storage)
- Tagged VLANs: 10
- Profile Name: "Storage"

Uplink Profile:
- All VLANs trunked
- Profile Name: "Uplink-All"

ESXi Network Configuration

Virtual Switch Setup

# Configure distributed virtual switches for VLANs
vSphere Client → Networking → Distributed Switches

Create dvSwitch:
- Name: "dvSwitch-10G"
- Uplinks: 2 (for redundancy)
- Port Groups:
  - Management (VLAN 10)
  - vMotion (VLAN 20)
  - Storage (VLAN 30)
  - Workload (VLAN 100)

Host Network Configuration

# Configure ESXi hosts for 10G networking
ESXi Host → Networking → Virtual Switches

Physical Adapters:
- vmnic0: 10G SFP+ (primary)
- vmnic1: 1G ethernet (backup)

Network Assignments:
- Management: vmnic1 (1G)
- vMotion: vmnic0 (10G, VLAN 20)
- Storage: vmnic0 (10G, VLAN 30)

Performance Optimization

Switch Optimization

# Enable jumbo frames for storage traffic
UniFi Controller → Devices → USW-Aggregation
Services → Enable Jumbo Frame (9000 MTU)

# Configure flow control for storage
Advanced → Flow Control: Enable

Network Tuning

# ESXi network optimizations
esxcli system settings advanced set -o /Net/TcpipHeapMax -i 1576960
esxcli system settings advanced set -o /Net/TcpipHeapSize -i 32
esxcli system settings advanced set -o /Disk/QFullSampleSize -i 32

Performance Testing

# Test 10G throughput between hosts
iperf3 -s  # On target host
iperf3 -c <target-ip> -t 60 -P 4  # On source host

# Expected results:
# 10G DAC: 9.4+ Gbps
# Storage: 8+ Gbps sustained

Troubleshooting

Common Issues

10G Link Not Establishing

Symptoms:

• Port shows down/disabled
• No link light on DAC cable

Solutions:

# Check cable compatibility
- Verify SFP+ vs SFP distinction
- Test with known good cable
- Check cable length limits (5m max for DAC)

# Verify port configuration
UniFi Controller → Devices → Ports
- Check port is enabled
- Verify no speed restrictions

Poor 10G Performance

Symptoms:

• Speed tests below 5 Gbps
• High latency between nodes

Solutions:

# Check for bottlenecks
- Verify jumbo frames enabled
- Check for packet loss
- Monitor CPU utilization

# Network optimization
- Enable flow control
- Adjust network buffers
- Check VLAN configuration

VLAN Connectivity Issues

Symptoms:

• Devices can’t communicate across VLANs
• No internet access from VLANs

Solutions:

# Verify VLAN configuration
- Check native VLAN settings
- Verify trunk port configuration
- Test with packet capture

# Gateway configuration
- Verify VLAN interfaces on gateway
- Check routing table
- Validate firewall rules

Diagnostic Commands

# UniFi Switch CLI
ssh admin@<switch-ip>
show interfaces
show vlan
show mac-address-table

# ESXi network diagnostics
esxcli network ip interface list
esxcli network vswitch standard list
vmkping -I vmk1 <target-ip>  # Test specific VMkernel

Performance Expectations

Throughput Targets

• Node-to-Node: 9+ Gbps
• Node-to-Storage: 8+ Gbps sustained
• Latency: <0.1ms between local nodes
• Storage IOPS: 80,000+ (with NVMe cache)

Real-World Usage

• VM Migration: 2-3x faster than 1G
• Container pulls: Near-instant for cached images
• Database operations: Significant improvement for I/O intensive workloads
• Build processes: Faster artifact transfers

Future Expansion

Additional 10G Devices

• Second NAS: Easy addition with spare ports
• Network appliances: Firewall, load balancer
• Development workstations: Direct 10G connection

Uplink Upgrade Considerations

• When needed: Heavy external traffic, cloud backup
• Requirements: Garage switch upgrade, fiber run
• Investment: Additional $400-500

References

Internal Documentation

• Network Architecture
• Hardware Components
• 2025 Upgrade Plan

External Resources

• Ubiquiti USW-Aggregation Datasheet
• 10GTek DAC Cable Compatibility
• VMware 10G Networking Best Practices

---

Last Updated: 2025-01-11 Maintained by: Mark Alston