Alternative Memory Tiering Configurations for MS-A2 Hosts

Current Situation Analysis

Hardware per MS-A2 Host:

128GB DDR5 RAM (physical memory)
Samsung 980 500GB (boot drive with ESXi partitions)
Samsung 990 PRO 4TB (currently used by vSAN capacity tier)
WD_BLACK SN850X 4TB (currently used by vSAN cache tier)

Current vSAN Configuration:

Cache Tier: WD_BLACK SN850X (full device)
Capacity Tier: Samsung 990 PRO (partitioned by vSAN)
Total vSAN Storage: 3.64TB usable

Option 1: Shared Samsung 990 PRO (Advanced)

Concept: Use remaining space on Samsung 990 PRO for memory tiering alongside vSAN

Technical Approach:

# Check current vSAN partition layout
partedUtil getptbl /vmfs/devices/disks/t10.NVMe____Samsung_SSD_990_PRO_4TB_________________72A9415145382500

# Create additional partition for memory tiering (if space available)
# This requires careful calculation to avoid vSAN conflicts

Pros:

Maximum hardware utilization
Achieves original Option A goals (640GB effective memory + vSAN)

Cons:

High Risk: Could corrupt vSAN datastore
Requires precise partition management
Unsupported configuration
Potential performance conflicts

Recommendation: ⚠️ Not recommended due to risk

Option 2: Boot Drive Memory Tiering (Conservative)

Concept: Use unused space on Samsung 980 500GB boot drives

Current Boot Drive Usage:

Samsung 980 500GB partitions:
- :1 = 100MB (UEFI boot)
- :5 = 4GB (Boot bank A)  
- :6 = 4GB (Boot bank B)
- :7 = 120GB (VMware Tools, logs)
- :8 = ~337GB (Available space)

Implementation:

# Create memory tiering partition from available space
partedUtil setptbl /vmfs/devices/disks/t10.NVMe____Samsung_SSD_980_500GB___________________7F17A051D3382500 \
  gpt "1 2048 206847 C12A7328F81F11D2BA4B00A0C93EC93B 0" \
      "5 206848 8595455 FC63DAF8-8483-4772-8E79-3D69D8477DE4 0" \
      "6 8595456 16984063 FC63DAF8-8483-4772-8E79-3D69D8477DE4 0" \
      "7 16984064 268435455 FC63DAF8-8483-4772-8E79-3D69D8477DE4 0" \
      "8 268435456 838860800 FC63DAF8-8483-4772-8E79-3D69D8477DE4 0" \
      "9 838860801 976773134 B3676DDDA38A4CD6B970718D7F873811 0"

# Enable memory tiering on new partition
esxcli system settings kernel set -s MemoryTiering -v TRUE
esxcli system tierdevice create -d /vmfs/devices/disks/t10.NVMe____Samsung_SSD_980_500GB___________________7F17A051D3382500:9
esxcli system settings advanced set -o /Mem/TierNvmePct -i 100

Expected Results:

Memory Tiering Space: ~68GB per host
Effective Memory: ~196GB per host (128GB DDR5 + 68GB NVMe)
vSAN: Unchanged (3.64TB)

Pros:

✅ Safe - doesn’t interfere with vSAN
✅ Uses otherwise unused storage space
✅ Provides meaningful memory expansion
✅ Low risk of conflicts

Cons:

Limited to 68GB tier space (not the full 512GB originally planned)
Boot drive may have different performance characteristics

Option 3: Physical Memory Upgrade (Hardware)

Concept: Expand physical DDR5 RAM instead of NVMe tiering

Current Configuration:

2x 64GB DDR5 modules = 128GB total

Upgrade Options:

Option 3A: Replace with 2x 96GB modules = 192GB total (+64GB)
Option 3B: Replace with 2x 128GB modules = 256GB total (+128GB)

Cost Analysis:

2x 96GB DDR5 modules: ~$800-1000
2x 128GB DDR5 modules: ~$1200-1500

Pros:

✅ True physical memory (faster than NVMe tiering)
✅ No storage complexity or conflicts
✅ Guaranteed compatibility
✅ Better performance than NVMe tiering

Cons:

Higher cost than using existing NVMe storage
Still limited compared to 512GB NVMe tiering potential

Option 4: vSAN Architecture Change (Radical)

Concept: Redesign vSAN to free up Samsung 990 PRO for memory tiering

Approach 4A - WD_BLACK Only vSAN:

# Remove Samsung 990 PRO from vSAN
# Reconfigure vSAN to use only WD_BLACK drives in single-tier mode
# Use Samsung 990 PRO purely for memory tiering

Expected Results:

vSAN Storage: ~7.28TB raw from WD_BLACK only (single-tier)
Memory Tiering: Full Samsung 990 PRO = 512GB tier per host
Effective Memory: 640GB per host (128GB DDR5 + 512GB NVMe)

Approach 4B - External Storage:

# Use Synology iSCSI for VM storage instead of vSAN
# Free up both NVMe drives for other uses
# Samsung 990 PRO for memory tiering
# WD_BLACK for high-performance local storage

Pros:

✅ Achieves original Option A goals
✅ Maximum NVMe memory tiering benefit
✅ Flexible storage architecture

Cons:

Major reconfiguration required
Loss of vSAN advanced features
Potential performance trade-offs

Option 5: Hybrid Approach (Balanced)

Concept: Combine multiple strategies for optimal balance

Configuration:

Boot Drive Memory Tiering: 68GB tier space from Samsung 980
Physical RAM Upgrade: Add 64-128GB DDR5
Keep Current vSAN: Maintain 3.64TB high-performance storage

Expected Results:

Physical Memory: 192-256GB DDR5
NVMe Memory Tier: +68GB from boot drives
Total Effective Memory: 260-324GB per host
vSAN Storage: 3.64TB unchanged

Implementation Phases:

Phase 1: Implement boot drive memory tiering (immediate, low-cost)
Phase 2: Physical RAM upgrade (future, when budget allows)

Recommendations by Use Case

For Maximum Memory (Lab/Development)

Recommended: Option 4A (vSAN redesign + full Samsung 990 PRO tiering)
Result: 640GB effective memory per host

For Production Stability

Recommended: Option 2 (Boot drive tiering) + Option 3A (RAM upgrade)
Result: 260GB effective memory, stable vSAN

For Budget-Conscious

Recommended: Option 2 (Boot drive memory tiering only)
Result: 196GB effective memory, no additional cost

For High-Performance Storage Priority

Recommended: Keep current configuration + Option 3B (RAM upgrade)
Result: 256GB physical memory, optimal vSAN performance

Implementation Safety Guidelines

Always backup vSAN configuration before making changes
Test memory tiering on one host first
Monitor performance after any changes
Keep rescue/recovery procedures ready
Document all partition changes for future reference

Next Steps Decision Matrix

Priority	Memory Need	Risk Tolerance	Recommended Option
Max Memory	>500GB	High	Option 4A
Balanced	200-300GB	Medium	Option 2 + 3A
Safe	150-200GB	Low	Option 2 only
Performance	Any	Low	Option 3B

Choose the option that best matches your priorities for this homelab environment.