MS-A2 NVMe Memory Tiering Configuration Guide
This guide covers the complete setup of NVMe memory tiering on MINISFORUM MS-A2 hosts with AMD Ryzen 9 9955HX processors.
Overview
Hardware Configuration:
- MS-A2-01: 192.168.10.12 (esxi-ms-a2-01.lab.markalston.net)
- MS-A2-02: 192.168.10.13 (esxi-ms-a2-02.lab.markalston.net)
Storage Layout per Host:
- Samsung 980 500GB: Boot drive (ESXi installation)
- Samsung 990 PRO 4TB: NVMe memory tiering device
- WD_BLACK SN850X 4TB: VM storage datastore
- 128GB DDR5: Base system memory
Memory Tiering Benefits
With NVMe memory tiering enabled:
- Base Memory: 128GB DDR5
- NVMe Memory Tier: ~512GB (400% of base memory)
- Total Effective Memory: ~640GB per host
- Performance: Near-DRAM speeds for frequently accessed memory pages
Storage Configuration
Step 1: Verify Hardware Detection
# SSH to each host and verify all NVMe devices are detected
ssh root@192.168.10.12 # MS-A2-01
ssh root@192.168.10.13 # MS-A2-02
# List all NVMe devices
esxcli storage core device list | grep -E 'Display Name|Size'
Expected Output:
- Samsung_SSD_980_500GB (476,940 MB)
- Samsung_SSD_990_PRO_4TB (3,815,447 MB)
- WD_BLACK_SN850X_4000GB (3,815,447 MB)
Step 2: Create VM Storage Datastores
Configure WD_BLACK SN850X drives for VM storage:
# Check current partition table
partedUtil getptbl /vmfs/devices/disks/t10.NVMe____WD_BLACK_SN850X_4000GB__________________[SERIAL]
# Create GPT partition table with VMFS partition
partedUtil setptbl /vmfs/devices/disks/t10.NVMe____WD_BLACK_SN850X_4000GB__________________[SERIAL] gpt "1 2048 7814037068 AA31E02A400F11DB9590000C2911D1B8 0"
# Create VMFS6 filesystem
vmkfstools -C vmfs6 -S '[DATASTORE_NAME]' /vmfs/devices/disks/t10.NVMe____WD_BLACK_SN850X_4000GB__________________[SERIAL]:1
# Verify datastore creation
esxcli storage filesystem list | grep wd-sn850x
Device Names:
- MS-A2-01:
t10.NVMe____WD_BLACK_SN850X_4000GB__________________E92917428B441B00 - MS-A2-02:
t10.NVMe____WD_BLACK_SN850X_4000GB__________________3C2917428B441B00
Datastore Names:
- MS-A2-01:
esxi-ms-a2-01-wd-sn850x - MS-A2-02:
esxi-ms-a2-02-wd-sn850x
NVMe Memory Tiering Configuration
Step 3: Configure Memory Tiering
Run on each MS-A2 host:
# Enable memory tiering kernel setting
esxcli system settings kernel set -s MemoryTiering -v TRUE
# Create tier device using Samsung 990 PRO
esxcli system tierdevice create -d /vmfs/devices/disks/t10.NVMe____Samsung_SSD_990_PRO_4TB_________________[SERIAL]
# Set NVMe percentage to 400% (4x base memory = ~512GB tier)
esxcli system settings advanced set -o /Mem/TierNvmePct -i 400
# Reboot to activate memory tiering
reboot
Samsung 990 PRO Device Names:
- MS-A2-01:
t10.NVMe____Samsung_SSD_990_PRO_4TB_________________72A9415145382500 - MS-A2-02:
t10.NVMe____Samsung_SSD_990_PRO_4TB_________________84A9415145382500
Step 4: Verify Memory Tiering Configuration
After reboot, verify the configuration:
# Check tier device status
esxcli system tierdevice list
# Verify memory tiering settings
esxcli system settings kernel list -o MemoryTiering
esxcli system settings advanced list -o /Mem/TierNvmePct
Expected Results:
# Tier device should show full Samsung 990 PRO allocation
DeviceName PartitionId StartSector EndSector
-------------------------------------------------------------------- ----------- ----------- ---------
t10.NVMe____Samsung_SSD_990_PRO_4TB_________________[SERIAL] 1 2048 7814037134
# Memory tiering should be active
MemoryTiering Bool TRUE TRUE FALSE Enables Memory Tiering
# NVMe percentage should be 400%
Int Value: 400
Datastore Management
Step 5: Rename Boot Datastores
Use vCenter Web UI for consistent naming:
- Access vCenter: https://vcsa.markalston.net
- Navigate: Storage → Datastores
- Rename Boot Datastores:
datastore1→esxi-ms-a2-01-boot(MS-A2-01)datastore1 (1)→esxi-ms-a2-02-boot(MS-A2-02)
Final Datastore Configuration
MS-A2-01 (192.168.10.12):
esxi-ms-a2-01-boot- Samsung 980 500GB (Boot disk)esxi-ms-a2-01-wd-sn850x- WD_BLACK 4TB (VM storage)OSDATA-[UUID]- ESXi system partition
MS-A2-02 (192.168.10.13):
esxi-ms-a2-02-boot- Samsung 980 500GB (Boot disk)esxi-ms-a2-02-wd-sn850x- WD_BLACK 4TB (VM storage)OSDATA-[UUID]- ESXi system partition
Verification Commands
Memory and Storage Status
# Check total memory (should show ~128GB base)
esxcfg-info -F | grep "Physical Mem"
# Verify tier device allocation
esxcli system tierdevice list
# Check all datastores
esxcli storage filesystem list | grep VMFS
# Monitor memory tiering performance
esxtop -b -n 1 | grep -E "MEM|TIER"
Performance Monitoring
# Check NVMe device performance
esxcli storage core device stats get -d [DEVICE_NAME]
# Monitor memory tiering activity
vsish -e get /memory/comprehensive | grep -i tier
Troubleshooting
Common Issues
- Memory Tiering Not Active
- Verify kernel setting:
esxcli system settings kernel list -o MemoryTiering - Check reboot completed: Runtime should show
TRUE
- Verify kernel setting:
- Tier Device Creation Failed
- Ensure Samsung 990 PRO is not already partitioned
- Verify device path is correct
- Check for existing tier devices:
esxcli system tierdevice list
- Datastore Rename Failed
- Use vCenter Web UI instead of CLI
- Ensure no VMs are running on the datastore
- Check vCenter connection status
Recovery Commands
# Remove tier device if needed
esxcli system tierdevice remove -d [DEVICE_NAME]
# Reset memory tiering
esxcli system settings kernel set -s MemoryTiering -v FALSE
esxcli system settings advanced set -o /Mem/TierNvmePct -i 25
Performance Impact
Benefits:
- 5x increase in effective memory capacity (128GB → 640GB)
- Reduced memory pressure for large VMs
- Improved performance for memory-intensive workloads
- Optimal for VCF management domain deployment
Considerations:
- Samsung 990 PRO dedicated to memory tiering (not available for VM storage)
- Slight latency increase for cold memory pages
- Requires ESXi 8.0+ for optimal performance
References
- William Lam: NVMe Memory Tiering
- VMware ESXi 8.0 NVMe Memory Tiering Documentation
- MS-A2 Hardware Installation Guide:
/docs/guides/ms-a2/hardware-installation-guide.md
Configuration completed: 2025-01-15 Tested on: ESXi 8.0.3, MINISFORUM MS-A2 with AMD Ryzen 9 9955HX