NVMe Memory Tiering Quick Reference

The new NVMe memory tiering (vSphere memory tiering) is a new feature that intelligently allows memory to be tiered to fast flash drives such as NVMe storage. It allows you to use NVMe devices that are installed locally in the host as tiered memory. This is not “dumb paging” as some have asked about the new feature. No, it is more than that. It can intelligently choose which VM memory is stored in which location.

This is a quick reference for configuring NVMe memory tiering on MS-A2 hosts. For the complete guide, see: /docs/guides/ms-a2/nvme-memory-tiering-guide.md

Device Names (MS-A2 Hosts)

MS-A2-01 (192.168.10.12) 

# Samsung 980 500GB (Boot)
t10.NVMe____Samsung_SSD_980_500GB___________________7F17A051D3382500

# Samsung 990 PRO 4TB (Memory Tiering)
t10.NVMe____Samsung_SSD_990_PRO_4TB_________________72A9415145382500

# WD_BLACK SN850X 4TB (VMFS Datastore)
t10.NVMe____WD_BLACK_SN850X_4000GB__________________E92917428B441B00

MS-A2-02 (192.168.10.13) 

# Samsung 980 500GB (Boot)
t10.NVMe____Samsung_SSD_980_500GB___________________9E17A051D3382500

# Samsung 990 PRO 4TB (Memory Tiering)
t10.NVMe____Samsung_SSD_990_PRO_4TB_________________84A9415145382500

# WD_BLACK SN850X 4TB (VMFS Datastore)
t10.NVMe____WD_BLACK_SN850X_4000GB__________________3C2917428B441B00

Quick Configuration Commands

Create WD_BLACK VM Storage Datastore

[!WARNING] DON’T DO THIS IF YOU ARE GOING TO USE IT FOR vSAN

This configuration is not compatible with vSAN deployments. If you plan to use vSAN, skip this datastore creation step.

for MS-A2-01

# Create partition table (~3.64 TB VMFS datastore on my 4TB WD Black SN850X)
partedUtil setptbl /vmfs/devices/disks/t10.NVMe____WD_BLACK_SN850X_4000GB__________________E92917428B441B00 gpt "1 2048 7814037068 AA31E02A400F11DB9590000C2911D1B8 0"

# Create VMFS6 datastore
vmkfstools -C vmfs6 -S 'esxi-ms-a2-01-wd-sn850x' /vmfs/devices/disks/t10.NVMe____WD_BLACK_SN850X_4000GB__________________E92917428B441B00:1

esxcli storage filesystem list | grep esxi-ms

for MS-A2-02

# Create partition table (~3.64 TB VMFS datastore on my 4TB WD Black SN850X)
partedUtil setptbl /vmfs/devices/disks/t10.NVMe____WD_BLACK_SN850X_4000GB__________________3C2917428B441B00 gpt "1 2048 7814037068 AA31E02A400F11DB9590000C2911D1B8 0"

# Create VMFS6 datastore
vmkfstools -C vmfs6 -S 'esxi-ms-a2-02-wd-sn850x' /vmfs/devices/disks/t10.NVMe____WD_BLACK_SN850X_4000GB__________________3C2917428B441B00:1

esxcli storage filesystem list | grep esxi-ms

Enable NVMe Memory Tiering

for MS-A2-01

# Put the host into maintenance mode. It is important that no changes are made to partitions on NVMe devices used as tiered memory while VMs are operational.
esxcli system maintenanceMode set --enable true

# List the NVMe devices on the host and choose the NVMe device to use as tiered memory and note the NVMe device path
esxcli storage core adapter device list

# Create tier device (Samsung 990 PRO)
esxcli system tierdevice create -d /vmfs/devices/disks/t10.NVMe____Samsung_SSD_990_PRO_4TB_________________72A9415145382500

# Enable memory tiering
esxcli system settings kernel set -s MemoryTiering -v TRUE

# Set 400% tier percentage (512GB from 4TB drive)
# esxcli system settings advanced set -o /Mem/TierNvmePct -i 400

# Conservative setting (25% of RAM as NVMe tier)
esxcli system settings advanced set -o /Mem/TierNvmePct -i 25

# Reboot to activate
reboot

for MS-A2-02

# Put the host into maintenance mode. It is important that no changes are made to partitions on NVMe devices used as tiered memory while VMs are operational.
esxcli system maintenanceMode set --enable true

# List the NVMe devices on the host and choose the NVMe device to use as tiered memory and note the NVMe device path
esxcli storage core adapter device list

# Create tier device (Samsung 990 PRO)
esxcli system tierdevice create -d /vmfs/devices/disks/t10.NVMe____Samsung_SSD_990_PRO_4TB_________________84A9415145382500

# Enable memory tiering
esxcli system settings kernel set -s MemoryTiering -v TRUE

# Conservative setting (25% of RAM as NVMe tier) -- so this will add an addtional 32GB of memory to 128GB (currently) for roughly 160GB of total RAM
esxcli system settings advanced set -o /Mem/TierNvmePct -i 25

# Maximum setting (400% of RAM as NVMe tier)-- will increase ram by 512GB (4 x 128) for rougly 640GB of total RAM
# esxcli system settings advanced set -o /Mem/TierNvmePct -i 400

# Reboot to activate
reboot

Verification Commands 

# Check tier devices
esxcli system tierdevice list
# [root@esxi-ms-a2-01:~] esxcli system tierdevice list
# DeviceName                                                            PartitionId  StartSector  EndSector
# --------------------------------------------------------------------  -----------  -----------  ---------
# t10.NVMe____Samsung_SSD_990_PRO_4TB_________________72A9415145382500  1            2048         7814037134

# Verify settings
esxcli system settings kernel list -o MemoryTiering
esxcli system settings advanced list -o /Mem/TierNvmePct

# List datastores
esxcli storage filesystem list | grep VMFS

Final Configuration

Effective Memory per Host: ~640GB (128GB DDR5 + 512GB NVMe tier)

Datastores:

  • esxi-ms-a2-01-boot / esxi-ms-a2-02-boot (Samsung 980 500GB)
  • esxi-ms-a2-01-wd-sn850x / esxi-ms-a2-02-wd-sn850x (WD_BLACK 4TB)
  • Samsung 990 PRO 4TB dedicated to memory tiering (no datastore)

Quick reference - see full guide for detailed instructions and troubleshooting

This project is for educational and home lab purposes.