Disks, SSDs & NVMe
HDDs, SSDs, NVMe, and how the kernel sees them
The Storage Stack at a Glance
Before looking at devices, it helps to see where they sit. A userspace write travels through multiple abstraction layers before electrons land on magnetic platters or NAND cells.
Note
Scope of this page: we focus on the bottom three layers — what the hardware looks like, how the kernel enumerates it, and which tools and knobs exist at the block device level. Filesystems are covered in storage-02-filesystems.
Hard Disk Drives (HDDs)
A rotating stack of magnetic platters read by mechanical arms. The design dates to 1956 (IBM 350) and the fundamentals have not changed — only the density, bus, and controller.
Internal Geometry
Mechanical Anatomy
- Platters: rigid aluminum or glass disks coated with a ferromagnetic film. Modern drives stack 1–10 platters, double-sided. Each side is a surface.
- Heads: one read/write head per surface, mounted on a shared actuator arm. All heads move together, so at any instant they read the same cylinder.
- Tracks: concentric circles on a platter surface where data is written.
- Sectors: fixed-size arcs of a track — historically 512 bytes, now 4096 (Advanced Format / 4Kn).
- Cylinders: the set of same-radius tracks across all surfaces — useful because reading a cylinder requires no head movement, only platter rotation.
- Spindle motor: spins the platter stack at 5400, 7200, 10K, or 15K RPM.
- Actuator: a voice-coil motor that positions the head arm radially.
The Latency Breakdown
HDD latency is the dominant reason they are slow for random I/O. Two physical delays matter:
| Component | What it is | Typical value |
|---|---|---|
| Seek time | Time for the actuator to move the head to the correct track | 4–10 ms (avg random seek) |
| Rotational latency | Time for the target sector to rotate under the head | 2–6 ms (½ rotation at 5400–15000 RPM) |
| Transfer time | Time to read/write the sector once located | < 0.1 ms per sector |
| Command overhead | Bus + controller processing | ~0.1 ms |
Note
Rule of thumb: a 7200 RPM HDD can sustain roughly 100–200 random IOPS. Sequential throughput is 150–250 MB/s. Any workload with scattered small reads is where you feel the pain.
SATA and SAS — The HDD Interfaces
- Serial ATA — consumer/prosumer interface
- AHCI controller protocol, 1 queue × 32 commands (NCQ)
- Up to 6 Gb/s (SATA III)
- Half-duplex command path, no multi-host
- Cheap cables, point-to-point only
- Serial Attached SCSI — enterprise
- Full-duplex, dual-port for HA (two paths per drive)
- Up to 22.5 Gb/s (SAS-4)
- Expanders allow hundreds of drives on one controller
- Backward compatible with SATA drives in SAS backplanes (not vice versa)
Solid State Drives (SSDs)
No moving parts. Data lives in floating-gate transistors (NAND flash) arranged in pages and blocks. Electrons are trapped in the floating gate during a program operation and swept out during an erase.
NAND Cell Density
Each NAND cell stores charge that represents one or more bits. More bits per cell means more capacity per wafer — and worse everything else.
| Type | Bits/cell | States | P/E cycles | Use case |
|---|---|---|---|---|
| SLC | 1 | 2 | ~100,000 | Industrial, caches, write-heavy |
| MLC | 2 | 4 | ~10,000 | Legacy enterprise |
| TLC | 3 | 8 | ~3,000 | Mainstream consumer + enterprise |
| QLC | 4 | 16 | ~1,000 | Read-heavy, bulk storage |
| PLC | 5 | 32 | ~100–300 | Archival (rare) |
Note
P/E cycles = program/erase cycles. Each NAND block can only be erased a finite number of times before the insulating oxide layer degrades and charge leaks. Modern drives use wear-leveling to spread writes evenly across the entire flash chip.
Why SSDs Need a Controller
NAND flash is not a block device. It has annoying properties that the drive's controller hides from the host:
NAND Reality
- Read and program at page granularity (4–16 KiB)
- Erase only at block granularity (many pages, e.g. 256 KiB–4 MiB)
- Cannot overwrite a page — must erase the entire containing block first
- Wears out after N P/E cycles
Controller Responsibilities
- FTL (Flash Translation Layer): maps logical block addresses (LBAs) to physical NAND pages
- Wear leveling: redirect writes to the least-used blocks
- Garbage collection: reclaim partially-invalid blocks
- ECC: correct bit errors that get worse with wear
- Over-provisioning: hidden spare capacity (7–28%) used for GC and replacement blocks
Write Amplification
Because the controller must copy live pages out of a block before erasing it, writing 4 KiB from the host can trigger many more KiB of actual NAND writes. This is write amplification (WAF):
Bytes written by host ÷ Bytes written to NAND
Note
WAF < 1 is possible with compression. WAF ≈ 1.1–1.5 is good (enterprise, sequential). WAF > 3 indicates a full, fragmented drive doing heavy GC — performance collapses and endurance burns fast.
Garbage Collection (GC)
Writes arrive
Host writes are programmed to free pages in open blocks
Overwrites invalidate
Old page marked stale; new page written elsewhere (log-structured)
Block accumulates stale pages
Eventually mostly invalid, a few live pages
GC kicks in
Live pages copied out, block erased, returned to free pool
NVMe — The Protocol Built for Flash
SATA and AHCI were designed around the mechanical latencies of spinning disks. When you bolt a flash device to a SATA port, the protocol itself becomes the bottleneck. NVMe (Non-Volatile Memory Express) was designed from scratch for devices where the media latency is microseconds.
Why PCIe Beats SATA for SSDs
- Single command queue of 32 entries
- 6 Gb/s shared bus, ~550 MB/s real-world ceiling
- Command set inherited from parallel ATA (1980s)
- One I/O in flight per queue slot — no parallelism across cores
- Interrupt per command, handled by a single CPU
- Up to 65,535 queues × 65,535 entries each (one queue per CPU core possible)
- PCIe 4.0 x4 ≈ 7.8 GB/s, PCIe 5.0 x4 ≈ 15.7 GB/s
- Minimal, flash-aware command set (13 mandatory admin commands)
- MSI-X interrupts distributed across CPUs
- Lockless submission/completion — each core has its own SQ/CQ pair
Tip
The 64K × 64K headline is the theoretical spec. Consumer NVMe drives typically expose 8–16 queues (matching typical core counts); enterprise drives expose 64–128. What matters is that the kernel's blk-mq layer can submit I/O from many cores simultaneously without lock contention — the opposite of the AHCI world.
NVMe Namespaces
An NVMe device (controller) can expose one or more namespaces. A namespace is an independent LBA range that looks like a separate block device to the host. This is analogous to LUNs in SCSI, and is how enterprise drives support multi-tenant isolation, different sector sizes, or different protection types on the same physical device.
# A single drive exposing two namespaces:
/dev/nvme0 # character device for controller admin commands
/dev/nvme0n1 # namespace 1 as a block device
/dev/nvme0n2 # namespace 2 as a block device
/dev/nvme0n1p1 # partition 1 of namespace 1Form Factors
| Form factor | Interface | Typical use | Notes |
|---|---|---|---|
| M.2 2280 | PCIe x4 or SATA | Laptops, consumer desktops, home servers | 22 mm wide × 80 mm long. Check the key (M-key = PCIe/NVMe, B-key = SATA). |
| U.2 / U.3 | PCIe x4 (SFF-8639) | Enterprise 2.5" hot-swap | Same physical bay as SAS, hot-pluggable, better thermals than M.2. |
| E1.S | PCIe x4 | Datacenter / hyperscale 1U | EDSFF ruler form factor. Dense, hot-swap, good airflow. |
| E3.S / E3.L | PCIe x4/x8 | Datacenter 2U+ | Replaces U.2/U.3 in modern servers. Up to 32 per 2U. |
| AIC | PCIe x4/x8/x16 | Workstations, legacy servers | Add-in card (half-height or full-height). |
How the Kernel Sees Block Devices
/dev Entries and Major/Minor Numbers
Every block device in Linux is represented by a device special file in /dev. Two integers uniquely identify a device to the kernel:
- Major number: identifies the driver (sd = 8, nvme = 259, virtio-blk = 254, loop = 7)
- Minor number: identifies a specific device instance (and partition) handled by that driver
$ ls -l /dev/sda /dev/sda1 /dev/nvme0n1 /dev/nvme0n1p1
brw-rw---- 1 root disk 8, 0 Apr 20 10:12 /dev/sda
brw-rw---- 1 root disk 8, 1 Apr 20 10:12 /dev/sda1
brw-rw---- 1 root disk 259, 0 Apr 20 10:12 /dev/nvme0n1
brw-rw---- 1 root disk 259, 1 Apr 20 10:12 /dev/nvme0n1p1
Note
The leading b in brw-rw---- means block device (vs c for character). The two numbers after the group are major and minor.
Naming Conventions
| Pattern | Meaning | Example |
|---|---|---|
/dev/sd[a-z] | SCSI disk (SATA, SAS, USB, iSCSI, FC — anything going through the SCSI subsystem) | /dev/sda, /dev/sdb1 |
/dev/nvme[N]n[M] | NVMe: controller N, namespace M. Partitions suffixed with p. | /dev/nvme0n1p2 |
/dev/vd[a-z] | virtio-blk — KVM/Proxmox paravirtualized disk | /dev/vda, /dev/vdb |
/dev/xvd[a-z] | Xen virtual disk (older AWS EC2 instances) | /dev/xvda |
/dev/mmcblk[N]p[M] | eMMC / SD card | /dev/mmcblk0p1 |
/dev/loop[N] | Loop device (file-backed block device) | /dev/loop0 |
/dev/dm-[N] | device-mapper (LVM, LUKS, multipath) | /dev/dm-0 |
/dev/md[N] | mdraid software RAID | /dev/md0 |
Warning
Do not rely on sda/sdb ordering. Enumeration order depends on probe timing. For persistent mounts, use /dev/disk/by-uuid/, /dev/disk/by-id/, or /dev/disk/by-partlabel/ — all maintained by udev and stable across reboots and bus changes.
udev — The Userspace Device Manager
When the kernel detects a new block device, it emits a uevent. udev (part of systemd) receives the event, consults rule files in /etc/udev/rules.d/ and /lib/udev/rules.d/, and creates symlinks in /dev/disk/:
/dev/disk/
├── by-id/ nvme-Samsung_SSD_980_PRO_1TB_S5P2NF0R123456
├── by-uuid/ d3b07384-d9a0-4f4d-8b3a-... # filesystem UUID
├── by-partuuid/ 12345678-1234-1234-1234-... # GPT partition UUID
├── by-partlabel/ root swap data
└── by-path/ pci-0000:04:00.0-nvme-1
Inspection Tools
lsblk — The Tree View
$ lsblk -o NAME,SIZE,TYPE,FSTYPE,MOUNTPOINTS,MODEL
NAME SIZE TYPE FSTYPE MOUNTPOINTS MODEL
nvme0n1 1.8T disk Samsung SSD 980 PRO 2TB
├─nvme0n1p1 512M part vfat /boot/efi
├─nvme0n1p2 1G part ext4 /boot
└─nvme0n1p3 1.8T part crypto_LUKS
└─cryptroot 1.8T crypt LVM2_member
├─vg-root 100G lvm ext4 /
├─vg-swap 16G lvm swap [SWAP]
└─vg-home 1.6T lvm ext4 /home
sda 931.5G disk WDC WD10EZEX-00BN5A0
└─sda1 931.5G part ext4 /mnt/backup
blkid — UUIDs and Labels
$ blkid
/dev/nvme0n1p1: UUID="1234-ABCD" TYPE="vfat" PARTLABEL="EFI" PARTUUID="..."
/dev/nvme0n1p2: UUID="a1b2c3d4-..." TYPE="ext4" PARTLABEL="boot" PARTUUID="..."
/dev/nvme0n1p3: UUID="9a8b7c6d-..." TYPE="crypto_LUKS" PARTLABEL="crypt" PARTUUID="..."
/dev/mapper/cryptroot: UUID="..." TYPE="LVM2_member"
smartctl — SMART Data
SMART (Self-Monitoring, Analysis, and Reporting Technology) is a standard set of health and wear counters exposed by every modern drive. smartctl from smartmontools is the canonical tool.
$ smartctl -a /dev/nvme0n1
=== START OF INFORMATION SECTION ===
Model Number: Samsung SSD 980 PRO 2TB
Firmware Version: 5B2QGXA7
Total NVM Capacity: 2,000,398,934,016 [2.00 TB]
=== START OF SMART DATA SECTION ===
SMART/Health Information (NVMe Log 0x02)
Critical Warning: 0x00
Temperature: 42 Celsius
Available Spare: 100%
Available Spare Threshold: 10%
Percentage Used: 3%
Data Units Read: 12,345,678 [6.32 TB]
Data Units Written: 98,765,432 [50.56 TB]
Host Read Commands: 234,567,890
Host Write Commands: 987,654,321
Controller Busy Time: 1,234
Power Cycles: 456
Power On Hours: 7,890
Unsafe Shutdowns: 3
Media and Data Integrity Errors: 0
Error Information Log Entries: 0
SMART Attributes That Matter (HDDs)
| ID | Attribute | Why it matters |
|---|---|---|
| 5 | Reallocated Sectors Count | Number of bad sectors the drive has remapped to spares. Any non-zero value means the drive has physical defects — watch the rate of growth. |
| 187 | Reported Uncorrectable Errors | ECC could not correct. Data was lost. Non-zero is a strong replace signal. |
| 188 | Command Timeout | Drive failed to respond in time. Non-zero often precedes failure. |
| 197 | Current Pending Sectors | Sectors the drive suspects are bad but hasn't remapped yet. Will become reallocated on next write. Non-zero = imminent. |
| 198 | Offline Uncorrectable | Sector unreadable during offline scan. Data loss. |
| 9 | Power-On Hours | Lifetime in hours. For context, not prediction. |
| 12 | Power Cycle Count | More cycles = more thermal stress. Matters more for HDDs than SSDs. |
SMART Attributes That Matter (SSDs)
| Attribute | Why it matters |
|---|---|
| Wear Leveling Count / Percentage Used | Fraction of rated endurance consumed. 0% = new, 100% = rated end-of-life (not immediate death, but warranty is out). |
| Available Spare / Spare Threshold | Fraction of reserve blocks remaining. When it drops below the threshold (often 10%), the drive is running out of replacement blocks. |
| Total Bytes Written (TBW) / Data Units Written | Lifetime host writes. Compare to the manufacturer's TBW rating. |
| Media and Data Integrity Errors | Unrecoverable ECC errors. Should be zero. |
| Unsafe Shutdowns | Power loss without flush. High count + lack of PLP = risk of data corruption. |
| Critical Warning bits | Bitmask — spare below threshold, temperature above threshold, reliability degraded, read-only mode, volatile memory backup failed. |
Tip
Monitoring in practice: enable smartd (the daemon from smartmontools). It runs short/long self-tests on a schedule and emails/logs when attributes cross thresholds. Prometheus exporters (node_exporter --collector.smartmon or smartctl_exporter) feed these into dashboards.
nvme-cli — The NVMe-Specific Swiss Army Knife
# nvme list
Node SN Model Namespace Usage Format FW Rev
/dev/nvme0n1 S5P2NF0R... Samsung SSD 980 PRO 1 2.00 TB/2.00 TB 512 B + 0 B 5B2QGXA7
# nvme id-ctrl /dev/nvme0 | head -20
NVME Identify Controller:
vid : 0x144d
ssvid : 0x144d
sn : S5P2NF0R123456
mn : Samsung SSD 980 PRO 2TB
fr : 5B2QGXA7
rab : 2
ieee : 002538
cmic : 0
mdts : 9 # max data transfer size = 2^9 × MPSMIN
cntlid : 0x6
ver : 0x10400 # NVMe 1.4
oacs : 0x17 # optional admin commands supported
acl : 7
aerl : 3
# nvme smart-log /dev/nvme0
# nvme error-log /dev/nvme0
# nvme format /dev/nvme0n1 --lbaf=1 --ses=1 # reformat namespace to 4K sectors, secure erase
hdparm — Legacy but Useful
# hdparm -I /dev/sda # identify info
# hdparm -tT /dev/sda # quick read benchmark (cached + buffered)
# hdparm -W 0 /dev/sda # disable drive write cache (for consistency guarantees)
/sys/block/ — The Kernel's View
Every block device has a directory under /sys/block/ exposing runtime state. The queue/ subdir holds the block-layer knobs:
/sys/block/nvme0n1/
├── size # size in 512-byte sectors
├── stat # cumulative I/O counters
├── queue/
│ ├── scheduler # active I/O scheduler, [brackets] = current
│ ├── nr_requests # queue depth for the scheduler
│ ├── rotational # 1 = HDD, 0 = SSD
│ ├── logical_block_size # usually 512
│ ├── physical_block_size# often 4096 on modern drives
│ ├── discard_granularity# smallest TRIM unit, 0 if unsupported
│ ├── discard_max_bytes # largest TRIM in one command
│ ├── read_ahead_kb # readahead window (default 128)
│ ├── max_sectors_kb # max I/O size in one request
│ └── write_cache # "write back" or "write through"
└── nvme0n1p1/ # each partition also appears as a child
TRIM / Discard
The SSD's FTL knows which LBAs have been written but has no way to know when the filesystem has deleted data. Without hints, the controller considers all ever-written LBAs as live and must copy their stale contents during garbage collection — wasting endurance and slowing writes.
TRIM (ATA), UNMAP (SCSI), and Dataset Management / Deallocate (NVMe) are the commands the filesystem uses to tell the drive "these LBAs are now free; don't bother preserving them." The kernel calls this discard.
Three Ways to Discard
1. Continuous (mount option)
Add discard to the mount options (or issue_discards in LVM). Every rm/truncate triggers a discard immediately.
Pro: drive always knows the truth.
Con: small synchronous discards can stall the filesystem. Some older drives have slow discard.
2. Periodic (fstrim.timer)
Run fstrim -av from a systemd timer (usually weekly). It walks free space and issues large batched discards.
Pro: amortized cost, no impact on foreground I/O.
Con: drive operates blind between runs.
Default on most distros.
3. Manual
fstrim /mnt/data on demand, or blkdiscard /dev/nvmeXnY for whole-device wipe (destructive).
Useful before benchmarks or after bulk deletions.
# systemctl status fstrim.timer
fstrim.timer - Discard unused blocks once a week
Loaded: loaded (/lib/systemd/system/fstrim.timer; enabled)
Active: active (waiting)
Trigger: Mon 2026-04-27 00:17:43 UTC; 6 days left
# fstrim -av
/boot: 398.4 MiB (417714176 bytes) trimmed on /dev/nvme0n1p2
/: 47.2 GiB (50693652480 bytes) trimmed on /dev/mapper/vg-root
The LUKS Discard Trade-off
Warning
LUKS --allow-discards lets the kernel forward TRIM commands through the encryption layer to the underlying SSD. Security cost: an attacker with raw-disk access can distinguish allocated (random-looking ciphertext) from unallocated (reads as zeros). They learn the size and rough shape of your filesystem, though not its contents.
For most threat models the trade-off is worth it — disabling discard on a full-disk encrypted SSD causes steady write amplification and endurance loss. For adversaries who must not know whether the disk holds 10 GB or 900 GB of data, leave discards off.
I/O Schedulers
The block layer sits between filesystems and device drivers. When multiple requests are pending, the I/O scheduler decides the order in which they are dispatched. Modern Linux uses blk-mq (multi-queue block layer); the schedulers below are the multi-queue variants.
| Scheduler | How it works | When to use |
|---|---|---|
| none | FIFO — submit in the order requests arrive. Zero overhead. | NVMe SSDs — the device has its own queues and reordering logic; adding a software scheduler just burns CPU. Default for NVMe. |
| mq-deadline | Two queues (read, write) with per-request deadlines; prefers reads. Simple and predictable. | SATA SSDs, shared storage, databases where read latency matters. Safe default. |
| bfq | Budget Fair Queueing — per-process fair sharing with low-latency heuristics for interactive workloads. | Desktop/laptop HDDs, mixed-workload spinning disks. Good responsiveness, higher CPU overhead. |
| kyber | Tracks completion latency and throttles queue depth to keep tail latency in check. | Fast SSDs where you want to cap p99 latency at the cost of some throughput. Rarely the default. |
$ cat /sys/block/nvme0n1/queue/scheduler
[none] mq-deadline kyber bfq
# echo mq-deadline > /sys/block/sda/queue/scheduler # switch SATA disk to deadline
Tip
# /etc/udev/rules.d/60-ioschedulers.rules
ACTION=="add|change", KERNEL=="nvme[0-9]*", ATTR{queue/scheduler}="none"
ACTION=="add|change", KERNEL=="sd[a-z]", ATTR{queue/rotational}=="0", ATTR{queue/scheduler}="mq-deadline"
ACTION=="add|change", KERNEL=="sd[a-z]", ATTR{queue/rotational}=="1", ATTR{queue/scheduler}="bfq"Alignment and Block Sizes
Logical vs Physical Sector Size
Historically, disks used 512-byte sectors. Modern drives use 4096-byte physical sectors (Advanced Format, aka 4Kn or 512e):
Physical = 512, Logical = 512. Legacy drives only.
Physical = 4096, Logical = 512. The drive buffers and RMW's for the host. Misalignment is invisible but catastrophic for performance.
Note
Physical = Logical = 4096. Enterprise SSDs, modern NVMe, high-capacity HDDs. The OS must speak 4K — older BIOSes and OSes cannot boot from 4Kn disks.
$ cat /sys/block/nvme0n1/queue/logical_block_size
512
$ cat /sys/block/nvme0n1/queue/physical_block_size
4096
Why Misalignment Kills Performance
If the filesystem's 4 KiB block straddles two 4 KiB physical sectors, every write becomes a Read-Modify-Write (RMW): the drive reads both physical sectors, patches the modified bytes, and writes both back. Throughput drops by up to 50% and write amplification balloons.
Note
The 1 MiB rule: modern partitioning tools align the first partition to an LBA that is a multiple of 2048 sectors (1 MiB). 1 MiB is a common multiple of every realistic physical sector size (4K, 8K, 16K), NAND page size, NAND erase block size (up to 1 MiB), and RAID stripe size. Aligning to 1 MiB is correct for all modern devices.
$ fdisk -l /dev/nvme0n1 | head
Disk /dev/nvme0n1: 1.82 TiB, 2000398934016 bytes, 3907029168 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: gpt
Device Start End Sectors Size Type
/dev/nvme0n1p1 2048 1050623 1048576 512M EFI System
/dev/nvme0n1p2 1050624 3147775 2097152 1G Linux filesystem
Tip
First partition starts at LBA 2048 (= 2048 × 512 B = 1 MiB). Every subsequent partition ends on an LBA whose next sector is also aligned. This is what parted, sgdisk, and modern fdisk do by default.
Cheat Sheet — HDD vs SATA SSD vs NVMe
| Metric | HDD (7.2K) | SATA SSD | NVMe (PCIe 4.0) |
|---|---|---|---|
| Random 4K IOPS | ~150 | ~90,000 | ~1,000,000 |
| Sequential throughput | ~200 MB/s | ~550 MB/s | ~7,000 MB/s |
| Access latency | ~8 ms | ~70 µs | ~20 µs |
| Queue depth | 1 × 32 | 1 × 32 | 64K × 64K |
| Default scheduler | bfq / mq-deadline | mq-deadline | none |
| Endurance limit | MTBF / head crash | P/E cycles (TBW) | P/E cycles (TBW) |
| TRIM needed | No | Yes | Yes |
| Use case | Bulk cold storage, backups | General-purpose | Databases, VMs, tiered hot data |