Skip to content
Menu

Linux Storage16 min read

Disks, SSDs & NVMe

HDDs, SSDs, NVMe, and how the kernel sees them

The Storage Stack at a Glance

Before looking at devices, it helps to see where they sit. A userspace write travels through multiple abstraction layers before electrons land on magnetic platters or NAND cells.

7 Application bytes write(2), pwrite(2), mmap(2)
6 VFS inodes / files Virtual FileSystem — path resolution, caching
5 Filesystem blocks ext4, xfs, btrfs, zfs — metadata, journaling, CoW
4 Block Layer bio / request page cache, I/O scheduler, multi-queue (blk-mq)
3 Device Driver SCSI / NVMe commands sd, nvme, virtio-blk, drbd
2 Transport / Bus frames SATA (AHCI), SAS, PCIe, USB, iSCSI, FC
1 Physical Medium sectors / pages Magnetic platters, NAND flash, 3D XPoint

Note

Scope of this page: we focus on the bottom three layers — what the hardware looks like, how the kernel enumerates it, and which tools and knobs exist at the block device level. Filesystems are covered in storage-02-filesystems.

Hard Disk Drives (HDDs)

A rotating stack of magnetic platters read by mechanical arms. The design dates to 1956 (IBM 350) and the fundamentals have not changed — only the density, bus, and controller.

Internal Geometry

Mechanical Anatomy

  • Platters: rigid aluminum or glass disks coated with a ferromagnetic film. Modern drives stack 1–10 platters, double-sided. Each side is a surface.
  • Heads: one read/write head per surface, mounted on a shared actuator arm. All heads move together, so at any instant they read the same cylinder.
  • Tracks: concentric circles on a platter surface where data is written.
  • Sectors: fixed-size arcs of a track — historically 512 bytes, now 4096 (Advanced Format / 4Kn).
  • Cylinders: the set of same-radius tracks across all surfaces — useful because reading a cylinder requires no head movement, only platter rotation.
  • Spindle motor: spins the platter stack at 5400, 7200, 10K, or 15K RPM.
  • Actuator: a voice-coil motor that positions the head arm radially.

The Latency Breakdown

HDD latency is the dominant reason they are slow for random I/O. Two physical delays matter:

ComponentWhat it isTypical value
Seek timeTime for the actuator to move the head to the correct track4–10 ms (avg random seek)
Rotational latencyTime for the target sector to rotate under the head2–6 ms (½ rotation at 5400–15000 RPM)
Transfer timeTime to read/write the sector once located< 0.1 ms per sector
Command overheadBus + controller processing~0.1 ms

Note

Rule of thumb: a 7200 RPM HDD can sustain roughly 100–200 random IOPS. Sequential throughput is 150–250 MB/s. Any workload with scattered small reads is where you feel the pain.

SATA and SAS — The HDD Interfaces

SATA
  • Serial ATA — consumer/prosumer interface
  • AHCI controller protocol, 1 queue × 32 commands (NCQ)
  • Up to 6 Gb/s (SATA III)
  • Half-duplex command path, no multi-host
  • Cheap cables, point-to-point only
SAS
  • Serial Attached SCSI — enterprise
  • Full-duplex, dual-port for HA (two paths per drive)
  • Up to 22.5 Gb/s (SAS-4)
  • Expanders allow hundreds of drives on one controller
  • Backward compatible with SATA drives in SAS backplanes (not vice versa)

Solid State Drives (SSDs)

No moving parts. Data lives in floating-gate transistors (NAND flash) arranged in pages and blocks. Electrons are trapped in the floating gate during a program operation and swept out during an erase.

NAND Cell Density

Each NAND cell stores charge that represents one or more bits. More bits per cell means more capacity per wafer — and worse everything else.

TypeBits/cellStatesP/E cyclesUse case
SLC12~100,000Industrial, caches, write-heavy
MLC24~10,000Legacy enterprise
TLC38~3,000Mainstream consumer + enterprise
QLC416~1,000Read-heavy, bulk storage
PLC532~100–300Archival (rare)

Note

P/E cycles = program/erase cycles. Each NAND block can only be erased a finite number of times before the insulating oxide layer degrades and charge leaks. Modern drives use wear-leveling to spread writes evenly across the entire flash chip.

Why SSDs Need a Controller

NAND flash is not a block device. It has annoying properties that the drive's controller hides from the host:

NAND Reality

  • Read and program at page granularity (4–16 KiB)
  • Erase only at block granularity (many pages, e.g. 256 KiB–4 MiB)
  • Cannot overwrite a page — must erase the entire containing block first
  • Wears out after N P/E cycles

Controller Responsibilities

  • FTL (Flash Translation Layer): maps logical block addresses (LBAs) to physical NAND pages
  • Wear leveling: redirect writes to the least-used blocks
  • Garbage collection: reclaim partially-invalid blocks
  • ECC: correct bit errors that get worse with wear
  • Over-provisioning: hidden spare capacity (7–28%) used for GC and replacement blocks

Write Amplification

Because the controller must copy live pages out of a block before erasing it, writing 4 KiB from the host can trigger many more KiB of actual NAND writes. This is write amplification (WAF):

Bytes written by host ÷ Bytes written to NAND

Note

WAF < 1 is possible with compression. WAF ≈ 1.1–1.5 is good (enterprise, sequential). WAF > 3 indicates a full, fragmented drive doing heavy GC — performance collapses and endurance burns fast.

Garbage Collection (GC)

  1. Writes arrive

    Host writes are programmed to free pages in open blocks

  2. Overwrites invalidate

    Old page marked stale; new page written elsewhere (log-structured)

  3. Block accumulates stale pages

    Eventually mostly invalid, a few live pages

  4. GC kicks in

    Live pages copied out, block erased, returned to free pool

NVMe — The Protocol Built for Flash

SATA and AHCI were designed around the mechanical latencies of spinning disks. When you bolt a flash device to a SATA port, the protocol itself becomes the bottleneck. NVMe (Non-Volatile Memory Express) was designed from scratch for devices where the media latency is microseconds.

Why PCIe Beats SATA for SSDs

SATA / AHCI (legacy)
  • Single command queue of 32 entries
  • 6 Gb/s shared bus, ~550 MB/s real-world ceiling
  • Command set inherited from parallel ATA (1980s)
  • One I/O in flight per queue slot — no parallelism across cores
  • Interrupt per command, handled by a single CPU
NVMe / PCIe
  • Up to 65,535 queues × 65,535 entries each (one queue per CPU core possible)
  • PCIe 4.0 x4 ≈ 7.8 GB/s, PCIe 5.0 x4 ≈ 15.7 GB/s
  • Minimal, flash-aware command set (13 mandatory admin commands)
  • MSI-X interrupts distributed across CPUs
  • Lockless submission/completion — each core has its own SQ/CQ pair

Tip

The 64K × 64K headline is the theoretical spec. Consumer NVMe drives typically expose 8–16 queues (matching typical core counts); enterprise drives expose 64–128. What matters is that the kernel's blk-mq layer can submit I/O from many cores simultaneously without lock contention — the opposite of the AHCI world.

NVMe Namespaces

An NVMe device (controller) can expose one or more namespaces. A namespace is an independent LBA range that looks like a separate block device to the host. This is analogous to LUNs in SCSI, and is how enterprise drives support multi-tenant isolation, different sector sizes, or different protection types on the same physical device.

console
# A single drive exposing two namespaces:
/dev/nvme0          # character device for controller admin commands
/dev/nvme0n1        # namespace 1 as a block device
/dev/nvme0n2        # namespace 2 as a block device
/dev/nvme0n1p1      # partition 1 of namespace 1

Form Factors

Form factorInterfaceTypical useNotes
M.2 2280PCIe x4 or SATALaptops, consumer desktops, home servers22 mm wide × 80 mm long. Check the key (M-key = PCIe/NVMe, B-key = SATA).
U.2 / U.3PCIe x4 (SFF-8639)Enterprise 2.5" hot-swapSame physical bay as SAS, hot-pluggable, better thermals than M.2.
E1.SPCIe x4Datacenter / hyperscale 1UEDSFF ruler form factor. Dense, hot-swap, good airflow.
E3.S / E3.LPCIe x4/x8Datacenter 2U+Replaces U.2/U.3 in modern servers. Up to 32 per 2U.
AICPCIe x4/x8/x16Workstations, legacy serversAdd-in card (half-height or full-height).

How the Kernel Sees Block Devices

/dev Entries and Major/Minor Numbers

Every block device in Linux is represented by a device special file in /dev. Two integers uniquely identify a device to the kernel:

  • Major number: identifies the driver (sd = 8, nvme = 259, virtio-blk = 254, loop = 7)
  • Minor number: identifies a specific device instance (and partition) handled by that driver
console
console

    $ ls -l /dev/sda /dev/sda1 /dev/nvme0n1 /dev/nvme0n1p1

    brw-rw---- 1 root disk   8,   0 Apr 20 10:12 /dev/sda

    brw-rw---- 1 root disk   8,   1 Apr 20 10:12 /dev/sda1

    brw-rw---- 1 root disk 259,   0 Apr 20 10:12 /dev/nvme0n1

    brw-rw---- 1 root disk 259,   1 Apr 20 10:12 /dev/nvme0n1p1
  

Note

The leading b in brw-rw---- means block device (vs c for character). The two numbers after the group are major and minor.

Naming Conventions

PatternMeaningExample
/dev/sd[a-z]SCSI disk (SATA, SAS, USB, iSCSI, FC — anything going through the SCSI subsystem)/dev/sda, /dev/sdb1
/dev/nvme[N]n[M]NVMe: controller N, namespace M. Partitions suffixed with p./dev/nvme0n1p2
/dev/vd[a-z]virtio-blk — KVM/Proxmox paravirtualized disk/dev/vda, /dev/vdb
/dev/xvd[a-z]Xen virtual disk (older AWS EC2 instances)/dev/xvda
/dev/mmcblk[N]p[M]eMMC / SD card/dev/mmcblk0p1
/dev/loop[N]Loop device (file-backed block device)/dev/loop0
/dev/dm-[N]device-mapper (LVM, LUKS, multipath)/dev/dm-0
/dev/md[N]mdraid software RAID/dev/md0

Warning

Do not rely on sda/sdb ordering. Enumeration order depends on probe timing. For persistent mounts, use /dev/disk/by-uuid/, /dev/disk/by-id/, or /dev/disk/by-partlabel/ — all maintained by udev and stable across reboots and bus changes.

udev — The Userspace Device Manager

When the kernel detects a new block device, it emits a uevent. udev (part of systemd) receives the event, consults rule files in /etc/udev/rules.d/ and /lib/udev/rules.d/, and creates symlinks in /dev/disk/:

text
bash

/dev/disk/
├── by-id/          nvme-Samsung_SSD_980_PRO_1TB_S5P2NF0R123456
├── by-uuid/        d3b07384-d9a0-4f4d-8b3a-...  # filesystem UUID
├── by-partuuid/    12345678-1234-1234-1234-...   # GPT partition UUID
├── by-partlabel/   root swap data
└── by-path/        pci-0000:04:00.0-nvme-1
  

Inspection Tools

lsblk — The Tree View

console
console

    $ lsblk -o NAME,SIZE,TYPE,FSTYPE,MOUNTPOINTS,MODEL

    NAME          SIZE TYPE FSTYPE      MOUNTPOINTS   MODEL

    nvme0n1       1.8T disk                           Samsung SSD 980 PRO 2TB

    ├─nvme0n1p1   512M part vfat        /boot/efi

    ├─nvme0n1p2     1G part ext4        /boot

    └─nvme0n1p3   1.8T part crypto_LUKS

      └─cryptroot 1.8T crypt LVM2_member

        ├─vg-root 100G lvm  ext4        /

        ├─vg-swap  16G lvm  swap        [SWAP]

        └─vg-home 1.6T lvm  ext4        /home

    sda         931.5G disk                           WDC WD10EZEX-00BN5A0

    └─sda1      931.5G part ext4        /mnt/backup
  

blkid — UUIDs and Labels

console
console

    $ blkid

    /dev/nvme0n1p1: UUID="1234-ABCD" TYPE="vfat" PARTLABEL="EFI" PARTUUID="..."

    /dev/nvme0n1p2: UUID="a1b2c3d4-..." TYPE="ext4" PARTLABEL="boot" PARTUUID="..."

    /dev/nvme0n1p3: UUID="9a8b7c6d-..." TYPE="crypto_LUKS" PARTLABEL="crypt" PARTUUID="..."

    /dev/mapper/cryptroot: UUID="..." TYPE="LVM2_member"
  

smartctl — SMART Data

SMART (Self-Monitoring, Analysis, and Reporting Technology) is a standard set of health and wear counters exposed by every modern drive. smartctl from smartmontools is the canonical tool.

console
console

    $ smartctl -a /dev/nvme0n1

    === START OF INFORMATION SECTION ===

    Model Number: Samsung SSD 980 PRO 2TB

    Firmware Version: 5B2QGXA7

    Total NVM Capacity: 2,000,398,934,016 [2.00 TB]

    

    === START OF SMART DATA SECTION ===

    SMART/Health Information (NVMe Log 0x02)

    Critical Warning:                   0x00

    Temperature:                        42 Celsius

    Available Spare:                    100%

    Available Spare Threshold:          10%

    Percentage Used:                    3%

    Data Units Read:                    12,345,678 [6.32 TB]

    Data Units Written:                 98,765,432 [50.56 TB]

    Host Read Commands:                 234,567,890

    Host Write Commands:                987,654,321

    Controller Busy Time:               1,234

    Power Cycles:                       456

    Power On Hours:                     7,890

    Unsafe Shutdowns:                   3

    Media and Data Integrity Errors:    0

    Error Information Log Entries:      0
  

SMART Attributes That Matter (HDDs)

IDAttributeWhy it matters
5Reallocated Sectors CountNumber of bad sectors the drive has remapped to spares. Any non-zero value means the drive has physical defects — watch the rate of growth.
187Reported Uncorrectable ErrorsECC could not correct. Data was lost. Non-zero is a strong replace signal.
188Command TimeoutDrive failed to respond in time. Non-zero often precedes failure.
197Current Pending SectorsSectors the drive suspects are bad but hasn't remapped yet. Will become reallocated on next write. Non-zero = imminent.
198Offline UncorrectableSector unreadable during offline scan. Data loss.
9Power-On HoursLifetime in hours. For context, not prediction.
12Power Cycle CountMore cycles = more thermal stress. Matters more for HDDs than SSDs.

SMART Attributes That Matter (SSDs)

AttributeWhy it matters
Wear Leveling Count / Percentage UsedFraction of rated endurance consumed. 0% = new, 100% = rated end-of-life (not immediate death, but warranty is out).
Available Spare / Spare ThresholdFraction of reserve blocks remaining. When it drops below the threshold (often 10%), the drive is running out of replacement blocks.
Total Bytes Written (TBW) / Data Units WrittenLifetime host writes. Compare to the manufacturer's TBW rating.
Media and Data Integrity ErrorsUnrecoverable ECC errors. Should be zero.
Unsafe ShutdownsPower loss without flush. High count + lack of PLP = risk of data corruption.
Critical Warning bitsBitmask — spare below threshold, temperature above threshold, reliability degraded, read-only mode, volatile memory backup failed.

Tip

Monitoring in practice: enable smartd (the daemon from smartmontools). It runs short/long self-tests on a schedule and emails/logs when attributes cross thresholds. Prometheus exporters (node_exporter --collector.smartmon or smartctl_exporter) feed these into dashboards.

nvme-cli — The NVMe-Specific Swiss Army Knife

console
console

    # nvme list

    Node          SN           Model                  Namespace Usage        Format           FW Rev

    /dev/nvme0n1  S5P2NF0R...  Samsung SSD 980 PRO    1         2.00 TB/2.00 TB 512  B +  0 B   5B2QGXA7
    


    # nvme id-ctrl /dev/nvme0 | head -20

    NVME Identify Controller:

    vid       : 0x144d

    ssvid     : 0x144d

    sn        : S5P2NF0R123456

    mn        : Samsung SSD 980 PRO 2TB

    fr        : 5B2QGXA7

    rab       : 2

    ieee      : 002538

    cmic      : 0

    mdts      : 9        # max data transfer size = 2^9 × MPSMIN

    cntlid    : 0x6

    ver       : 0x10400  # NVMe 1.4

    oacs      : 0x17     # optional admin commands supported

    acl       : 7

    aerl      : 3
    


    # nvme smart-log /dev/nvme0

    # nvme error-log /dev/nvme0

    # nvme format /dev/nvme0n1 --lbaf=1 --ses=1     # reformat namespace to 4K sectors, secure erase
  

hdparm — Legacy but Useful

console
console

    # hdparm -I /dev/sda          # identify info

    # hdparm -tT /dev/sda         # quick read benchmark (cached + buffered)

    # hdparm -W 0 /dev/sda        # disable drive write cache (for consistency guarantees)
  

/sys/block/ — The Kernel's View

Every block device has a directory under /sys/block/ exposing runtime state. The queue/ subdir holds the block-layer knobs:

text
bash

/sys/block/nvme0n1/
├── size                  # size in 512-byte sectors
├── stat                  # cumulative I/O counters
├── queue/
│   ├── scheduler         # active I/O scheduler, [brackets] = current
│   ├── nr_requests       # queue depth for the scheduler
│   ├── rotational        # 1 = HDD, 0 = SSD
│   ├── logical_block_size # usually 512
│   ├── physical_block_size# often 4096 on modern drives
│   ├── discard_granularity# smallest TRIM unit, 0 if unsupported
│   ├── discard_max_bytes  # largest TRIM in one command
│   ├── read_ahead_kb      # readahead window (default 128)
│   ├── max_sectors_kb     # max I/O size in one request
│   └── write_cache        # "write back" or "write through"
└── nvme0n1p1/             # each partition also appears as a child
  

TRIM / Discard

The SSD's FTL knows which LBAs have been written but has no way to know when the filesystem has deleted data. Without hints, the controller considers all ever-written LBAs as live and must copy their stale contents during garbage collection — wasting endurance and slowing writes.

TRIM (ATA), UNMAP (SCSI), and Dataset Management / Deallocate (NVMe) are the commands the filesystem uses to tell the drive "these LBAs are now free; don't bother preserving them." The kernel calls this discard.

Three Ways to Discard

1. Continuous (mount option)

Add discard to the mount options (or issue_discards in LVM). Every rm/truncate triggers a discard immediately.

Pro: drive always knows the truth.

Con: small synchronous discards can stall the filesystem. Some older drives have slow discard.

2. Periodic (fstrim.timer)

Run fstrim -av from a systemd timer (usually weekly). It walks free space and issues large batched discards.

Pro: amortized cost, no impact on foreground I/O.

Con: drive operates blind between runs.

Default on most distros.

3. Manual

fstrim /mnt/data on demand, or blkdiscard /dev/nvmeXnY for whole-device wipe (destructive).

Useful before benchmarks or after bulk deletions.

console
console

    # systemctl status fstrim.timer

    fstrim.timer - Discard unused blocks once a week

       Loaded: loaded (/lib/systemd/system/fstrim.timer; enabled)

       Active: active (waiting)

      Trigger: Mon 2026-04-27 00:17:43 UTC; 6 days left
    


    # fstrim -av

    /boot: 398.4 MiB (417714176 bytes) trimmed on /dev/nvme0n1p2

    /: 47.2 GiB (50693652480 bytes) trimmed on /dev/mapper/vg-root
  

The LUKS Discard Trade-off

Warning

LUKS --allow-discards lets the kernel forward TRIM commands through the encryption layer to the underlying SSD. Security cost: an attacker with raw-disk access can distinguish allocated (random-looking ciphertext) from unallocated (reads as zeros). They learn the size and rough shape of your filesystem, though not its contents.

For most threat models the trade-off is worth it — disabling discard on a full-disk encrypted SSD causes steady write amplification and endurance loss. For adversaries who must not know whether the disk holds 10 GB or 900 GB of data, leave discards off.

I/O Schedulers

The block layer sits between filesystems and device drivers. When multiple requests are pending, the I/O scheduler decides the order in which they are dispatched. Modern Linux uses blk-mq (multi-queue block layer); the schedulers below are the multi-queue variants.

SchedulerHow it worksWhen to use
noneFIFO — submit in the order requests arrive. Zero overhead.NVMe SSDs — the device has its own queues and reordering logic; adding a software scheduler just burns CPU. Default for NVMe.
mq-deadlineTwo queues (read, write) with per-request deadlines; prefers reads. Simple and predictable.SATA SSDs, shared storage, databases where read latency matters. Safe default.
bfqBudget Fair Queueing — per-process fair sharing with low-latency heuristics for interactive workloads.Desktop/laptop HDDs, mixed-workload spinning disks. Good responsiveness, higher CPU overhead.
kyberTracks completion latency and throttles queue depth to keep tail latency in check.Fast SSDs where you want to cap p99 latency at the cost of some throughput. Rarely the default.
console
console

    $ cat /sys/block/nvme0n1/queue/scheduler

    [none] mq-deadline kyber bfq
    

    # echo mq-deadline > /sys/block/sda/queue/scheduler    # switch SATA disk to deadline
  

Tip

Persistent scheduler selection: udev rule is the clean way.
console
# /etc/udev/rules.d/60-ioschedulers.rules
ACTION=="add|change", KERNEL=="nvme[0-9]*", ATTR{queue/scheduler}="none"
ACTION=="add|change", KERNEL=="sd[a-z]", ATTR{queue/rotational}=="0", ATTR{queue/scheduler}="mq-deadline"
ACTION=="add|change", KERNEL=="sd[a-z]", ATTR{queue/rotational}=="1", ATTR{queue/scheduler}="bfq"

Alignment and Block Sizes

Logical vs Physical Sector Size

Historically, disks used 512-byte sectors. Modern drives use 4096-byte physical sectors (Advanced Format, aka 4Kn or 512e):

512n (native 512B)

Physical = 512, Logical = 512. Legacy drives only.

512e (emulated)

Physical = 4096, Logical = 512. The drive buffers and RMW's for the host. Misalignment is invisible but catastrophic for performance.

Note

4Kn (native 4K)

Physical = Logical = 4096. Enterprise SSDs, modern NVMe, high-capacity HDDs. The OS must speak 4K — older BIOSes and OSes cannot boot from 4Kn disks.

console
console

    $ cat /sys/block/nvme0n1/queue/logical_block_size

    512

    $ cat /sys/block/nvme0n1/queue/physical_block_size

    4096
  

Why Misalignment Kills Performance

If the filesystem's 4 KiB block straddles two 4 KiB physical sectors, every write becomes a Read-Modify-Write (RMW): the drive reads both physical sectors, patches the modified bytes, and writes both back. Throughput drops by up to 50% and write amplification balloons.

Note

The 1 MiB rule: modern partitioning tools align the first partition to an LBA that is a multiple of 2048 sectors (1 MiB). 1 MiB is a common multiple of every realistic physical sector size (4K, 8K, 16K), NAND page size, NAND erase block size (up to 1 MiB), and RAID stripe size. Aligning to 1 MiB is correct for all modern devices.

console
console

    $ fdisk -l /dev/nvme0n1 | head

    Disk /dev/nvme0n1: 1.82 TiB, 2000398934016 bytes, 3907029168 sectors

    Units: sectors of 1 * 512 = 512 bytes

    Sector size (logical/physical): 512 bytes / 512 bytes

    I/O size (minimum/optimal): 512 bytes / 512 bytes

    Disklabel type: gpt

    

    Device          Start        End    Sectors  Size Type

    /dev/nvme0n1p1   2048    1050623    1048576  512M EFI System

    /dev/nvme0n1p2  1050624    3147775    2097152    1G Linux filesystem
  

Tip

First partition starts at LBA 2048 (= 2048 × 512 B = 1 MiB). Every subsequent partition ends on an LBA whose next sector is also aligned. This is what parted, sgdisk, and modern fdisk do by default.

Cheat Sheet — HDD vs SATA SSD vs NVMe

MetricHDD (7.2K)SATA SSDNVMe (PCIe 4.0)
Random 4K IOPS~150~90,000~1,000,000
Sequential throughput~200 MB/s~550 MB/s~7,000 MB/s
Access latency~8 ms~70 µs~20 µs
Queue depth1 × 321 × 3264K × 64K
Default schedulerbfq / mq-deadlinemq-deadlinenone
Endurance limitMTBF / head crashP/E cycles (TBW)P/E cycles (TBW)
TRIM neededNoYesYes
Use caseBulk cold storage, backupsGeneral-purposeDatabases, VMs, tiered hot data
Solidnines — solidnines.com