Lost your dm-integrity tags under a RAID1 array after a reboot? We break down why this race condition happens and provide three real-world solutions to recover your array and prevent it from happening again.

RAID1 on dm-integrity: What To Do When You Lose The Tags

I still remember the 2 AM page. A junior engineer on my team, bless his heart, was doing routine kernel patching on prod-db-01. A simple reboot, something we do a hundred times a year. Except this time, the primary data array didn’t come back online. The RAID device, /dev/md0, was there, but it was degraded, and the underlying filesystem was nowhere to be found. Panic started to set in. After a bit of digging, we found the culprit: the dm-integrity layer we’d carefully placed on the physical disks had vanished, and mdadm had grabbed the raw disks directly, bypassing our data protection. It’s a subtle, infuriating problem that stems from a classic race condition, and if you’ve hit it, don’t worry—we’ve all been there. Let’s get you out of this mess.

The Root of the Problem: A Classic Race Condition

So, what’s actually happening here? It’s all about timing during the boot process. Your system is in a hurry to get things up and running, and different services are firing off at once.

1. mdadm Scans Early: The mdadm service is designed to be one of the first things to run. It scans all available block devices, looking for its RAID superblocks so it can assemble your arrays as quickly as possible.
2. dm-integrity Starts Later: The setup for dm-integrity, usually handled by systemd targets based on your /etc/integritytab file, happens a bit later in the boot sequence.

The result? mdadm finds the superblocks on /dev/sdb and /dev/sdc and immediately tries to assemble /dev/md0 with them. It effectively puts a “lock” on those raw devices. A few moments later, the dm-integrity service wakes up and tries to create its tagged, protected devices, but it’s too late. The underlying block devices are already in use by the RAID array, so it fails silently. You’re left with a RAID array running on raw, unprotected disks—exactly what you were trying to avoid.

Three Paths to Victory

Depending on whether your server is on fire right now or if you’re planning a new build, there are a few ways to tackle this. We’ll go from the quick-and-dirty fix to the permanent solution.

Solution 1: The “Get It Working NOW” Manual Recovery

It’s 2 AM, the database is down, and management is asking for an ETA. You don’t have time to architect a perfect solution; you just need to get the system back online. This is your emergency lever.

The goal here is to manually stop the incorrectly assembled array and rebuild the stack in the correct order. First, check that `mdadm` has indeed grabbed the wrong devices:

mdadm --detail /dev/md0

If you see /dev/sdb and /dev/sdc in there instead of a /dev/mapper/ device, you’ve confirmed the problem. Now, let’s fix it.

# 1. Stop the incorrectly assembled RAID array
mdadm --stop /dev/md0

# 2. Manually activate the dm-integrity layer for each disk.
#    Replace the tag names with whatever you defined in /etc/integritytab
integritysetup open /dev/sdb data-sdb-integrity
integritysetup open /dev/sdc data-sdc-integrity

# 3. Now, re-assemble the RAID array using the correct integrity devices
mdadm --assemble /dev/md0 /dev/mapper/data-sdb-integrity /dev/mapper/data-sdc-integrity

# 4. Mount your filesystem and verify your data
mount /dev/md0 /data

Heads Up: This is a temporary fix! It gets you back online, but the underlying race condition is still there. The next time prod-db-01 reboots, you’ll be right back here. Use this to stop the bleeding, then schedule a maintenance window to apply the permanent fix.

Solution 2: The “Let’s Fix This Properly” Permanent Fix with Udev

The real solution is to stop the race condition before it starts. We can do this by telling mdadm, “Hey, don’t even *think* about touching these specific disks during your initial scan. They’re reserved for something else.” The perfect tool for this job is udev, the device manager for the Linux kernel.

We’ll create a rule that identifies our specific disks and sets an environment variable that the mdadm-scan service respects, telling it to ignore them.

Step 1: Configure /etc/integritytab

First, make sure your integritytab is set up to automatically create the integrity devices on boot. It should look something like this, using a persistent UUID for reliability.

# <name>  <device>          <key file>
data-sdb-integrity   UUID=12345678-abcd-efab-cdef-1234567890ab   -
data-sdc-integrity   UUID=87654321-dcba-bafe-fedc-ba0987654321   -

Step 2: Create a Udev Rule

Now, let’s create the rule to shield these disks from mdadm. Find a unique, stable identifier for your disks, like their serial number. You can get this with `udevadm info -q property -n /dev/sdb | grep ID_SERIAL_SHORT`.

Create a new file: /etc/udev/rules.d/90-integrity-md-shield.rules

# Tell mdadm to ignore raw disks that are meant for dm-integrity
# This prevents mdadm from assembling the array before the integrity layer is up.
# Replace disk serials with your actual disk serials.

KERNEL=="sd*", ENV{ID_SERIAL_SHORT}=="SERIAL123", ENV{MD_DEV_CONTAINER}="ignore"
KERNEL=="sd*", ENV{ID_SERIAL_SHORT}=="SERIAL456", ENV{MD_DEV_CONTAINER}="ignore"

This rule tells the system that any block device matching these serial numbers should be ignored by mdadm‘s container scan. This gives integritysetup the time it needs to do its job first.

Step 3: Apply and Reboot

Reload the rules and trigger them to make sure they’re working, then update your initramfs to include the new rule so it’s active early in the boot process.

# Reload udev rules
udevadm control --reload-rules && udevadm trigger

# Update initramfs (command may vary by distro)
# For Debian/Ubuntu:
update-initramfs -u

# For RHEL/CentOS:
dracut -f

Now, reboot the server with confidence. Your array should assemble correctly on top of the dm-integrity devices every time.

Pro Tip: Always use persistent device names like /dev/disk/by-id/ or /dev/disk/by-uuid/ in your configs (fstab, mdadm.conf, integritytab). Device names like /dev/sdb can change between reboots, and that’s a whole other late-night call you don’t want.

Solution 3: The “Nuke and Pave” Rebuild

Sometimes things are just too tangled, or you’re setting up a new machine and want to get it right from the start. In that case, the cleanest option is to rebuild the entire stack from scratch, ensuring the configuration is correct from day one.

WARNING: THIS IS DESTRUCTIVE. This process will wipe all data on the target disks. Make sure you have a complete, verified backup before you even think about typing these commands.

Here’s the process for a clean build:

# 0. BACK UP YOUR DATA. I'M NOT KIDDING.

# 1. Stop and remove any existing arrays and integrity devices
mdadm --stop /dev/md0
integritysetup close data-sdb-integrity
integritysetup close data-sdc-integrity

# 2. Wipe the superblocks from the physical disks to be safe
mdadm --zero-superblock /dev/sdb
mdadm --zero-superblock /dev/sdc
wipefs -a /dev/sdb
wipefs -a /dev/sdc

# 3. Create the integrity layer on each disk
integritysetup format --tag="data-sdb-integrity" /dev/sdb
integritysetup format --tag="data-sdc-integrity" /dev/sdc

# 4. Open the newly created integrity devices
integritysetup open /dev/sdb data-sdb-integrity
integritysetup open /dev/sdc data-sdc-integrity

# 5. NOW, create the RAID1 array on top of the integrity devices
mdadm --create /dev/md0 --level=1 --raid-devices=2 /dev/mapper/data-sdb-integrity /dev/mapper/data-sdc-integrity

# 6. Create your filesystem, configure mdadm.conf, integritytab, and fstab correctly
mkfs.ext4 /dev/md0
mdadm --detail --scan >> /etc/mdadm/mdadm.conf
# (Then go set up your udev rule from Solution 2!)

# 7. Restore your data from backup

This approach guarantees a clean slate and, when combined with the udev rule from Solution 2, gives you a rock-solid, resilient storage stack.

Comparing Your Options

Here’s a quick cheat sheet for when to use each approach.

Solution	Pros	Cons	Best For
1. Manual Recovery	Fastest way to get back online.	Temporary; problem will recur on reboot.	Emergency outage response.
2. Udev Rule	Permanent, declarative, and robust.	Requires a reboot and initramfs update.	The proper fix for any existing system.
3. Nuke and Pave	Guarantees a clean, correct configuration.	Destructive, requires full data restore.	New server builds or terminally broken systems.

Final Thoughts

Layering storage technologies like dm-integrity and mdadm is powerful, but it exposes the complex dance of a modern Linux boot sequence. The “disappearing tags” problem is a perfect example of why understanding that order of operations is critical. Don’t just fix the symptom; fix the underlying race condition. A well-crafted udev rule is your best friend here, turning a potential late-night disaster into a predictable, stable system. Now, go get some sleep.

---

🤖 Frequently Asked Questions