Case study: recovery of a corrupted 12 TB multi-device pool

4/6/2026 at 5:47:48 AM

> This is not a bug report. [...] The goal is constructive, not a complaint.

Er, I appreciate trying to be constructive, but in what possible situation is it not a bug that a power cycle can lose the pool? And if it's not technically a "bug" because BTRFS officially specifies that it can fail like that, why is that not in big bold text at the start of any docs on it? 'Cuz that's kind of a big deal for users to know.

EDIT: From the longer write-up:

> Initial damage. A hard power cycle interrupted a commit at generation 18958 to 18959. Both DUP copies of several metadata blocks were written with inconsistent parent and child generations.

Did the author disable safety mechanisms for that to happen? I'm coming from being more familiar with ZFS, but I would have expected BTRFS to also use a CoW model where it wasn't possible to have multiple inconsistent metadata blocks in a way that didn't just revert you to the last fully-good commit. If it does that by default but there's a way to disable that protection in the name of improving performance, that would significantly change my view of this whole thing.

by yjftsjthsd-h

4/6/2026 at 6:29:13 AM

As far as I can see, no, the author disabled nothing of the sort that he documented.

I suspect that the author's intent is less "I do not view this as a bug" and more "I do not think it's useful to get into angry debates over whether something is a bug". I do not know whether this is a common thing on btrfs discussions, but I have certainly seen debates to that effect elsewhere.

(My personal favorite remains "it's not a data loss bug if someone could technically theoretically write something to recover the data". Perhaps, technically, that's true, but if nobody is writing such a tool, nobody is going to care about the semantics there.)

by rincebrain

4/6/2026 at 7:04:24 AM

> I suspect that the author's intent is less "I do not view this as a bug" and more "I do not think it's useful to get into angry debates over whether something is a bug".

Agreed, and I appreciate the attempt to channel things into a productive conversation.

by yjftsjthsd-h

4/6/2026 at 8:22:03 AM

btrfs's reputation is not great in this regard.

by rcxdude

4/6/2026 at 11:48:21 AM

As far as I understand, single device and RAID1 is solid, but as soon as you want to do RAID1+0 or RAID5/6 you’re entering dangerous territory with BTRFS.

by stingraycharles

4/6/2026 at 5:45:32 PM

I had a metadata corruption in metadata raid1c3 (raid1, 3 copies) over 4 disks. It happened after an unplanned power loss during a simulated disk failure replacement. Since manual cleanup of the filesystem metadata (list all files, get IO errors, delete IO errored files), the btrfs kernel driver segfaults in kernel space on any scrub or device replacment attenpt.

Honestly the code of btrfs is a bit scary to read too. I have lost all trust in this filesystem.

Too bad because btrfs has pretty compelling features.

by bombela

4/6/2026 at 7:03:02 AM

Unless I missed it the writeup never identifies a causal bug, only things that made recovery harder.

by Retr0id

4/6/2026 at 10:01:44 AM

Using DUP as the metadata profile sounds insane.

Changing the metadata profile to at least raid1 (raid1, raid1c3, raid1c4) is a good idea, especially for anyone, against recommendations, using raid5 or raid6 for a btrfs array (raid1c3 is more appropriate for raid6). That would make it very difficult for metadata to get corrupted, which is the lion's share of the higher-impact problems with raid5/6 btrfs.

check:

    btrfs fi df <mountpoint>

convert metadata:

    btrfs balance start -mconvert=raid1c3,soft <mountpoint>

(make sure it's -mconvert — m is for metadata — not -dconvert which would switch profiles for data, messing up your array)

by harshreality

4/6/2026 at 6:07:56 PM

I still got corrupted metadata with metadata raid1c3 on btrfs on a power loss. I never had this happen with ext4 alone or atop Linux raid.

I want to be clear that losing (meta)data in flight during a power loss is expected. But a broken filesystem after that is definitely not acceptable.

Some postgresql db endedup soft corrupted. Postgresql could not replay its log because btrfs threw IO errors on fsync. That's just plain not acceptable.

by bombela

4/6/2026 at 12:51:09 PM

This should be at the top, using metadata DUP on a 3 disk volume is already asking for it, and of course you loose data when you just use it as jbod with data stored only once. Unless this are enterprise disks with capacitors anything can happen when it suddenly looses power. Not the FSes fault.

With the same configuration this can happen with ZFS, bcachefs etc just as well.

by throwaway270925

4/6/2026 at 12:59:25 PM

Will it render the whole filesystem inaccessible and unrepairable on those filesystems as well? One of the issues with btrfs is that it's brittle: failure tends not to cause an inconsistency in the affected part of the filesystem but bring down the whole thing. In general people are a lot more understanding of a power failure resulting in data corruption around the files that are actively being written at the time (there are limits to how much consistency can be achieved here anyway), much less so when the blast radius expands a lot further.

by rcxdude

4/6/2026 at 2:44:55 PM

A few decades ago, XFS was notorious because a power failure would wipe out various files, even if they had been opened only for reading. For instance, I had seen many systems that were bricked because XFS wiped out /etc/fstab after a power failure.

Nevertheless, many, many years ago, the XFS problems have been removed and today it is very robust.

During the last few years, I have seen a great number of power failures on some computers without a UPS, where XFS was used intensively at the moment of the power failure. Despite that, in none of those cases there was any filesystem corruption whatsoever, but the worst that has ever happened was the loss of the last writes performed immediately before the power failure.

This is the behavior that is expected from any file system that claims to be journaled, even in the past many journaled file systems failed to keep their promises, e.g. a few decades ago I had seen corrupted file systems on all existing Linux file systems and also on NTFS. At that time only the FreeBSD UFS with "soft updates" was completely unaffected by any kind of power failures.

However, nowadays I would expect all these file systems to be much more mature and to have fixed any bugs long ago.

BTRFS appears to be the exception, as the stories about corruption events do not seem to diminish in time.

by adrian_b

4/7/2026 at 2:58:51 AM

> Unless this are enterprise disks with capacitors anything can happen when it suddenly looses power. Not the FSes fault.

Most filesystems just get a few files/directories damaged though. ZFS is famous for handling totally crazy things like broken hardware which damages data in-transit. ext4 has no checksum, but at least fsck will drop things into lost+found directory.

The "making all data inaccessible" part is pretty unique to btrfs, and lets not pretend nothing can be done about this.

by theamk

4/6/2026 at 12:58:36 PM

> A hard power cycle on a 3 device pool (data single, metadata DUP, DM-SMR disks) left the extent tree and free space tree in a state that no native repair path could resolve.

As a ZFS wrangler by day:

People in this thread seem to happily shit on btrfs here but this seems to be very much not like a sane, resilient configuration no matter the FS. Just something to keep in mind.

by throwaway270925

4/6/2026 at 3:04:09 PM

Might be true, but I don't see any aspect of that which is relevant to this event:

* Data single obviously means losing a single drive will cause data loss, but no drive was actually lost, right?

* Metadata DUP (not sure if it's across 2 disks or all 3) should be expected to be robust, I'd expect?

* I certainly eye DM-SMR disks with suspicion in general, but it doesn't sound like they were responsible for the damage: "Both DUP copies of several metadata blocks were written with inconsistent parent and child generations."

by scottlamb

4/6/2026 at 4:08:34 PM

> Metadata DUP (not sure if it's across 2 disks or all 3) should be expected to be robust, I'd expect?

No. DUP will happily put both copies on the same disk. You would need to use RAID1 (or RAID1c3 for a copy on all disks) if you wanted a guarantee of the metadata being on multiple disks.

by zootboy

4/6/2026 at 5:02:04 PM

Wow, yuck. (The "Why do we even have that lever?!" line comes to mind.)

...even so, without a disk failure, that probably wasn't the cause of this event.

by scottlamb

4/6/2026 at 5:27:18 PM

The DUP profile is meant for use with a single disk. The RAID* profiles are meant for use with multiple disks. Both are necessary to cover the full gamut of BTRFS use cases, but it would probably be good if mkfs.btrfs spat out a big warning if you use DUP on a multi-disk filesystem, as this is /usually/ a mistake.

by zootboy

4/6/2026 at 9:53:58 PM

[dead]

by scottlamb

4/7/2026 at 6:58:19 PM

ZFS has similar configurations possible (e.g. copies).

You can end up in this state with btrfs if you start with a single device (defaults to data=single,metadata=dup), and then add additional devices without changing the data/metadata profiles. Or you can choose this config explicitly.

I really wish the btrfs-progs had a --this-config-is-bad-but-continue-anyway flag since there are so many bad configurations possible (raid5/raid6, raid0/single/dup). The rescue tools are also bad and are about as likely to make the problem worse as fix it.

by singron

4/6/2026 at 6:47:41 AM

This is obviously LLM output, but perhaps LLM output that corresponds to a real scenario. It's plausible that Claude was able to autonomously recover a corrupted fs, but I would not trust its "insights" by default. I'd love to see a btrfs dev's take on this!

by Retr0id

4/6/2026 at 6:59:20 AM

This is also my first impulse. The second was, if this happened to me, I would not be able to recover it. All the custom c tool talk... If you ask Claude Code it will code something up.

Well that he recovered the disks is amazing in itself. I would have given up and just pulled a backup.

However, I would like to see a Dev saying: why didn't you use the --<flag> which we created for this Usecase

by number6

4/6/2026 at 3:17:35 PM

See this Reddit post for background: https://www.reddit.com/r/ClaudeAI/comments/1sdabux/hats_off_...

TLDR: The user got his filesystem corrupted on a forced reboot; native btrfs tools made the failure worse; the user asked Claude to autonomously debug and fix the problem; after multiple days of debugging, Claude wrote a set of custom low-level C scripts to recover 99.9% of the data; the user was impressed and asked Claude to submit an issue describing the whole thing.

by salt4034

4/6/2026 at 3:54:39 PM

Good to know that my claude-dar is still working.

by Retr0id

4/6/2026 at 7:00:46 AM

I was assuming real scenario with heavy LLM help to recover. Would be nice for the author to clarify. And, separately, for BTRFS devs to weigh in, though I'd somewhat prefer to get some indication that it's real before spending their time.

by yjftsjthsd-h

4/6/2026 at 7:54:50 AM

An LLM wouldn't make a mistake like "One paragraph summary"

by nslsm

4/6/2026 at 5:40:46 AM

> Case study: recovery of a severely corrupted 12 TB multi-device pool, plus constructive gap analysis and reference tool set #1107

Please don't be btrfs please don't be btrfs please don't be btrfs...

by stinkbeetle

4/6/2026 at 7:19:37 AM

I mean, the only other option was bcachefs, which might have been funny if this LLM-generated blogpost were written by the OpenClaw instance the developer has decided is sentient:

https://www.reddit.com/r/bcachefs/comments/1rblll1/the_blog_...

But no. It was btrfs.

As a side note, it's somewhat impressive that an LLM agent was able to produce a suite of custom tools that were apparently successfully used to recover some data from a corrupted btrfs array, even ad-hoc.

by toaste_

4/6/2026 at 7:28:58 AM

It could be ZFS. I'd be much more surprised, but it can still have bugs.

by yjftsjthsd-h

4/6/2026 at 7:54:28 AM

ZFS on Linux has had many bugs over the years, notably with ZFS-native encryption and especially sending/receiving encrypted volumes. Another issue is that using swap on ZFS is still guaranteed to hang the kernel in low memory scenarios, because ZFS needs to allocate memory to write to swap.

by praseodym

4/6/2026 at 12:42:28 PM

The swap issue isn't zfs' fault though, it works just fine on FreeBSD and illumos... its a issue with how the Linux kernel handles things.

by nubinetwork

4/6/2026 at 8:24:50 AM

The zero copy that zero copied unencrypted blocks onto encrypted file systems was genius. It’s almost like they don’t test.

by badgersnake

4/6/2026 at 11:15:10 AM

Where are all of the ZFS corruption stories? Or are there simply fewer of those?

by curt15

4/6/2026 at 11:23:36 AM

Not sure about the stats, but it does feel like there are fewer. So from what I know encryption and sending fs state had bugs in ZFS.

And on btrfs anything above raid1 (5,6 etc) has had very serious bugs. Actually read an opinion somewhere (don't remember where) raid5,6 on btrfs cannot work due to on-disk format being just bad for the case. I guess this is why raid1c3/c4 is being promoted and worked on now?

by Neikius

4/7/2026 at 12:18:28 AM

I love ZFS, but have been corrected a couple of times when I said it was bomb proof. Can't remember the details, but it has served me faithfully for 10 years or so? Plus the bugs were pretty niche if I recall correctly.

Edit: found some comments below: ZFS on Linux has had many bugs over the years, notably with ZFS-native encryption and especially sending/receiving encrypted volumes. Another issue is that using swap on ZFS is still guaranteed to hang the kernel in low memory scenarios, because ZFS needs to allocate memory to write to swap.

by abrookewood

4/6/2026 at 12:39:25 PM

Most of them are from new features that didn't get a proper shakedown test, like encryption.

by nubinetwork

4/6/2026 at 11:06:00 AM

People swear btrfs is "safe" now, but I've personally been bitten by data corruption more than once, so I stay away from it now.

by jamesnorden

4/6/2026 at 1:43:27 PM

I installed Fedora on BTRFS using their installer and I lost that partition entirely. Couldn't wrestle it back to life to even copy stuff off it.

I think what happened was that the machine ran out of battery in suspend, but an unclean shutdown shouldn't cause such a deep corruption.

by Avamander

4/6/2026 at 10:17:00 AM

Added to my list of reasons to never use btrfs in production.

by c-c-c-c-c

4/6/2026 at 6:51:43 PM

Btrfs allows migration from ext4 with a rather good rollback strategy...

Post-migration, a complete disk image of the original ext4 disk will exist within the new filesystem, using no additional disk space due to the magic of copy-on-write.

Why isn't the repair process the same? Fix the filesystem to get everything online asap, and leave a complete disk image of the old damaged filesystem so other recovery processes can be tried if necessary.

by londons_explore

4/6/2026 at 12:18:18 PM

Welp. Guess I need to figure out another fs to use for a few drives in a nonraid pool I haven't gotten around to setting up yet. I forget why zfs seemed out. xfs?

by duskdozer

4/6/2026 at 1:24:11 PM

ZFS is out because the Linux developers refuse to cooperate by providing the hooks it would need to avoid duplicating the disk cache.

That’s the only real reason. There are some papercuts, but they don’t compare to the risks described in this article.

by Filligree

4/6/2026 at 5:35:44 AM

To theal author: did you continue using btrfs after this ordeal? An FS that will not eat (all) your data upon a hard powercycle only at the cost of 14 custom C tools is a hard pass from me no matter how many distros try to push it down my throat as 'production-ready'...

Also, impressive work!

by phoronixrly

4/6/2026 at 8:45:09 AM

What are the alternatives to btrfs? At 12 TB data checksums are a must unless the data tolerate bit-rot. And if one wants to stick with the official kernel without out-of-tree modules, btrfs is the only choice.

by fpoling

4/6/2026 at 9:32:09 AM

I tried btrfs on three different occasions. Three times it managed to corrupt itself. I'll admit I was too enthousiastic the first time, trying it less than a year after it appeared in major distros. But the latter two are unforgiveable (I had to reinstall my mom's laptop).

I've been using ZFS for my NAS-like thing since then. It's been rock solid ().

(): I know about the block cloning bug, and the encryption bug. Luckily I avoided those (I don't tend to enable new features like block cloning, and I didn't have an encrypted dataset at the time). Still, all in all it's been really good in comparison to btrfs.

by aktau

4/6/2026 at 4:56:08 PM

Additional anecdata:

I've been using btrfs as the primary FS for my laptop for nearly twenty years, and for my desktop and multipurpose box for as long as they've existed (~eight and ~three years, respectively). I haven't had troubles with the laptop FS in like fifteen years, and have never had troubles with the desktop or multipurpose box.

I also used btrfs as the production FS for the volume management in our CI at $DAYJOB, as it was way faster than overlayfs. No problems there, either.

Go figure, I guess.

by simoncion

4/6/2026 at 9:28:36 AM

Could try ZFS or CephFS... even if several host roles are in VM containers (45Drives has a product setup that way.)

The btrfs solution has a mixed history, and had a lot of the same issues DRBD could get. They are great until some hardware/kernel-mod eventually goes sideways, and then the auto-heal cluster filesystems start to make a lot more sense. Note, with cluster based complete-file copy/repair object features the damage is localized to single files at worst, and folks don't have to wait 3 days to bring up the cluster on a crash.

Best of luck, =3

by Joel_Mckay

4/6/2026 at 9:18:57 AM

> if one wants to stick with the official kernel without out-of-tree modules

I wonder how could a requirement like that possibly arise. Especially with an obvious exception for zfs.

by egorfine

4/6/2026 at 9:34:28 AM

Bcachefs also fulfills the requirement of checksums (and multi device support).

Also out of tree.

by ThatPlayer

4/6/2026 at 11:26:34 AM

Isn't bcachefs even younger and less polished than btrfs? It does show more promise as btrfs seems to have fundamental design issues... but still I wouldn't use that for my important data.

by Neikius

4/6/2026 at 12:15:24 PM

I don't disagree. Gotta backups for important data either way too!

Just talking about filesystems with checksumming (and multidevice). Any new filesystem to support these features is going to be newer.

I've had both btrfs and bcachefs multidevice filesystems lock up read-only on me. So no real data loss, just a pain to get the data into a new file system, the time it was an 8 drive array on btrfs.

by ThatPlayer

4/6/2026 at 9:35:52 AM

Does it not also eat data though?

by phoronixrly

4/6/2026 at 5:21:07 PM

I think you could use dm-integrity over the raw disks to have checksums and protect against bitrot then you can use mdraid to make a RAID1/5/6 of the virtual blockdevs presented by dm-integrity.

I suspect this is still vulnerable to the write hole problem.

You can add LVM to get snapshots, but this still not an end-to-end copy-on-write solution that btrfs and ZFS should provide.

by raron

4/6/2026 at 9:35:13 AM

lvm offers lvmraid, integrity, and snapshots as one example. It's old unsexy tech, but losing data is not to my taste lately...

by phoronixrly

4/6/2026 at 11:16:36 AM

lvm only supports checksums for metadata. It does not checksum the data itself. For checksums with arbitrary filesystems one can have dm-integrity device rather than LVM. But the performance suffer due to separated journal writes by the device.

by fpoling

4/6/2026 at 11:41:09 AM

I said lvmraid[1][2].

[1] https://www.man7.org/linux/man-pages/man7/lvmraid.7.html#DAT...

[2] https://docs.redhat.com/en/documentation/red_hat_enterprise_...

by phoronixrly

4/6/2026 at 4:28:25 PM

But that is just raid on top of dm-integrity. And Redhat docs omits an important part when suggesting to use the bitmap mode with dm-integrity:

man 8 integritysetup:

       --integrity-bitmap-mode. -B
           Use alternate bitmap mode (available since Linux kernel 5.2) where dm-integrity uses bitmap instead of a journal. If a bit in the bitmap is 1, then corresponding region’s data and integrity tags are not synchronized - if the machine crashes, the unsynchronized regions will be recalculated. The bitmap mode is faster than the journal mode, because we don’t have to write the data twice, but it is also less reliable, because if data corruption happens when the machine crashes, it may not be detected.

I just do not see how without a direct filesystem support one can have both reliable checksums and performance.

by fpoling

4/6/2026 at 10:33:59 PM

> But that is just raid on top of dm-integrity

As I said -- boring tech. Just what I like when not in a mood to lose data.

by phoronixrly

4/6/2026 at 11:42:48 AM

Good thing all disks these days have data checksums, then!

(50TB+ on ext4 and xfs, and no, no bit rot. Yes, I've checked most of it against separate sha256sum files now and then. As long as you have ECC RAM, disks just magically corrupting your data is largely a myth.)

by Sesse__

4/6/2026 at 2:24:10 PM

Less mythic on SSDs than spinning rust, in my experience.

Not particularly frequent either way, but I have absolutely had models of SSDs where it became clear after a few months of use that a significant fraction of them appeared to be corrupting their internal state and serving incorrect data back to the host, leading to errors and panics.

(_usually_ this was accompanied by read or write errors. But _usually_ is notable when you've spent some time trying to figure out if the times it didn't were a different problem or the same problem but silent.)

There was also the notorious case with certain Samsung spinning rust and dropping data in their write cache if you issued SMART requests...

by rincebrain

4/6/2026 at 10:45:57 AM

What devices are you talking about, what's the UBER, over what period of time?

RAID and logical block redundancy has scaled to petabytes for years in serious production use, before btrfs was even developed.

by stinkbeetle

4/6/2026 at 4:19:53 PM

bulletproof/bulletproof/bulletproof .... Gemini LLM

by lnx01

4/6/2026 at 2:36:01 PM

[dead]

by devnotes77

4/6/2026 at 8:53:47 AM

[dead]

by weiyong1024

4/6/2026 at 2:59:38 PM

oh great here comes all the zfs fanboys to shit on btrfs again with made up stories of corruption

by blae

4/6/2026 at 3:50:40 PM

Why would you assume that people are making up reports of corruption? Is it really inconceivable to you that the thing could have bugs?

by yjftsjthsd-h

4/6/2026 at 5:21:29 PM

> Is it really inconceivable to you that the thing could have bugs?

Or user error, or hardware setups where the docs didn't say "don't do that". If zfs is somehow better in any of those three areas, that would result in fewer corruption stories as well. Hard to know without being able to control for popularity though

Seems really weird to me to assume people make up stories to promote their favorite filesystem. Of course I have one to share as well (opened the thread without knowing it was about btrfs to begin with, I'm not brigading...)

---

I tried btrfs once in my life. I wanted to (1) mirror two disks so a routine disk failure doesn't mean I lose X hours of updates since the last off-site backup, and (2) detect bit rot. And of course it resulted in a giant headache:

The disks got out of sync, put themselves in read-only mode with different data on each (which one has the latest data? Do they both have new fragments?), I eventually figured out which one has the latest data, and I mix up the source and destination device in the recovery command. Iirc the latter was caused by me stopping to read the man page when I found the info I was after and didn't read the whole thing carefully, where subsequent text would have clued me in

The recovery mess-up is user error but if this happens to people on btrfs more often than zfs, maybe zfs is more recommendable anyway. But I've not tried zfs so that's not a statement I can make

I'm back to ext4. Will just use backups and hope for the best. This constant risk of full-filesystem corruption isn't worth it to catch the few files that changed in the last hours, or the few bytes that will rot over my lifetime. On my todo list is writing a little tool that just stores sha2sum+mtime for each file and alerts me if the former changed without the latter, then I can retrieve it from backup and perhaps swap out the disk

by Aachen

4/7/2026 at 12:21:47 AM

I am a complete ZFS fanboy, but that doesn't mean I want BTRFS to fail. Competition is healthy and I don't think Oracle is going to change the licence on ZFS any time soon, so it's good to have an option that can be used without encumbrance.

by abrookewood