I love linux software ("md") raid. I use md raid1 on a zillion systems. I never has issues. Until today...
I get a call that a customer has lost all their emails for about a month and their apps' data appears to be old and/or missing. Strange.
I login to the linux server and see:
cat /proc/mdstat Personalities : [raid1] md122 : active raid1 sda1[0] 409536 blocks [2/1] [U_]
md123 : active raid1 sda2[0] 5242816 blocks [2/1] [U_]
md124 : active raid1 sda3[0] 1939865536 blocks [2/1] [U_]
md125 : active raid1 sdb1[1] 409536 blocks [2/1] [_U]
md126 : active raid1 sdb2[1] 5242816 blocks [2/1] [_U]
md127 : active raid1 sdb3[1] 1939865536 blocks [2/1] [_U]
That's not correct. These systems should have 3 partitions, not 6. Ah, md has done some really goofball things with this pathological case. It's separated the raid into duplicates and assembled each separately! Woah!
They said they had a accidental reboot today (kid hitting reset button). And it booted/rooted off the wrong schizo set (sda).
There appears to have been a drive failure/kick a month ago: Apr 4 10:10:32 firewall kernel: [1443781.218260] md/raid1:md127: Disk failure on sda3, disabling device. Apr 4 10:10:32 firewall kernel: [1443781.218262] <1>md/raid1:md127: Operation continuing on 1 devices.
And it hadn't rebooted since then, before today.
It gets stranger... I rebooted the system trying to test a few recovery ideas (offsite) out. On the next reboot it came up using the good/current sdb drive for boot/root! Huh? It's like it's picking which one to use at random! It still shows 6 md arrays, but it's using the properly 3 this time.
So is all this a bug?
1. Shouldn't the system have marked the sda as failed/bad PERMANENTLY so on next reboot it would ignore it. OK, I can understand that if it thought the whole drive was bad, it wouldn't be able to write to the sda superblock to survive the reboot. But couldn't it have written the info to sdb's superblock? If a system can't remember what has failed, then I don't see how this behaviour can be avoided.
2. Why did linux md bring up both sets of arrays? It can see they are the same array. Why on earth would it ever split them? That seems majorly screwy to me.
Still, thank God it didn't try to start syncing the stale set to the good set! We had backups, but it's a pain to recover. In the end, just rebooting until luck gives us the current set was all it took. I'll head on-site to replace the bad disk and do a proper resync.
I have had hardware RAID systems (ARAID99) in this exact situation go into a schizo state where the disks were unsynched yet both were being used for writes! The problems always seem to revolve around a disk going "soft" bad and then coming alive after reboot.