I will pay money for the correct RAID recovery instructions

[Ian Young]

I've been trying to fix a degraded array for a couple of months now
and it's getting frustrating enough that I'm willing to put a bounty
on the correct solution. The array can start in a degraded state and
the data is accessible, so I know this is possible to fix. Any
takers? I'll bet someone could use some beer money or a contribution
to their web hosting costs.

Here's how the system is set up: There are (6) 3 TB drives. Each
drive has a BIOS boot partition. The rest of the space on each drive
is a large GPT partition that is combined in a RAID 10 array. On top
of the array there are four LVM volumes: /boot, /root, swap, and /srv.

Here's the problem: /dev/sdf failed. I replaced it but as it was
resyncing, read errors on /dev/sde kicked the new sdf out and made it
a spare. The array is now in a precarious degraded state. All it
would take for the entire array to fail is for /dev/sde to fail, and
it's already showing signs that it will. I have tried forcing the
array to assemble using /dev/sd[abcde]2 and then forcing it to add
/dev/sdf2. That still adds sdf2 as a spare. I've tried "echo check >
/sys/block/md0/md/sync_action" but that finishes immediately and
changes nothing.

Can anyone solve this? I'd be happy to pay you for your knowledge.
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to ***@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html

[Robin Hill]

Post by Ian Young
I've been trying to fix a degraded array for a couple of months now
and it's getting frustrating enough that I'm willing to put a bounty
on the correct solution. The array can start in a degraded state and
the data is accessible, so I know this is possible to fix. Any
takers? I'll bet someone could use some beer money or a contribution
to their web hosting costs.
Here's how the system is set up: There are (6) 3 TB drives. Each
drive has a BIOS boot partition. The rest of the space on each drive
is a large GPT partition that is combined in a RAID 10 array. On top
of the array there are four LVM volumes: /boot, /root, swap, and /srv.
Here's the problem: /dev/sdf failed. I replaced it but as it was
resyncing, read errors on /dev/sde kicked the new sdf out and made it
a spare. The array is now in a precarious degraded state. All it
would take for the entire array to fail is for /dev/sde to fail, and
it's already showing signs that it will. I have tried forcing the
array to assemble using /dev/sd[abcde]2 and then forcing it to add
/dev/sdf2. That still adds sdf2 as a spare. I've tried "echo check >
/sys/block/md0/md/sync_action" but that finishes immediately and
changes nothing.

If sdf didn't finish syncing then it's no use adding it to the array as
anything other than a spare. Also, you can't run a check on a degraded
array (as there's nothing to check against), which is why that's
finishing immediately.

If sde is giving a read error during rebuild then the solution is to
stop the array (you'll need to do this via a bootable CD/USB stick I
guess) and use ddrescue to duplicate sde onto a new disk, The
read errors may well mean that some can't be copied (though ddrescue
will try very hard to do so), which may cause file/filesystem corruption
later. You can then reassemble the (degraded) array with the old sda-sdd
and the new sde, then add sdf and wait for the array to recover. You
can then run a fsck on the filesystem to check for any corruption there.
File corruption is a lot trickier to spot - if you have checksums for
the files then that's one way, otherwise you may be able to work out
what files are affected based on the offsets of the missing data (that's
rather beyond the limits of my knowledge though).

HTH,
Robin

[Ian Young]

Ok, if I can pull this off I owe you a beer.

Post by Robin Hill

Post by Ian Young
I've been trying to fix a degraded array for a couple of months now
and it's getting frustrating enough that I'm willing to put a bounty
on the correct solution. The array can start in a degraded state and
the data is accessible, so I know this is possible to fix. Any
takers? I'll bet someone could use some beer money or a contribution
to their web hosting costs.
Here's how the system is set up: There are (6) 3 TB drives. Each
drive has a BIOS boot partition. The rest of the space on each drive
is a large GPT partition that is combined in a RAID 10 array. On top
of the array there are four LVM volumes: /boot, /root, swap, and /srv.
Here's the problem: /dev/sdf failed. I replaced it but as it was
resyncing, read errors on /dev/sde kicked the new sdf out and made it
a spare. The array is now in a precarious degraded state. All it
would take for the entire array to fail is for /dev/sde to fail, and
it's already showing signs that it will. I have tried forcing the
array to assemble using /dev/sd[abcde]2 and then forcing it to add
/dev/sdf2. That still adds sdf2 as a spare. I've tried "echo check >
/sys/block/md0/md/sync_action" but that finishes immediately and
changes nothing.

If sdf didn't finish syncing then it's no use adding it to the array as
anything other than a spare. Also, you can't run a check on a degraded
array (as there's nothing to check against), which is why that's
finishing immediately.
If sde is giving a read error during rebuild then the solution is to
stop the array (you'll need to do this via a bootable CD/USB stick I
guess) and use ddrescue to duplicate sde onto a new disk, The
read errors may well mean that some can't be copied (though ddrescue
will try very hard to do so), which may cause file/filesystem corruption
later. You can then reassemble the (degraded) array with the old sda-sdd
and the new sde, then add sdf and wait for the array to recover. You
can then run a fsck on the filesystem to check for any corruption there.
File corruption is a lot trickier to spot - if you have checksums for
the files then that's one way, otherwise you may be able to work out
what files are affected based on the offsets of the missing data (that's
rather beyond the limits of my knowledge though).
HTH,
Robin
--
___
/ / ) | Little Jim says .... |
// !! | "He fallen in de water !!" |

--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to ***@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html

[John Stoffel]

Ian,

It would also help if you posted the details of your setup using:

cat /proc/partitions
cat /proc/mdstat

mdadm -D /dev/md#
- for each of the devices above.

mdadm -E /dev/sd<drive><#>
- for each disk or partition in the array from above.



But the suggestions to ddrescue the going bad drive onto a new disk is
a good one. On my debian system, I would do the following:

sudo apt-get install gddrescue
ddrescue /dev/sde /dev/sdf /var/tmp/ddrecue-sde.log

and see how that goes.

Good luck,
John
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to ***@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html

[Another Sillyname]

Sorry to state the obvious but......

To restore the degraded array would (based on the info you've posted)
likely take longer then temporarily moving the data to a different set
of drives.

As time seems to be a major consideration here (likely/possible
failure of sde) then surely the optimal strategy has to be to get the
data off first, then look at the rebuilding the degraded array?

Just my 2c

Post by John Stoffel
Ian,
cat /proc/partitions
cat /proc/mdstat
mdadm -D /dev/md#
- for each of the devices above.
mdadm -E /dev/sd<drive><#>
- for each disk or partition in the array from above.
But the suggestions to ddrescue the going bad drive onto a new disk is
sudo apt-get install gddrescue
ddrescue /dev/sde /dev/sdf /var/tmp/ddrecue-sde.log
and see how that goes.
Good luck,
John
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
More majordomo info at http://vger.kernel.org/majordomo-info.html

--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to ***@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html

[John Stoffel]

Another> Sorry to state the obvious but...... To restore the degraded
Another> array would (based on the info you've posted) likely take
Another> longer then temporarily moving the data to a different set of
Another> drives.

How do you figure this? He hasn't posted any useful data on his array
at all. We know it's six 3Tb disks in RAID10, so that's at most 9Tb
of data. Moving 3Tb (bad disk ddrescue'd onto new disk) is only 3Tb
to move. Then the resync will of course take time.

But if he doesn't have the spare disk space, it's probably the only way.

Another> As time seems to be a major consideration here (likely/possible
Another> failure of sde) then surely the optimal strategy has to be to get the
Another> data off first, then look at the rebuilding the degraded array?

Another> Just my 2c

Post by John Stoffel
Ian,
cat /proc/partitions
cat /proc/mdstat
mdadm -D /dev/md#
- for each of the devices above.
mdadm -E /dev/sd<drive><#>
- for each disk or partition in the array from above.
But the suggestions to ddrescue the going bad drive onto a new disk is
sudo apt-get install gddrescue
ddrescue /dev/sde /dev/sdf /var/tmp/ddrecue-sde.log
and see how that goes.
Good luck,
John
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
More majordomo info at http://vger.kernel.org/majordomo-info.html

Another> --
Another> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
Another> the body of a message to ***@vger.kernel.org
Another> More majordomo info at http://vger.kernel.org/majordomo-info.html
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to ***@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html