Discussion:
UEFI and mdadm questions.
Wilson, Jonathan
2014-10-01 16:33:15 UTC
Permalink
From what I can tell with UEFI I need to set up a UEFI partition with a
FAT format.

On my current BIOS system I have a Biosboot 1M, /boot Raid1 200M and /
Raid 1 40G.

Obviously Grub installs to the mbr, and then installs a bit into
Biosboot which can read raids, hence it can read and boot from /boot.

Further, from what I can tell, into the UEFI partition can go either a
kernel & initramfs with UEFI support, or a "loader" that then loads the
kernel.

What I am unsure about are...

1) Can the loader/kernel understand md raid? so the / can be in a bog
standard raid1 v1.2?

2) I'm guessing I would no longer need the /boot as that would be
replaced by what ever was in the UEFI partition?

3) as my "/boot" is currently in a raid 1 my life is simple, should any
changes occur they are replicated to the drives I have set up as /boot
raid 1, and I installed the mbr portion of boot loader manually on each
disk and tested pulling one, and then booting from another.. it
worked :-)
So I would like to keep things, if not simple, at least less likely to
have problems because I forgot to install duplicates of the UEFI on all
the disks... so can I use a .90v raid1 on the UEFI partition, then
format it as fat... so that all copies of what ever is in the UEFI
partition are replicated across the multiple raid1 disks, instead of
having to remember to copy what ever is in there manually to ach disk?

Jon.

--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to ***@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Anthonys Lists
2014-10-02 00:17:55 UTC
Permalink
On 01/10/2014 17:33, Wilson, Jonathan wrote:

I've just been struggling with exactly this trying to move my gentoo
system on to raid ...
From what I can tell with UEFI I need to set up a UEFI partition with a
FAT format.
I believe so, yes.
On my current BIOS system I have a Biosboot 1M, /boot Raid1 200M and /
Raid 1 40G.
Obviously Grub installs to the mbr, and then installs a bit into
Biosboot which can read raids, hence it can read and boot from /boot.
Is this grub1, or grub2? I gather grub1 actually CAN'T read raid, which
is why you need to use raid version 0.9. Grub then thinks the disk is
plain ext4 or whatever, and reads it fine. grub2 actually handles raid,
and therefore is happy with newer raids like 1.2
Further, from what I can tell, into the UEFI partition can go either a
kernel & initramfs with UEFI support, or a "loader" that then loads the
kernel.
What I am unsure about are...
1) Can the loader/kernel understand md raid? so the / can be in a bog
standard raid1 v1.2?
Don't think so. Apparently the kernel can NOT put a raid array together
so you have a catch-22 - the kernel needs the raid in order to start
user-space, but it needs user-space in order to find the raid ... :-(

So you need an initramfs to solve the conundrum. And of course, grub1
doesn't work with uefi :-(
2) I'm guessing I would no longer need the /boot as that would be
replaced by what ever was in the UEFI partition?
You do need /boot - not least so grub2 can find its config file (that's
not quite true, but close enough...)
3) as my "/boot" is currently in a raid 1 my life is simple, should any
changes occur they are replicated to the drives I have set up as /boot
raid 1, and I installed the mbr portion of boot loader manually on each
disk and tested pulling one, and then booting from another.. it
worked :-)
In the world of grub2/raid1.2 I wish things looked that simple. From my
"badly burned novice" viewpoint, I don't think it is.
So I would like to keep things, if not simple, at least less likely to
have problems because I forgot to install duplicates of the UEFI on all
the disks... so can I use a .90v raid1 on the UEFI partition, then
format it as fat... so that all copies of what ever is in the UEFI
partition are replicated across the multiple raid1 disks, instead of
having to remember to copy what ever is in there manually to ach disk?
Interesting idea ... but I get the impression that once you've set up
the uefi partition and got it working, you shouldn't normally ever need
to go back to it. This is, I think, one of those cases you need to set
up a crib sheet and not do anything without it, as you'll forget from
one time to the next (which is why, I think, you want to use raid :-)
Personally, I *wouldn't* want to use raid, on the assumption that I'm
going to screw it up and don't want a mirror trashing disk2 as I trash
disk1.

Cheers,
Wol
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to ***@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Chris Murphy
2014-10-03 05:04:05 UTC
Permalink
From what I can tell with UEFI I need to set up a UEFI partition with=
a
FAT format.
It's a particular kind of FAT, that's defined as EFI FAT, the idea bein=
g that if the originating FAT ever changes, EFI FAT won't.
=20
On my current BIOS system I have a Biosboot 1M, /boot Raid1 200M and =
/
Raid 1 40G.
=20
Obviously Grub installs to the mbr, and then installs a bit into
Biosboot which can read raids, hence it can read and boot from /boot.
BIOSboot applies to GPT disks on BIOS computers, not MBR. On MBR disks,=
the GRUB stage1 code jumps to stage2 code in the MBR gap which is the =
region between the MBR and the first partition's starting LBA.
=20
Further, from what I can tell, into the UEFI partition can go either =
a
kernel & initramfs with UEFI support, or a "loader" that then loads t=
he
kernel.
No. An OSLoader is required, it's an EFI application. Its job is to loa=
d a kernel and initramfs. The kernel and initramfs could be on the ESP =
(EFI System partition) but this is fraught with limitations. The expect=
ation is that the kernel and initramfs are on some other partition of t=
he same disk. Of course if you're using GRUB it doesn't care and will f=
ind a kernel/initramfs off another disk also, or even off md raid.

In the case of GRUB, it directly understands pretty much every filesyst=
em used on linux, so it can read kernel+initramfs from ext4, or ext4 on=
mdraid, or on LVM, or on Btrfs (including multiple devices).
=20
What I am unsure about are...=20
=20
1) Can the loader/kernel understand md raid? so the / can be in a bo=
g
standard raid1 v1.2?
If the EFI OSLoader is GRUB2, then yes.
=20
2) I'm guessing I would no longer need the /boot as that would be
replaced by what ever was in the UEFI partition?
The ESP isn't a replacement for /boot. But you can have a unified /boot=
and / on a single file system, and GRUB2 can boot from it, including i=
f that volume is on an md device. GRUB2 even boots off md raid6 (it's s=
orta crazy and badass at the same time, but in my testing it does work =
with the BIOS being the limiting factor as to how many drives are recog=
nized at this stage).
=20
3) as my "/boot" is currently in a raid 1 my life is simple, should a=
ny
changes occur they are replicated to the drives I have set up as /boo=
t
raid 1, and I installed the mbr portion of boot loader manually on ea=
ch
disk and tested pulling one, and then booting from another.. it
worked :-)
So I would like to keep things, if not simple, at least less likely t=
o
have problems because I forgot to install duplicates of the UEFI on a=
ll
the disks=85 so can I use a .90v raid1 on the UEFI partition,
That solution isn't that simple because it requires too many things tha=
t aren't supported by any distribution. And also it increases the chanc=
es of corruption because of course the EFI firmware only sees these par=
tition/volumes as single devices, not as md devices that need to be ass=
embled first into a logical device. Some EFI firmwares do reportedly wr=
ite to the ESP, so if they write to one and not the other, while also n=
ot changing the metadata on either partition, there's no way to know wh=
ich one is valid now. So that's just a mess. And even if your firmware =
doesn't write to the ESP ever, what about dual boot? That too would be =
proscribed because the ESP is shared among multiple OS's. So the idea i=
s just way off the rails of anything standardized or widely supported. =
Will it work? Yes. But do you want to support something this non-standa=
rd?

An alternative is a single static grub.cfg on the EFI System partition =
that points to the real grub.cfg on an mdraid1 / or /boot. Since the ES=
P's are identical, and never need updating again, this is much safer. Y=
ou can also dispense with this ridiculously bad idea most all linux dis=
tros have right now of persistently mounting the ESP at /boot/efi (and =
persistently mount it rw as well! really bad!).

The way to do this is with the grub configfile command. The most straig=
htforward way to do it is let grub-mkconfig create a grub.cfg for you, =
and then copy past the parts that tell GRUB how to find /. For example =
it needs to know the mduuid so it knows there's an mdraid volume to ass=
embled, and also the volume uuid so it knows what filesystem volume it =
should find the file one, and then "configfile /boot/grub/grub.cfg" and=
it'll load that grub.cfg. And it's that grub.cfg that your system shou=
ld update - of course you have to figure out how your system knows to u=
pdate the grub.cfg so that it does it correctly without stepping on you=
r ESP grub.cfg (for one comment out the /boot/efi line in fstab and the=
n unmount the ESP). This is distribution specific.

Either way it's a bit complicated to figure out initially.


Chris Murphy

--
To unsubscribe from this list: send the line "unsubscribe linux-raid" i=
n
the body of a message to ***@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Phil Turmel
2014-10-05 18:18:46 UTC
Permalink
On Oct 1, 2014, at 12:33 PM, "Wilson, Jonathan"
From what I can tell with UEFI I need to set up a UEFI partition
with a FAT format.
It's a particular kind of FAT, that's defined as EFI FAT, the idea
being that if the originating FAT ever changes, EFI FAT won't.
On my current BIOS system I have a Biosboot 1M, /boot Raid1 200M
and / Raid 1 40G.
Obviously Grub installs to the mbr, and then installs a bit into
Biosboot which can read raids, hence it can read and boot from /boot.
BIOSboot applies to GPT disks on BIOS computers, not MBR. On MBR
disks, the GRUB stage1 code jumps to stage2 code in the MBR gap which
is the region between the MBR and the first partition's starting
LBA.
Further, from what I can tell, into the UEFI partition can go
either a kernel & initramfs with UEFI support, or a "loader" that
then loads the kernel.
No. An OSLoader is required, it's an EFI application. Its job is to
load a kernel and initramfs. The kernel and initramfs could be on the
ESP (EFI System partition) but this is fraught with limitations. The
expectation is that the kernel and initramfs are on some other
partition of the same disk. Of course if you're using GRUB it doesn't
care and will find a kernel/initramfs off another disk also, or even
off md raid.
An option to consider is to compile a kernel using the EFI stub option,
a pre-set command line, and an embedded initramfs. Then the kernel can
boot directly from the EFI FAT partition with *no bootloader*. The
embedded initramfs can support any raid/lvm/partitioning scheme
what-so-ever.

http://kroah.com/log/blog/2013/09/02/booting-a-self-signed-linux-kernel/

The cryptography is optional if you just want to boot really fast.

If your BIOS can be configured to try multiple boot images, it should be
possible to have true raid fallback without using motherboard or
hardware raid. (Set up md raid1 with metadata v1.0 of multiple copies
of the EFI FAT partition.) I've been meaning to try this....

Phil
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to ***@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Chris Murphy
2014-10-05 20:22:59 UTC
Permalink
Post by Chris Murphy
=20
No. An OSLoader is required, it's an EFI application. Its job is to
load a kernel and initramfs. The kernel and initramfs could be on th=
e
Post by Chris Murphy
ESP (EFI System partition) but this is fraught with limitations. The
expectation is that the kernel and initramfs are on some other
partition of the same disk. Of course if you're using GRUB it doesn'=
t
Post by Chris Murphy
care and will find a kernel/initramfs off another disk also, or even
off md raid.
=20
An option to consider is to compile a kernel using the EFI stub optio=
n,
a pre-set command line, and an embedded initramfs. Then the kernel c=
an
boot directly from the EFI FAT partition with *no bootloader*. The
embedded initramfs can support any raid/lvm/partitioning scheme
what-so-ever.
Sure but then it means the kernel+initramfs is on FAT32. Ick. And there=
's no advantage if using maintstream storage options like md raid, lvm,=
btrfs, xfs, ext234, reiserfs, ufs, zfs, hfs+, ntfs and more, because G=
RUB2 already understands these things so the kernel can be located on t=
hem.

=46or appliances, there could be some advantage since there'd only be o=
ne kernel present at a time, no fallback necessary. My Android phone, f=
or example, has 35 partitions.
=20
If your BIOS can be configured to try multiple boot images, it should=
be
possible to have true raid fallback without using motherboard or
hardware raid. (Set up md raid1 with metadata v1.0 of multiple copie=
s
of the EFI FAT partition.) I've been meaning to try this=85.
Problems with this: a.) new Windows 8 hardware might require you boot W=
indows to get to the feature enabling the firmware setup, because on su=
ch hardware USB isn't initialized by default.
http://mjg59.dreamwidth.org/24869.html

I don't know why we don't have free software to initiate this, but I ha=
ven't come across it so far.

b.) There's no guarantee the firmware won't write to the ESP, thus rend=
ering the individual md raid members out of sync and without their meta=
data being updated, i.e. in effect, the logical device they become late=
r, is corrupt. Separately they aren't corrupt, merely out of sync, but =
you don't have an obvious way of knowing which one.

c.) strictly speaking any partition with mdadm metadata should have the=
linux raid partition type GUID set; not the EFI System partition type =
GUID. Those GUIDs are mutually exclusive.

This is why I'm still not a fan of using mdadm to raid1 an EFI System p=
artition.


Chris Murphy

--
To unsubscribe from this list: send the line "unsubscribe linux-raid" i=
n
the body of a message to ***@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Phil Turmel
2014-10-06 15:58:02 UTC
Permalink
Post by Chris Murphy
=20
[trim /]
Post by Chris Murphy
Post by Phil Turmel
If your BIOS can be configured to try multiple boot images, it
should be possible to have true raid fallback without using
motherboard or hardware raid. (Set up md raid1 with metadata v1.0
of multiple copies of the EFI FAT partition.) I've been meaning to
try this=E2=80=A6.
=20
Problems with this: a.) new Windows 8 hardware might require you boot
Windows to get to the feature enabling the firmware setup, because on
such hardware USB isn't initialized by default.
http://mjg59.dreamwidth.org/24869.html
=20
I don't know why we don't have free software to initiate this, but I
haven't come across it so far.
Good to know, but totally immaterial to the boot sequence I'm
recommending. Boot linux of of an EFI FAT and let linux initialize the
USB hardware in its own good time.

Matthew Garrett's post is really all about how to get linux into the
Win8 box in the first place. Once there, manipulate the boot sequence
as you please.
Post by Chris Murphy
b.) There's no guarantee the firmware won't write to the ESP, thus
rendering the individual md raid members out of sync and without
their metadata being updated, i.e. in effect, the logical device they
become later, is corrupt. Separately they aren't corrupt, merely out
of sync, but you don't have an obvious way of knowing which one.
This is a very good point. In fact, I withdraw my recommendation to
raid these partitions. Simply have one on every disk the BIOS could
possibly boot from, and place an EFI bootable kernel in each one (with
embedded initramfs).
Post by Chris Murphy
c.) strictly speaking any partition with mdadm metadata should have
the linux raid partition type GUID set; not the EFI System partition
type GUID. Those GUIDs are mutually exclusive.
The former is not true at all--mdadm does not care *at all* what
partition types are set. Grub might care, but it's moot if you don't
use Grub. :-)
Post by Chris Murphy
This is why I'm still not a fan of using mdadm to raid1 an EFI System
partition.
One further point: the failure decision tree is nicer if you boot
directly into a kernel.

1) Bios locates and attempts to boot from 1st configured kernel image
2a) Corrupted image or other disk error blocks complete load of kernel
image--bios moves to next EFI choice (possibly on a different disk).
2b) Successful EFI kernel load, boot encounters missing/corrupt root
=46S--kernel drops to initramfs rescue shell

versus:

1) Bios locates and attempts to boot from 1st configured grub image
2a) Corrupted image or other disk error blocks complete load of
grub--bios moves to next EFI choice (possibly on a different disk).
2b) Successful EFI grub load, grub encounters corrupt config or grub
module--drop to grub shell
2c) Successful EFI grub load, kernel & initramfs load by grub, boot
encounters missing/corrupt root FS--kernel drops to initramfs rescue sh=
ell

I haven't had time to set it up yet, but the clear reduction in points
of failure is compelling. Faster boot is just icing on the cake.

Phil
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" i=
n
the body of a message to ***@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Chris Murphy
2014-10-07 22:59:51 UTC
Permalink
This post might be inappropriate. Click to display it.
Francis Moreau
2014-10-06 08:43:57 UTC
Permalink
Hello Chris,
=20
=20
From what I can tell with UEFI I need to set up a UEFI partition wit=
h a
FAT format.
=20
It's a particular kind of FAT, that's defined as EFI FAT, the idea be=
ing that if the originating FAT ever changes, EFI FAT won't.
=20
On my current BIOS system I have a Biosboot 1M, /boot Raid1 200M and=
/
Raid 1 40G.
Obviously Grub installs to the mbr, and then installs a bit into
Biosboot which can read raids, hence it can read and boot from /boot=
=2E
=20
BIOSboot applies to GPT disks on BIOS computers, not MBR. On MBR disk=
s, the GRUB stage1 code jumps to stage2 code in the MBR gap which is th=
e region between the MBR and the first partition's starting LBA.
=20
Further, from what I can tell, into the UEFI partition can go either=
a
kernel & initramfs with UEFI support, or a "loader" that then loads =
the
kernel.
=20
No. An OSLoader is required, it's an EFI application. Its job is to l=
oad a kernel and initramfs. The kernel and initramfs could be on the ES=
P (EFI System partition) but this is fraught with limitations. The expe=
ctation is that the kernel and initramfs are on some other partition of=
the same disk. Of course if you're using GRUB it doesn't care and will=
find a kernel/initramfs off another disk also, or even off md raid.
=20
I think the kernel can be compiled as an EFI application, therefore
OSLoader is not always required. The advantage of this is that you can
get rid of grub entirely (cool !). But the downside is that you can't
customize/edit the kernel command line at boot time.
In the case of GRUB, it directly understands pretty much every filesy=
stem used on linux, so it can read kernel+initramfs from ext4, or ext4 =
on mdraid, or on LVM, or on Btrfs (including multiple devices).
=20
What I am unsure about are...=20
1) Can the loader/kernel understand md raid? so the / can be in a b=
og
standard raid1 v1.2?
=20
If the EFI OSLoader is GRUB2, then yes.
=20
=20
2) I'm guessing I would no longer need the /boot as that would be
replaced by what ever was in the UEFI partition?
=20
The ESP isn't a replacement for /boot. But you can have a unified /bo=
ot and / on a single file system, and GRUB2 can boot from it, including=
if that volume is on an md device. GRUB2 even boots off md raid6 (it's=
sorta crazy and badass at the same time, but in my testing it does wor=
k with the BIOS being the limiting factor as to how many drives are rec=
ognized at this stage).
=20
=20
=20
3) as my "/boot" is currently in a raid 1 my life is simple, should =
any
changes occur they are replicated to the drives I have set up as /bo=
ot
raid 1, and I installed the mbr portion of boot loader manually on e=
ach
disk and tested pulling one, and then booting from another.. it
worked :-)
So I would like to keep things, if not simple, at least less likely =
to
have problems because I forgot to install duplicates of the UEFI on =
all
the disks=85 so can I use a .90v raid1 on the UEFI partition,
=20
That solution isn't that simple because it requires too many things t=
hat aren't supported by any distribution. And also it increases the cha=
nces of corruption because of course the EFI firmware only sees these p=
artition/volumes as single devices, not as md devices that need to be a=
ssembled first into a logical device. Some EFI firmwares do reportedly =
write to the ESP, so if they write to one and not the other, while also=
not changing the metadata on either partition, there's no way to know =
which one is valid now. So that's just a mess. And even if your firmwar=
e doesn't write to the ESP ever, what about dual boot? That too would b=
e proscribed because the ESP is shared among multiple OS's. So the idea=
is just way off the rails of anything standardized or widely supported=
=2E Will it work? Yes. But do you want to support something this non-st=
andard?
=20
An alternative is a single static grub.cfg on the EFI System partitio=
n that points to the real grub.cfg on an mdraid1 / or /boot. Since the =
ESP's are identical, and never need updating again, this is much safer.=
You can also dispense with this ridiculously bad idea most all linux d=
istros have right now of persistently mounting the ESP at /boot/efi (an=
d persistently mount it rw as well! really bad!).
=20
The way to do this is with the grub configfile command. The most stra=
ightforward way to do it is let grub-mkconfig create a grub.cfg for you=
, and then copy past the parts that tell GRUB how to find /. For exampl=
e it needs to know the mduuid so it knows there's an mdraid volume to a=
ssembled, and also the volume uuid so it knows what filesystem volume i=
t should find the file one, and then "configfile /boot/grub/grub.cfg" a=
nd it'll load that grub.cfg. And it's that grub.cfg that your system sh=
ould update - of course you have to figure out how your system knows to=
update the grub.cfg so that it does it correctly without stepping on y=
our ESP grub.cfg (for one comment out the /boot/efi line in fstab and t=
hen unmount the ESP). This is distribution specific.
=20
Either way it's a bit complicated to figure out initially.
=20
After thinking about it for a while, I'm not sure about the point of
having ESP mirrored or at least I'm not sure it really worths the
trouble: ESP and /boot are very static partitions and should be written
mostly during kernel upgrade. So even if something bad happens in those
partitions, the system should be able to run without any issues.

The annoying part is that if you decide/have to reboot and ESP is
damaged you won't be able to boot your system again. But that said if
you created initially a copy of /boot on each disk, you should be able
to boot again even though those copies are not uptodate.

Thanks.

--
To unsubscribe from this list: send the line "unsubscribe linux-raid" i=
n
the body of a message to ***@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Francis Moreau
2014-10-06 06:14:59 UTC
Permalink
Hello,
From what I can tell with UEFI I need to set up a UEFI partition with a
FAT format.
On my current BIOS system I have a Biosboot 1M, /boot Raid1 200M and /
Raid 1 40G.
Obviously Grub installs to the mbr, and then installs a bit into
Biosboot which can read raids, hence it can read and boot from /boot.
Further, from what I can tell, into the UEFI partition can go either a
kernel & initramfs with UEFI support, or a "loader" that then loads the
kernel.
What I am unsure about are...
1) Can the loader/kernel understand md raid? so the / can be in a bog
standard raid1 v1.2?
2) I'm guessing I would no longer need the /boot as that would be
replaced by what ever was in the UEFI partition?
3) as my "/boot" is currently in a raid 1 my life is simple, should any
changes occur they are replicated to the drives I have set up as /boot
raid 1, and I installed the mbr portion of boot loader manually on each
disk and tested pulling one, and then booting from another.. it
worked :-)
So I would like to keep things, if not simple, at least less likely to
have problems because I forgot to install duplicates of the UEFI on all
the disks... so can I use a .90v raid1 on the UEFI partition, then
format it as fat... so that all copies of what ever is in the UEFI
partition are replicated across the multiple raid1 disks, instead of
having to remember to copy what ever is in there manually to ach disk?
There was already a post on that subject you can find here:
http://comments.gmane.org/gmane.linux.raid/45611

It might answer some of your questions.

Bye.

--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to ***@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Loading...