'Device not ready' issue on mpt2sas since 3.1.10

Discussion:

Matthias Prager

2012-07-09 14:40:15 UTC

Hello linux-scsi and linux-raid,

I did some further research regarding my problem.
It appears to me the fault does not lie with the mpt2sas driver (not
that I can definitely exclude it), but with the md implementation.

I reproduced what I think is the same issue on a different machine (also
running Vmware ESXi 5 and an LSI 9211-8i in IR mode) with a different
set of hard-drives of the same model. Using systemrescuecd
(2.8.1-beta003) and booting the 64bit 3.4.4 kernel, I issued the
following commands:

1) 'hdparm -y /dev/sda' (to put the hard-drive to sleep)
2) 'mdadm --create /dev/md1 --metadata 1.2 --level=mirror
--raid-devices=2 --name=test1 /dev/sda missing'
3) 'fdisk -l /dev/md127' (for some reason /proc/mdstat indicates the md
is being created as md127)

2) gave me this feedback:
------
mdadm: super1.x cannot open /dev/sda: Device or resource busy
mdadm: /dev/sda is not suitable for this array.
mdadm: create aborted
-------
Even though it says creating aborted it still created md127.

And 3) lead to these lines in dmesg:
-------
[ 604.838640] sd 2:0:0:0: [sda] Device not ready
[ 604.838645] sd 2:0:0:0: [sda] Result: hostbyte=DID_OK
driverbyte=DRIVER_SENSE
[ 604.838655] sd 2:0:0:0: [sda] Sense Key : Not Ready [current]
[ 604.838663] sd 2:0:0:0: [sda] Add. Sense: Logical unit not ready,
initializing command required
[ 604.838668] sd 2:0:0:0: [sda] CDB: Read(10): 28 00 00 00 08 00 00 00
20 00
[ 604.838680] end_request: I/O error, dev sda, sector 2048
[ 604.838688] Buffer I/O error on device md127, logical block 0
[ 604.838695] Buffer I/O error on device md127, logical block 1
[ 604.838699] Buffer I/O error on device md127, logical block 2
[ 604.838702] Buffer I/O error on device md127, logical block 3
[ 604.838783] sd 2:0:0:0: [sda] Device not ready
[ 604.838785] sd 2:0:0:0: [sda] Result: hostbyte=DID_OK
driverbyte=DRIVER_SENSE
[ 604.838789] sd 2:0:0:0: [sda] Sense Key : Not Ready [current]
[ 604.838793] sd 2:0:0:0: [sda] Add. Sense: Logical unit not ready,
initializing command required
[ 604.838797] sd 2:0:0:0: [sda] CDB: Read(10): 28 00 00 00 08 00 00 00
08 00
[ 604.838805] end_request: I/O error, dev sda, sector 2048
[ 604.838808] Buffer I/O error on device md127, logical block 0
[ 604.838983] sd 2:0:0:0: [sda] Device not ready
[ 604.838986] sd 2:0:0:0: [sda] Result: hostbyte=DID_OK
driverbyte=DRIVER_SENSE
[ 604.838989] sd 2:0:0:0: [sda] Sense Key : Not Ready [current]
[ 604.838993] sd 2:0:0:0: [sda] Add. Sense: Logical unit not ready,
initializing command required
[ 604.838998] sd 2:0:0:0: [sda] CDB: Read(10): 28 00 57 54 65 d8 00 00
08 00
[ 604.839006] end_request: I/O error, dev sda, sector 1465148888
[ 604.839009] Buffer I/O error on device md127, logical block 183143355
[ 604.839087] sd 2:0:0:0: [sda] Device not ready
[ 604.839090] sd 2:0:0:0: [sda] Result: hostbyte=DID_OK
driverbyte=DRIVER_SENSE
[ 604.839093] sd 2:0:0:0: [sda] Sense Key : Not Ready [current]
[ 604.839097] sd 2:0:0:0: [sda] Add. Sense: Logical unit not ready,
initializing command required
[ 604.839102] sd 2:0:0:0: [sda] CDB: Read(10): 28 00 57 54 65 d8 00 00
08 00
[ 604.839110] end_request: I/O error, dev sda, sector 1465148888
[ 604.839113] Buffer I/O error on device md127, logical block 183143355
[ 604.839271] sd 2:0:0:0: [sda] Device not ready
[ 604.839274] sd 2:0:0:0: [sda] Result: hostbyte=DID_OK
driverbyte=DRIVER_SENSE
[ 604.839278] sd 2:0:0:0: [sda] Sense Key : Not Ready [current]
[ 604.839282] sd 2:0:0:0: [sda] Add. Sense: Logical unit not ready,
initializing command required
[ 604.839286] sd 2:0:0:0: [sda] CDB: Read(10): 28 00 00 00 08 00 00 00
20 00
[ 604.839321] end_request: I/O error, dev sda, sector 2048
[ 604.839324] Buffer I/O error on device md127, logical block 0
[ 604.839330] Buffer I/O error on device md127, logical block 1
[ 604.840494] sd 2:0:0:0: [sda] Device not ready
[ 604.840497] sd 2:0:0:0: [sda] Result: hostbyte=DID_OK
driverbyte=DRIVER_SENSE
[ 604.840504] sd 2:0:0:0: [sda] Sense Key : Not Ready [current]
[ 604.840512] sd 2:0:0:0: [sda] Add. Sense: Logical unit not ready,
initializing command required
[ 604.840516] sd 2:0:0:0: [sda] CDB: Read(10): 28 00 00 00 08 00 00 00
08 00
[ 604.840526] end_request: I/O error, dev sda, sector 2048
------

This excludes hardware-errors (different physical machine and devices)
as cause and also ext4 which the other system was using as filesystem.
Maybe Neil Brown (who scripts/get_maintainer.pl identified as the
maintainer of the md-code) can make bits and pieces of this. It may well
be this is the same problem but a different error-path - I don't know.

I will try to make the scenario more generic, but I don't have a
non-virtual machine to spare atm. Also please do let me know if I'm
posting this to the wrong lists (linux-scsi and linux-raid) or if there
is anything which might not be helpful with the way I'm reporting this.

Regards,
Matthias Prager
--
To unsubscribe from this list: send the line "unsubscribe linux-scsi" in
the body of a message to ***@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html

Robert Trace

2012-07-09 19:37:09 UTC

Permalink

Post by Matthias Prager
I did some further research regarding my problem.
It appears to me the fault does not lie with the mpt2sas driver (not
that I can definitely exclude it), but with the md implementation.

I'm actually discovering some of the same issues (LSI 9211-8i w/ SATA
disks), but I've come to a slightly different conclusion.

I noticed that when my SATA disks are on a SATA controller and they spin
down (or are spun down via hdparm -y), then they response to TUR (TEST
UNIT READY) commands with an OK. Any I/O sent to these disks simply
wait while the disks spin up and then complete as usual.

However, my SATA disks on the SAS controller respond to TUR with the
sense error "Not Ready/Initializing command required". Any I/O sent to

Post by Matthias Prager
[ 604.838640] sd 2:0:0:0: [sda] Device not ready
[ 604.838645] sd 2:0:0:0: [sda] Result: hostbyte=DID_OK
driverbyte=DRIVER_SENSE
[ 604.838655] sd 2:0:0:0: [sda] Sense Key : Not Ready [current]
[ 604.838663] sd 2:0:0:0: [sda] Add. Sense: Logical unit not ready,
initializing command required
[ 604.838668] sd 2:0:0:0: [sda] CDB: Read(10): 28 00 00 00 08 00 00 00
20 00
[ 604.838680] end_request: I/O error, dev sda, sector 2048
[ 604.838688] Buffer I/O error on device md127, logical block 0
[ 604.838695] Buffer I/O error on device md127, logical block 1
[ 604.838699] Buffer I/O error on device md127, logical block 2
[ 604.838702] Buffer I/O error on device md127, logical block 3

Sending an explicit START UNIT command to these sleeping disks will wake
them up and then they behave normally. (BTW, you can issue TURs and
START UNITs via the sg_turs and sg_start commands).

I've reproduced this behavior on the raw disks themselves, no MD layer
involved (although the freak-out by my MD layer is what alerted me to
this issue too... Having your entire array punted the first time you
access it is a little scary :-). I'm also on raw hardware and I've seen
this behavior on kernels 3.0.33 through 3.4.4.

So, SATA disks respond differently depending on the controller they're
on. I don't know if this is a SCSI thing, a SAS thing or a
firmware/driver thing for the 9211.

Now, whether or not the MD layer should be assembling arrays from
"failed" disks is, I think, a separate issue.

-- Rob
--
To unsubscribe from this list: send the line "unsubscribe linux-scsi" in
the body of a message to ***@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html

Darrick J. Wong

2012-07-09 20:45:13 UTC

Permalink

Post by Robert Trace

I'm actually discovering some of the same issues (LSI 9211-8i w/ SATA
disks), but I've come to a slightly different conclusion.
I noticed that when my SATA disks are on a SATA controller and they spin
down (or are spun down via hdparm -y), then they response to TUR (TEST
UNIT READY) commands with an OK. Any I/O sent to these disks simply
wait while the disks spin up and then complete as usual.
However, my SATA disks on the SAS controller respond to TUR with the
sense error "Not Ready/Initializing command required". Any I/O sent to

I suspect that /sys/devices/<bunch of sas topology here>/manage_start_stop = 0
for the SATA devices hanging off the SAS controller. Setting that sysfs
attribute to 1 is supposed to enable the SCSI layer to send TUR when it sees
"LU not ready", as well as spin down the drives at suspend/poweroff time.

--D

Post by Robert Trace
Now, whether or not the MD layer should be assembling arrays from
"failed" disks is, I think, a separate issue.
-- Rob
--
To unsubscribe from this list: send the line "unsubscribe linux-scsi" in
More majordomo info at http://vger.kernel.org/majordomo-info.html

--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to ***@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html

Robert Trace

2012-07-09 22:24:15 UTC

Permalink

Post by Darrick J. Wong
I suspect that /sys/devices/<bunch of sas topology here>/manage_start_stop = 0
for the SATA devices hanging off the SAS controller.

Yep, looks like you're right. For my system:

# cat /sys/block/sd?/device/scsi_disk/*/manage_start_stop
1
1
1
1
1
0
0
0
0
0
0
0
0

Those first 5 disks are SATA disks on SATA controllers. The last 8
disks are SATA disks on the SAS controller.

Post by Darrick J. Wong
Setting that sysfs
attribute to 1 is supposed to enable the SCSI layer to send TUR when it sees
"LU not ready", as well as spin down the drives at suspend/poweroff time.

Setting it to 1 doesn't seem to have made any difference, however.

# cat /sys/block/sdm/device/scsi_disk/14\:0\:7\:0/manage_start_stop
0
# echo 1 > /sys/block/sdm/device/scsi_disk/14\:0\:7\:/manage_start_stop
# cat /sys/block/sdm/device/scsi_disk/14\:0\:7\:0/manage_start_stop
1
# hdparm -y /dev/sdm

/dev/sdm:
issuing standby command
# hdparm -C /dev/sdm

/dev/sdm:
drive state is: standby
# dd if=/dev/sdm of=/dev/null bs=512 count=1
dd: reading `/dev/sdm': Input/output error
0+0 records in
0+0 records out
0 bytes (0 B) copied, 0.00117802 s, 0.0 kB/s

... and on the scsi logging side, I see the read(10) to the disk which
immediately returns "Not Ready" and the I/O failure bubbles up the
chain. And afterwards, the disk is still asleep.

# hdparm -C /dev/sdm

/dev/sdm:
drive state is: standby

Also, TURs don't appear to actually wake the disk up (should they?).
The only thing I've found that'll wake the disk up is an explicit START
UNIT command.

-- Rob
--
To unsubscribe from this list: send the line "unsubscribe linux-scsi" in
the body of a message to ***@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html

Matthias Prager

2012-07-10 00:21:09 UTC

Permalink

Post by Robert Trace
Also, TURs don't appear to actually wake the disk up (should they?).
The only thing I've found that'll wake the disk up is an explicit START
UNIT command.

I haven't checked the scsi logging side, but about the only commands
that wake up the disks are 'smartctl -a /dev/sda' and 'sg_start'
(smartcl maybe issuing a START UNIT command on it's own).

Matthias
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to ***@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html

Darrick J. Wong

2012-07-10 16:54:48 UTC

Permalink

Post by Robert Trace

Post by Darrick J. Wong
I suspect that /sys/devices/<bunch of sas topology here>/manage_start_stop = 0
for the SATA devices hanging off the SAS controller.

# cat /sys/block/sd?/device/scsi_disk/*/manage_start_stop
1
1
1
1
1
0
0
0
0
0
0
0
0
Those first 5 disks are SATA disks on SATA controllers. The last 8
disks are SATA disks on the SAS controller.

Post by Darrick J. Wong
Setting that sysfs
attribute to 1 is supposed to enable the SCSI layer to send TUR when it sees
"LU not ready", as well as spin down the drives at suspend/poweroff time.

Setting it to 1 doesn't seem to have made any difference, however.
# cat /sys/block/sdm/device/scsi_disk/14\:0\:7\:0/manage_start_stop
0
# echo 1 > /sys/block/sdm/device/scsi_disk/14\:0\:7\:/manage_start_stop
# cat /sys/block/sdm/device/scsi_disk/14\:0\:7\:0/manage_start_stop
1
# hdparm -y /dev/sdm
issuing standby command
# hdparm -C /dev/sdm
drive state is: standby
# dd if=/dev/sdm of=/dev/null bs=512 count=1
dd: reading `/dev/sdm': Input/output error
0+0 records in
0+0 records out
0 bytes (0 B) copied, 0.00117802 s, 0.0 kB/s
... and on the scsi logging side, I see the read(10) to the disk which
immediately returns "Not Ready" and the I/O failure bubbles up the
chain. And afterwards, the disk is still asleep.
# hdparm -C /dev/sdm
drive state is: standby
Also, TURs don't appear to actually wake the disk up (should they?).
The only thing I've found that'll wake the disk up is an explicit START
UNIT command.

Sorry, I misspoke, manage_start_stop=1 sends START UNIT, not TUR. Also, it
only manages spindown/up at suspend/resume time, hence the behavior you see.
The relevant source code is sd_start_stop_device() in drivers/scsi/sd.c.

--D

Post by Robert Trace
-- Rob

--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to ***@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html

Matthias Prager

2012-07-10 00:12:04 UTC

Permalink

Post by Robert Trace

I'm actually discovering some of the same issues (LSI 9211-8i w/ SATA
disks), but I've come to a slightly different conclusion.
I noticed that when my SATA disks are on a SATA controller and they spin
down (or are spun down via hdparm -y), then they response to TUR (TEST
UNIT READY) commands with an OK. Any I/O sent to these disks simply
wait while the disks spin up and then complete as usual.
However, my SATA disks on the SAS controller respond to TUR with the
sense error "Not Ready/Initializing command required". Any I/O sent to

Sending an explicit START UNIT command to these sleeping disks will wake
them up and then they behave normally. (BTW, you can issue TURs and
START UNITs via the sg_turs and sg_start commands).

Thanks for these pointers.

Post by Robert Trace
I've reproduced this behavior on the raw disks themselves, no MD layer
involved (although the freak-out by my MD layer is what alerted me to
this issue too... Having your entire array punted the first time you
access it is a little scary :-). I'm also on raw hardware and I've seen
this behavior on kernels 3.0.33 through 3.4.4.

This is interesting - are you sure about 3.0.33? I'm running this kernel
atm for it gives me no trouble (as opposed to >=3.1.10). The SATA disks
are spun up when I access data on them.

Post by Robert Trace
So, SATA disks respond differently depending on the controller they're
on. I don't know if this is a SCSI thing, a SAS thing or a
firmware/driver thing for the 9211.
Now, whether or not the MD layer should be assembling arrays from
"failed" disks is, I think, a separate issue.

I realize now in my cases the MD layer behaved correctly.

Post by Robert Trace
-- Rob

--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to ***@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html

NeilBrown

2012-07-09 22:08:31 UTC

Permalink

Post by Matthias Prager
Hello linux-scsi and linux-raid,
I did some further research regarding my problem.
It appears to me the fault does not lie with the mpt2sas driver (not
that I can definitely exclude it), but with the md implementation.
I reproduced what I think is the same issue on a different machine (also
running Vmware ESXi 5 and an LSI 9211-8i in IR mode) with a different
set of hard-drives of the same model. Using systemrescuecd
(2.8.1-beta003) and booting the 64bit 3.4.4 kernel, I issued the
1) 'hdparm -y /dev/sda' (to put the hard-drive to sleep)
2) 'mdadm --create /dev/md1 --metadata 1.2 --level=mirror
--raid-devices=2 --name=test1 /dev/sda missing'
3) 'fdisk -l /dev/md127' (for some reason /proc/mdstat indicates the md
is being created as md127)
------
mdadm: super1.x cannot open /dev/sda: Device or resource busy
mdadm: /dev/sda is not suitable for this array.
mdadm: create aborted
-------
Even though it says creating aborted it still created md127.

One of my pet peeves in when people interpret the observations wrongly and
then report their interpretation instead of their observation. However
sometimes it is very hard to separate the two. You comment above looks
perfectly reasonable and looks like a clean observation and not and
interpretation. Yet it is an interpretation :-)

The observation would be
"Even though it says creating abort, md127 was still created".

You see, it wasn't this mdadm that created md127 - it certainly shouldn't
have as you asked it to create md1.

I don't know the exact sequence of events, but something - possibly relating
to the error messages reported below - caused udev to notice /dev/sda.
udev then ran "mdadm -I /dev/sda" and as it had some metadata on it, it
created an array with it. As the name information in that metadata was
probably "test1" or similar, rather than "1", mdadm didn't know what number
was wanted for the array, so it chose a free high number - 127.

This metadata must have been left over from an earlier experiment.

So it might have been something like.

- you run mdadm (call this mdadm-1).
- mdadm tries to open sda
- driver notices that device is asleep, and wakes it up
- the waking up of the device causes a CHANGE uevent to udev
- this cause udev to run a new mdadm - mdadm-2
- mdadm-2 reads the metadata, sees old metadata, assembled sda in a new md127
- mdadm-1 gets scheduled again, tries to get O_EXCL access to sda and fails,
because sda is now part of md127

Clearly undesirable behaviour. I'm not sure which bit is "wrong".

NeilBrown

Post by Matthias Prager
-------
[ 604.838640] sd 2:0:0:0: [sda] Device not ready
[ 604.838645] sd 2:0:0:0: [sda] Result: hostbyte=DID_OK
driverbyte=DRIVER_SENSE
[ 604.838655] sd 2:0:0:0: [sda] Sense Key : Not Ready [current]
[ 604.838663] sd 2:0:0:0: [sda] Add. Sense: Logical unit not ready,
initializing command required
[ 604.838668] sd 2:0:0:0: [sda] CDB: Read(10): 28 00 00 00 08 00 00 00
20 00
[ 604.838680] end_request: I/O error, dev sda, sector 2048
[ 604.838688] Buffer I/O error on device md127, logical block 0
[ 604.838695] Buffer I/O error on device md127, logical block 1
[ 604.838699] Buffer I/O error on device md127, logical block 2
[ 604.838702] Buffer I/O error on device md127, logical block 3
[ 604.838783] sd 2:0:0:0: [sda] Device not ready
[ 604.838785] sd 2:0:0:0: [sda] Result: hostbyte=DID_OK
driverbyte=DRIVER_SENSE
[ 604.838789] sd 2:0:0:0: [sda] Sense Key : Not Ready [current]
[ 604.838793] sd 2:0:0:0: [sda] Add. Sense: Logical unit not ready,
initializing command required
[ 604.838797] sd 2:0:0:0: [sda] CDB: Read(10): 28 00 00 00 08 00 00 00
08 00
[ 604.838805] end_request: I/O error, dev sda, sector 2048
[ 604.838808] Buffer I/O error on device md127, logical block 0
[ 604.838983] sd 2:0:0:0: [sda] Device not ready
[ 604.838986] sd 2:0:0:0: [sda] Result: hostbyte=DID_OK
driverbyte=DRIVER_SENSE
[ 604.838989] sd 2:0:0:0: [sda] Sense Key : Not Ready [current]
[ 604.838993] sd 2:0:0:0: [sda] Add. Sense: Logical unit not ready,
initializing command required
[ 604.838998] sd 2:0:0:0: [sda] CDB: Read(10): 28 00 57 54 65 d8 00 00
08 00
[ 604.839006] end_request: I/O error, dev sda, sector 1465148888
[ 604.839009] Buffer I/O error on device md127, logical block 183143355
[ 604.839087] sd 2:0:0:0: [sda] Device not ready
[ 604.839090] sd 2:0:0:0: [sda] Result: hostbyte=DID_OK
driverbyte=DRIVER_SENSE
[ 604.839093] sd 2:0:0:0: [sda] Sense Key : Not Ready [current]
[ 604.839097] sd 2:0:0:0: [sda] Add. Sense: Logical unit not ready,
initializing command required
[ 604.839102] sd 2:0:0:0: [sda] CDB: Read(10): 28 00 57 54 65 d8 00 00
08 00
[ 604.839110] end_request: I/O error, dev sda, sector 1465148888
[ 604.839113] Buffer I/O error on device md127, logical block 183143355
[ 604.839271] sd 2:0:0:0: [sda] Device not ready
[ 604.839274] sd 2:0:0:0: [sda] Result: hostbyte=DID_OK
driverbyte=DRIVER_SENSE
[ 604.839278] sd 2:0:0:0: [sda] Sense Key : Not Ready [current]
[ 604.839282] sd 2:0:0:0: [sda] Add. Sense: Logical unit not ready,
initializing command required
[ 604.839286] sd 2:0:0:0: [sda] CDB: Read(10): 28 00 00 00 08 00 00 00
20 00
[ 604.839321] end_request: I/O error, dev sda, sector 2048
[ 604.839324] Buffer I/O error on device md127, logical block 0
[ 604.839330] Buffer I/O error on device md127, logical block 1
[ 604.840494] sd 2:0:0:0: [sda] Device not ready
[ 604.840497] sd 2:0:0:0: [sda] Result: hostbyte=DID_OK
driverbyte=DRIVER_SENSE
[ 604.840504] sd 2:0:0:0: [sda] Sense Key : Not Ready [current]
[ 604.840512] sd 2:0:0:0: [sda] Add. Sense: Logical unit not ready,
initializing command required
[ 604.840516] sd 2:0:0:0: [sda] CDB: Read(10): 28 00 00 00 08 00 00 00
08 00
[ 604.840526] end_request: I/O error, dev sda, sector 2048
------
This excludes hardware-errors (different physical machine and devices)
as cause and also ext4 which the other system was using as filesystem.
Maybe Neil Brown (who scripts/get_maintainer.pl identified as the
maintainer of the md-code) can make bits and pieces of this. It may well
be this is the same problem but a different error-path - I don't know.
I will try to make the scenario more generic, but I don't have a
non-virtual machine to spare atm. Also please do let me know if I'm
posting this to the wrong lists (linux-scsi and linux-raid) or if there
is anything which might not be helpful with the way I'm reporting this.
Regards,
Matthias Prager
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
More majordomo info at http://vger.kernel.org/majordomo-info.html

Matthias Prager

2012-07-10 00:03:55 UTC

Permalink

Post by NeilBrown

Post by Matthias Prager
Even though it says creating aborted it still created md127.

Sry - I jumped to conclusions without knowing what was actually going on.

Post by NeilBrown
I don't know the exact sequence of events, but something - possibly relating
to the error messages reported below - caused udev to notice /dev/sda.
udev then ran "mdadm -I /dev/sda" and as it had some metadata on it, it
created an array with it. As the name information in that metadata was
probably "test1" or similar, rather than "1", mdadm didn't know what number
was wanted for the array, so it chose a free high number - 127.
This metadata must have been left over from an earlier experiment.

That is correct (as am just realizing now). There is metadata of an
raid1 array left on the disk even though it was used (for a short time)
with zfs on freebsd before doing these experiments.

Post by NeilBrown
So it might have been something like.
- you run mdadm (call this mdadm-1).
- mdadm tries to open sda
- driver notices that device is asleep, and wakes it up
- the waking up of the device causes a CHANGE uevent to udev
- this cause udev to run a new mdadm - mdadm-2
- mdadm-2 reads the metadata, sees old metadata, assembled sda in a new md127
- mdadm-1 gets scheduled again, tries to get O_EXCL access to sda and fails,
because sda is now part of md127
Clearly undesirable behaviour. I'm not sure which bit is "wrong".

As it turns out mdadm is doing everything right. md127 is actually
already present (though inactive) at boot-time. So mdadm is absolutly
correct in saying sda is busy and refusing to do anything further.

Post by NeilBrown
NeilBrown

The real problem seems to be located in some layer below md, which is
not waking up the disk for any i/o (at all - not even for fdisk -l).

Matthias
--
To unsubscribe from this list: send the line "unsubscribe linux-scsi" in
the body of a message to ***@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html