Sun Storage 6320 System
 
Sun Storage 6120 Array
 
Sun Storage 6920 System
 

GCS>Sun Microsystems>Sun Alert>Criteria Category>Availability
 
GCS>Sun Microsystems>Sun Alert>Release Phase>Resolved
 
GCS>Sun Microsystems>Sun Alert>Criteria Category>Data Loss
 

Upgrading a SE6x20 controller to firmware 3.2.4 may result in future drive failures that cannot be replaced. The replacement drive does not come online and remains in a degraded state. If a second drive failure occurs it may result in the loss of customer data.

This issue occurs in the following releases:

• Sun StorEdge T3B Arrays with Array firmware 3.2.4 (as delivered in patch 116930-05)
• Sun StorEdge 6120 Arrays with Array firmware 3.2.4 (as delivered in patch 116931-20)
• Sun StorEdge 6320 Arrays with 6020 Array firmware 3.2.4 (6320 Release 1.3.2)
• Sun StorEdge 6920 Arrays with 6020 Array firmware 3.2.4 (6920 Release 3.0.1.19)
• Sun StorEdge 6920 Arrays with 6020 Array firmware 3.2.4 (6920 Release 3.0.1.20)

Note: this issue is only with firmware 3.2.4. Please see Resolution section below.

The most prevalent symptom is that a failed drive remains in a "fault disabled" state when viewed from StorADE. It becomes more evident after replacing the same drive with a new drive component.

The following messages mark the original drive failure and are an indication of this issue. These messages can be found in the array syslog for T3B and 6120 arrays. They are also found in the messages.array file for the 6320 and 6920 arrays when viewed using StorADE Solution Extract utility.

Example #1
Dec 23 14:08:21 ISR1[1]: N: u1d11 SCSI Disk Error Occurred (path = 0x1)
Dec 23 14:08:21 ISR1[1]: N: u1d11 Sense Key = 0x1, Asc = 0x5d, Ascq = 0x0
Dec 23 14:08:21 ISR1[1]: N: u1d11 Sense Data Description = Failure
Prediction Threshold Exceeded
Dec 23 14:08:21 ISR1[1]: N: u1d11 SVD_CHECK_ERROR: prediction err: 01/5D
Example #2
Jul 29 22:32:51 LPCT[1]: W: u1d03 Not present
Jul 29 22:34:35 ISR1[1]: E: Drive u1d03 Additional errors sense 5 asc 36
ascq 0 detected during copy recon. Drive is disabled
Jul 29 22:34:35 ISR1[1]: W: u1d03 SCSI error occurred: Not Ready (sense
key = 0x2). Logical Unit Not Ready, Initializing CMD Required.
Jul 29 22:34:35 ISR1[1]: E: Drive u1d03 Additional errors sense 2 asc 4 ascq
2 detected during copy recon. Drive is disabled
Jul 29 22:34:36 MNXT[1]: N: u1d01 System area recon fail due to write error
Jul 29 22:34:36 MNXT[1]: W: u1d03 could not create system area
Jul 29 22:34:37 LPCT[1]: N: u1d03 Bypassed on loop 2
Jul 29 22:34:36 MNXT[2]: N: u1d03 Unable to access the drive (err = 3)
Jul 29 22:34:37 MNXT[2]: N: u1d03 Unable to access the drive (err = 3)
Jul 29 22:34:39 LPCT[1]: N: u1d03 Bypassed on loop 1
Jul 29 22:34:38 MNXT[2]: N: u1d03 Unable to access the drive (err = 3)
Jul 29 22:34:40 MNXT[2]: N: u1d03 Unable to access the drive (err = 3)

Do not upgrade to the 3.2.4 array firmware for the above arrays.

For customers already at firmware 3.2.4, an array firmware downgrade to the previous revision is available, as part of the workaround and recovery of this issue. Contact Sun Services for the proper recovery procedure for your model of array.

This issue is addressed in the following releases:

• Sun StorEdge T3B Arrays with Array firmware 3.2.5 (as delivered in patch 116930-06 or later)
• Sun StorEdge 6120 Arrays with Array firmware 3.2.5 (as delivered in patch 116931-21 or later)
• Sun StorEdge 6320 Arrays with 6020 Array firmware 3.2.5
• Sun StorEdge 6920 Arrays with 6020 Array firmware 3.2.5
• Sun StorEdge 6920 Arrays with 6020 Array firmware 3.2.5

Note: Once the upgrade is completed if the failed drive has already been replaced, it will spin up properly when the new firmware is booted. If the failed drive has not been replaced, normal disk replacement will be successful

Moving forward, although the 01/5d errors may still occur from time to time, normal supported drive replacement procedures will function as expected.

• Updated Relief/Workaround section

• Updated Resolution section.
• State: Resolved

References

 The problem in v3.2.4 will affect both T4 and T3B. 

 The fix in v3.2.5 should cover both of them too.

    Jim Gou - Engenio Storage Group - LSI Logic

    Sun Office phone: (510)936-2727

 

The problem is caused by the use of a Mask Bit, previously unused in the array firmware to solve an earlier bug(Bug 6366175 and Bug 6359654. See Sun Alert 102193). Under certain error conditions a "Mask Bit" in the sysarea will be set during a drive failure. Array f/w releases 3.2.3 and earlier did not utilize the Mask Bit even though it was set.

Array f/w 3.2.4 now utilizes the Mask Bit. If 3.2.4 sees the Mask Bit set for a drive, that drive will be put into a fault/disabled state regardless of the actual condition.

The drive may in fact be 100% functional.

We know of various ways the Mask Bit can get set. They are:

1) In array f/w 3.1.x, the 01/5D Sense Key will trigger the Mask Bit. In these releases the drive fault is most likely legitimate. The drive CAN be swapped successfully and brought back to a Ready/Enabled state. However, the Mask Bit will remain set. Upon upgrading to 3.2.4, which could be weeks or months later, the problem will be triggered.

2) In array f/w 3.2.x, various BEFIT functions will also set the Mask Bit. These conditions are known to be associated with "Failure Prediction Threshold Exceeded" messages. However, in some cases, NO message appears when the Mask Bit is set.

Once the Mask Bit is set, the drive will be fault/disabled. Replacement drives will remain fault/disabled.

3) In Array f/w 3.2.4, a device failed by BEFIT, legitimately, could also set the Mask Bit. Replacement drives will remain fault/disabled. Please be aware, the Mask Bit will only be set under a limited set of conditions.

Field and Solution Center Engineers are required to open an escalation into PTS-STORAGE for the proper corrective action. And reference this Sun Alert.

NOTE 1: Avoidance and recovery of this condition require the downgrade of the array firmware(3.2.3 for T3B/6120/6320, 3.2.2 for 6920), in addition to steps indicated by the assigned PTS engineer. Please defer to the PTS Engineer instructions prior to taking any action.

NOTE 2: As stated in the bug, the other possible workaround is to initialize the sysarea of the array, effectively destroying the data, and requiring volume/vdisk/volslice rebuild and restore. As with the firmware downgrade, Field and Solution Center Engineers are required to open an escalation into PTS-STORAGE, for the proper corrective action.

This solution has no attachment