| Document Audience: | INTERNAL |
| Document ID: | I1125-1 |
| Title: | StorEdge 6120 and 6320 Arrays with firmware prior to 3.1.x may experience downtime or data integrity if double disk errors or back-end loop hangs occur. |
| Copyright Notice: | Copyright © 2005 Sun Microsystems, Inc. All Rights Reserved |
| Update Date: | 2004-11-17 |
___________________________________________________
*** Sun Confidential: Internal Use and Authorized VARs Only ***
__________________________________________________________________
This message including any attachments is confidential information
of Sun Microsystems, Inc. Disclosure, copying or distribution is
prohibited without permission of Sun. If you are not the intended
recipient, please reply to the sender and then delete this message.
__________________________________________________________________
FIELD INFORMATION NOTICE
(For Authorized Distribution by Sun Services)FIN #: I1125-1
Synopsis: StorEdge 6120 and 6320 Arrays with firmware prior to 3.1.x may experience downtime or data integrity if double disk errors or back-end loop hangs occur.Create Date: Oct/29/04
SunAlert: No
Top FIN/FCO Report: Yes
Products Reference: Sun StorEdge 6120/6320 Arrays
Product Category: Storage / SW Admin
Product Affected:
Systems Affected:
-----------------
Mkt_ID Platform Model Description Serial Number
------ -------- ----- ----------- -------------
- Anysys - System Platform Independent -
X-options Affected:
--------- --------
Mkt_ID Platform Model Description Serial Number
------ -------- ----- ----------- -------------
- SE6120 ALL Sun StorEdge 6120 Array -
- SE6320 ALL Sun StorEdge 6320 Array -
Parts Affected:
----------------------
Part Number Description Model
----------- ----------- -----
540-5559-XX ASSY T4 2.5 CONTRLR W/SLED -
References:
BugId: Refer to Patch README file for a complete list of fixed bugs.
PatchId: 115179-11: 6120 Array FW patch ID.
116655-02: 6120 Mgmt Software Patch ID.
115589-06: 6320 SP Software Patch ID.
115179-11: 6320 Firmware Patch ID.
114591-20: 6320 StorADE Patch ID.
116819-01: 6320 NTC Firmware Patch ID.
117106-01: 6320 FBR Patch ID.
116656-02: 6320 Management Software.
113193-04: 6320 Patchpro Patch ID.
Manual: 819-0051-10: Disk Scrubbing Sun StorEdge Arrays.
817-0201-19: Product Release Notes for 6120.
816-7880-19: Product Release Notes for 6320.
URL: https://spe.sun.com/spx/control/Login
http://ns-qcc.ebay/HealthCheck/index.html
http://sunsolve.sun.com/handbook_pub/Systems/6120/docs.html
http://sunsolve.central.sun.com/handbook_internal/Systems/6120/docs.html
http://sunsolve.sun.com/handbook_pub/Systems/6320/docs.html
http://sunsolve.central.sun.com/handbook_internal/Systems/6320/docs.html
http://sdpsweb.central/FIN_FCO/FIN/FINI1125-1_dir/SPE/Cust_List.sxc
http://sdpsweb.central/FIN_FCO/FIN/FINI1125-1_dir/SPE/Cust_Letter.sxwIssue Description:
-------------------------------------------------------------------------
| Change History: |
| =============== |
| Oct/29/04 Updated Step 1 and Step 2 in Corrective Action. |
| |
| Bug 5065023. |
-------------------------------------------------------------------------
Sun StorEdge 6120/6320 Arrays with firmware (FW) versions earlier than 3.1.x
may cause system downtime and data integrity if double disk errors or back-end
loop hangs occur.
The configurations affected by this issue are as follows:
. Any storage 6120/6320 arrays running 3.0.x, or 3.1.0, 3.1.1, 3.1.2 FW.
It should be noted that, while 3.1.0, 3.1.1, 3.1.2 and 3.1.3 have the disk
scrubber and BEFIT features available, there have been quality improvements
in 3.1.4 or later which would make it beneficial to upgrade those systems
as well.
To determine the firmware version for an array, log on to the array, and
run the "ver" command at the array prompt.
These issues are addressed by the following software patches.
. For Storage 6120 Arrays:
Array FW patch ID: 115179-11
Mgmt Software Patch ID: 116655-02
. For Storage 6320 Arrays:
SP Software Patch ID: 115589-06
Firmware Patch ID: 115179-11
StorADE Patch ID: 114591-20
NTC Firmware Patch ID: 116819-01
FBR Patch ID: 117106-01
Management Software: 116656-02
Patchpro Patch ID: 113193-04
The new array firmware provides these features.
Back End Fault Isolation:
-------------------------
BEFIT monitors the back end loops of the array and isolates
failed components which may hang both backend loops. As an
example, if a drive fails and hangs both loops, BEFIT would
detect the loops being hung and start the process of isolating
each drive to find the faulty one. Once found, that drive will
be taken offline and the system will continue to run.
Disk Scrubber:
--------------
Disk scrubber is a software component that continually checks
storage array disk drives for latent media read error conditions
and fixes them automatically. Disk Scrubbing is very important
to maintain the high availability of array volumes. The disks
are electromechanical devices and have potential to get media
read error related problems. The best practice is to run the
Disk Scrubber continuously in background which fixes the media
error conditions.
Explicit LUN Failover:
----------------------
ELF provides a way for multi-pathing host driver to manage the
LUN ownership so that the ping-pong effect that exists in
Implicit LUN Failover implementations can be eliminated.
Implicit LUN failover will be disabled when Explicit LUN
failover is used. Explicit LUN failover and Implicit LUN
failover are mutually exclusive.
Other Changes:
--------------
. Better I/O Queue depth management
. 4-node cluster support
. Codebase was C-styled and lint cleaned up
. The latest disk drive firmware included.
To review the root cause of an issue, please refer to the Disk Scrubber
White Paper document on the following documents.
. Manual P/N 819-0051-10 - Disk Scrubbing Sun StorEdge Arrays.
817-0201-19 - Product Release Notes for 6120.
816-7880-19 - Product Release Notes for 6320.
. Sun System Handbook URL for 6120 on:
http://sunsolve.sun.com/handbook_pub/Systems/6120/docs.html
http://sunsolve.central.sun.com/handbook_internal/Systems/6120/docs.html
. Sun System Handbook URL for 6320 on:
http://sunsolve.sun.com/handbook_pub/Systems/6320/docs.html
http://sunsolve.central.sun.com/handbook_internal/Systems/6320/docs.htmlImplementation:
---
| X | MANDATORY (Fully Proactive)
---
---
| | CONTROLLED PROACTIVE (per Sun Geo Plan)
---
---
| | REACTIVE (As Required)
---Corrective Action:
The following recommendation is provided as a guideline for authorized
Sun Services Field Representatives who may encounter the above
mentioned issue.
Follow the steps below to upgrade array, disk, and system firmware to the
recommended levels. Please use the following Customer List to identify
sites which may be affected.
http://sdpsweb.central/FIN_FCO/FIN/FINI1125-1_dir/SPE/Cust_List.sxc
Note that this Customer List is shared with FIN I1121-1 and includes
StorEdge T3+/3900/6900 systems related to that FIN.
Use the following Customer Letter as needed to communicate this issue
to customers.
http://sdpsweb.central/FIN_FCO/FIN/FINI1125-1_dir/SPE/Cust_Letter.sxw
STEP 1:
For 6120 Arrays and 6120-HA and 6320 Arrays:
--------------------------------------------
NOTE: 6320 vol verifies may be done through Storade.
Prior to the upgrade of controller and disk firmware, a vol verify
needs to be performed on every volume in the array. The Field does not
have to be present to do the verifications but is responsible for
their successful completion. This step is required if you are using
3.0.X firmware or lower. The following guidelines may be followed in
this verification process.
A. Do not use the "fix" modifier. If dual disk errors
are seen, this can result in the unmounting of the volume and
unscheduled down time. The verify operation is for reporting
purposes only.
B. Depending on host I/O activity the optional "rate"
modifier (rate 1 - 8) may or may not be applied.
C. Appropriate time needs to be given to complete all
verifies on all volumes. Depending on host activity, drive
sizes, raid types, etc, the verifies can take several days.
D. There must be a complete interrrogation of the syslog
to confirm that all verify operations have completed
successfully. Here is a verify "start" and "end" entry with no
disk errors in between.
Oct 09 07:09:17 MASD[1]: N: Vol verify (t1pool0) started
Oct 09 13:12:14 MASD[1]: N: Vol verify (t1pool0) ended
If errors are encountered, syslog could contain something such as
the following:
13:21:12 t3b3 sh78[1]: N: Volume vol1 verification started
13:33:50 t3b3 ISR1[1]: W: u1d8 SCSI Disk Error Occurred
13:33:50 t3b3 ISR1[1]: W: Sense Key = 0x3, Asc = 0x11, Ascq = 0x0
13:33:50 t3b3 ISR1[1]: W: Sense Data Description =
Unrecovered Read Error
Any such errors MUST be addressed prior to the continuation of the
upgrade. Appropriate repairs, drive swaps, lun rebuilds etc, need
to be done through the normal support channels taken by the
accounts. The field CANNOT proceed with the upgrades until such
verifications (and repairs) are confirmed by the attending SUN
representative.
STEP 2: Upgrade drive FW to the latest level if not running
at the minimum levels specified below.
Drive Model Minimum FW FW Patch
----------- ---------- --------
ST336752FC 0205 109962-14
ST336753FC 0149 116748-03
MAS3367FC 0701 116816-01
ST373307FC 0207 114708-04
MAP3735FC 1401 116514-05
DK32EJ72FC FQ0C 116464-01
ST3146807FC 0207 114709-04
MAP3147FC 1401 116815-03
DK32J14FC 2Q0A 116465-01
ST373453FC 0349 113673-01
MAS3735FC 0701 116817-01
For 6120 Arrays:
----------------
STEP 3: Upgrade Array FW to 3.1.4 or later.
STEP 4: Reboot the array.
STEP 5: Upgrade host-side Management Software, if installed.
For 6120-HA:
------------
STEP 3: Failover the master controller to the alternate controller.
STEP 4: Upgrade the master controller.
STEP 5: Failback the alternate controller to the master controller.
STEP 6: Upgrade the alternate controller.
STEP 7: Upgrade host-side Management Software, if installed
For 6320:
---------
STEP 3: Run vol verify via StorADE on each of the arrays if your
system is at MIRE 1.1.x or lower. Correct any disk errors found
by vol verify.
STEP 4: Upgrade the system to MIRE 1.2.3.
Refer to the following reference material as needed.
819-0051-10: Disk Scrubbing Sun StorEdge Arrays.
817-0201-19: Product Release Notes for 6120.
816-7880-19: Product Release Notes for 6320.
Healthcheck website url at:
http://ns-qcc.ebay/HealthCheck/index.htmlComments:
Please refer to Patch README file for a complete list of fixed bugs.
============================================================================
Implementation Footnote:
i) In case of MANDATORY FINs, Sun Services will attempt to
contact all affected customers to recommend implementation of
the FIN.
ii) For CONTROLLED PROACTIVE FINs, Sun Services mission critical
support teams will recommend implementation of the FIN (to their
respective accounts), at the convenience of the customer.
iii) For REACTIVE FINs, Sun Services will implement the FIN as the
need arises.
----------------------------------------------------------------------------
All released FINs and FCOs can be accessed using your favorite network
browser as follows:
SunSolve Internal Access:
_______________________
* Access the SunSolve Online URL at http://sunsolve.Central/
* From there, select the appropriate link to browse the FIN or FCO index.
Internet Access:
_______________
* Access the top level URL of https://spe.sun.com
FIN/FCO Homepage Access:
_________________________
* Access the top level URL of http://sdpsweb.Central/FIN_FCO/index.html
* From there, select the appropriate link to query or browse the FIN and
FCO Homepage collections.
To submit either a FIN or FCO refer to the following URLs for templates
and instructions;
* For FCO: http://pronto.central/fco.html
* For FIN: http://pronto.central/fin.html
--------------------------------------------------------------------------
General:
--------
* Send questions or comments to finfco-manager@Sun.COM
--------------------------------------------------------------------------