Controller replacement in Sun StorageTek 6140 and Sun StorageTek 6540 Arrays may initiate a Crystal firmware upgrade on the other controller.

Asset ID:	1-73-1019884.1
Update Date:	2010-09-21
Keywords:

Solution Type FAB (standard) Sure

Solution 1019884.1 : Controller replacement in Sun StorageTek 6140 and Sun StorageTek 6540 Arrays may initiate a Crystal firmware upgrade on the other controller.

Impact

Controller replacement for Sun StorageTek 6140 or Sun StorageTek 6540 Arrays may inadvertently upgrade the array to Crystal Firmware (7.10.25.10). If this occurs the Virtual Disks (CAM) or Volume Groups (SANtricity) will not be imported and data access to the associated volumes will be lost.

If this issue occurs the Virtual Disks (CAM) or Volume Groups (SANtricity) will not be importable and data access to the associated volumes will be lost. Additionally new licenses will need to be generated and the array's Cache Block Size may have changed from 16K to 4K. If the Virtual Disks or Volume Groups are then successfully imported, their numbering will be 0 for the first, 0_1 for the second, 0_2 for the third and so on. This numbering is cosmetic only.

Contributing Factors

Any Sun StorageTek 6140 or Sun StorageTek 6540 Array running 6.x firmware that experiences a controller failure may encounter this issue during the replacement. Services spares have been loaded with 7.x firmware in our factory.

The manner in which 7.x and 6.x firmware handles Virtual Disks or Volume Groups is radically different and is why such an upgrade requires a special process and utility as described in FAB 236969. This utility automates the handling of the Virtual Disks or Volume Groups so that they are properly imported at the end of the upgrade.

Arrays already running 7.x firmware are not affected.

Symptoms

Shortly after the replacement controller comes online, the survivor may reboot. Once it completes it will be upgraded to 7.x firmware.

Customer will no longer have access to their data. Any drive that was part of any volume will show up as offline. Drives that were either unassigned or a Global Hot Spare will be upgraded to Rev4 DACstore and work as designed in a 7.x array.

Attempts to import the Virtual Disks or Volume Groups may also see the following error message:

"Error 449 - A volume group being imported has incompatable drive(s)."

or

"Vdisk state is not valid."

Root Cause

Since June 16, 2008 spares have been loaded with 7.x firmware.

Even if a controller is properly replaced there is a possibility for a peering problem to cause the reboot of the surviving controller. If this happens the replacement controller becomes the survivor and Auto Code Syncs the newly rebooted controller.

Controller replacement should NEVER be done while the array is powered off unless under the specific direction of Engineering.

Corrective Action

Workaround:

If the array has any unassigned or hot spare drives it may be possible to import Virtual Disks or Volume Groups and restore data access. In such a configuration those unassigned or hot spare drives would have been updated to a Rev 4 DACstore. This gives you a valid 7.x configuration and should allow you to attempt to import the Virtual Disks or Volume Groups. This will not fix the licensing issue. This workaround will also require the Management Host to be running CAM 6.1 or higher or SANtricity 10.10.xx.xx or higher. If any issues are encountered during this process please escalate to PTS immediately.

Import Virtual Disks using CAM:

1. Select the array in the CAM naviagation tree.
2. Select Service Advisor in the CAM banner (upper right corner).
3. Expand Portable Virtual Disk Management.
4. Select Import a VIrtual Disk and follow the instructions.
5. If CAM 6.2 or higher is installed, Virtual Disks can be relabeled to fix
    the renumbering caused by the import. This is not a required step as the
    labels only need to be different.

Import Volume Groups using SANtricity:

1. Double-click the array in the Enterprise Management window.
2. Select a Volume Group in the Logical side of the Array Management window.

    Note: exported Groups have a small yellow arrow pointing to the right in
    the logical view and yellow diamonds in the physical view.

3. Select Advanced -> Maintanence -> Import Volume Group in the menu bar.
4. Click on OK in the pop-up window.
5. Click on Import, then answer yes in the next pop-up window.
6. Confirm that import was successful and click on Close.
7. Repeat until all Volume Groups are imported.
8. Volume Groups can be relabeled from Volume Group -> Rename menu
    option in the Array Management window.

If the customer has no unassigned or Hot Spare assigned drives or will not allow the completion of an upgrade to 7.x firmware, please escalate to PTS immediately so it can be arranged. Please note that this option will take longer than going foward and may not be doable in all cases.

Either option will still require the customer to have any Premium Feature Licenses to be regenerated. Please contact the License Center via the below URL;

http://www.sun.com/software/licensingcenter/

Resolution:

Crystal Maintenance 1 firmware (7.15.11.12) will contain a check to prevent a properly replaced controller from performing an unplanned upgrade to 7.x firmware. It will be installed on all future controller FRUs. Proper identification of the affected part numbers is covered below. FRUs with the proper firmware have been updated with a new revision:

375-3335-03   STK6140 RAID Controller, 2GB memory, 4 host ports
375-3375-03   STK6140 RAID Controller, 1GB memory, 2 host ports
375-3581-02   STK6140 RAID Controller, 2GB memory, 4 host ports
375-3582-02   STK6140 RAID Controller, 1GB memory, 2 host ports
371-1806-02   STK6540 RAID Controller, 2GB memory, 4 host ports
371-1810-02   STK6540 RAID Controller, 4GB memory, 4 host ports
371-1814-02   STK6540 RAID Controller, 8GB memory, 4 host ports

Identification of Affected Parts (how to):

FRU serial numbers come in the form of AAYWWZZZZZ where AA are an Alpha character supplier code, Y is a single digit year (8 for 2008), WW is the week in that year and ZZZZZ is a unique sequence number.

Any controller replacement FRU listed above with a serial number whose YWW digits are 824 or higher is a possible candidate for this issue when being used in an array running 6.x firwmare if the dash level of the part number is:

375-3335-02 or lower
375-3375-02 or lower
375-3581-01
375-3582-01
371-1806-01
371-1810-01
371-1814-01

References:

   BugID: 6757957
   Escalation ID: 71960691
   Resolution Patches: 140060-01, 140061-01, 140062-01
   ECO: 40462, 40919
   GSAP: 4649, 4701
   Radiance ID: 66091467

For information about FAB documents, its release processes, implementation strategies and billing information, go to the following URL:

http://tns.central/fab

For Sun Authorized Service Providers go to:

http://tns.central/SPE/fab/

In addition to the above you may email:

FAB-Manager@sun.com

Modification History
Changes since initial publication.

06-Jan-2009

Corrections to how board serial numbers are interpreted in the "How To" section.

09-Feb-2009

Additional corrections to how board serial numbers are interpreted in the "How To" section.

19-Mar-2009

Added 375-3335-02 and 375-3375-02 to Affected Parts and Resolution sections.

Internal Contributor/submitter
Bob.Deguc@Sun.COM

Internal Eng Responsible Engineer
Rich.Floyd@Sun.COM Responsible Manager: Brian.Hanvey@Sun.COM

Internal Services Knowledge Engineer
Joe.Davis@Sun.COM

Internal Eng Business Unit Group
NWS (Network Storage)

Internal Sun Alert & FAB Admin Info
18-Dec-2008: Completed draft and sent to Extended Review.
22-Dec-2008: No feedback from Ext Rvw - sending to Publish.

Attachments

This solution has no attachment