A SIM card in a JBOD Diskshelf attached to an S7000 at f/w 3420 while its peer SIM card is at f/w 3524 can lead to the Appliance CLI posting a message that lists the affected chassis as "faulted".

Asset ID:	1-73-1307357.1
Update Date:	2011-03-29
Keywords:

Solution Type FAB (standard) Sure

Solution 1307357.1 : A SIM card in a JBOD Diskshelf attached to an S7000 at f/w 3420 while its peer SIM card is at f/w 3524 can lead to the Appliance CLI posting a message that lists the affected chassis as "faulted".

Applies to:

Sun ZFS Storage 7120 - Version: Not Applicable to Not Applicable - Release: N/A to N/A
Sun Storage 7310 Unified Storage System - Version: Not Applicable to Not Applicable   [Release: N/A to N/A]
Sun ZFS Storage 7320 - Version: Not Applicable to Not Applicable   [Release: N/A to N/A]
Sun Storage 7410 Unified Storage System - Version: Not Applicable to Not Applicable   [Release: N/A to N/A]
Sun ZFS Storage 7420 - Version: Not Applicable to Not Applicable   [Release: N/A to N/A]
Information in this document applies to any platform.
__________

SUNBUG: 7020272

Affected Parts:

542-0194 - Disk Shelf 24x3.5" SAS-2 Interface Module (SIM)

Symptoms

When a SIM with 3520 is in the same JBOD with a SIM that has 3524, the Appliance BUI will not detect that SIM and all associated devices down that SIM's path will not go online.

An example of out put would look like this.

From the appliance CLI this situation may look like the SIM is absent:

pt7420-020:maintenance hardware> list

NAME       STATE     MANUFACTURER        MODEL                   SERIAL
chassis-000 pt7420-020 ok        Oracle                  Sun ZFS Storage 7420    1027DF0013
chassis-001 1031QAQ03A ok        Sun Microsystems, Inc. Sun Disk Shelf (SAS-2) 1031QAQ03A
chassis-002 1031QAQ030 ok        Sun Microsystems, Inc. Sun Disk Shelf (SAS-2) 1031QAQ030
chassis-003 1031QAQ042 faulted   Sun Microsystems, Inc. Sun Disk Shelf (SAS-2) 1031QAQ042
chassis-004 1020QAQ007 ok        Sun Microsystems, Inc. Sun Disk Shelf (SAS-2) 1020QAQ007
chassis-005 1031QAQ029 ok        Sun Microsystems, Inc. Sun Disk Shelf (SAS-2) 1031QAQ029
chassis-006 1031QAQ021 ok        Sun Microsystems, Inc. Sun Disk Shelf (SAS-2) 1031QAQ021
chassis-007 1020QAQ01A ok        Sun Microsystems, Inc. Sun Disk Shelf (SAS-2) 1020QAQ01A
chassis-008 1031QAQ022 ok        Sun Microsystems, Inc. Sun Disk Shelf (SAS-2) 1031QAQ022

pt7420-020:maintenance hardware> select chassis-003 select slot show

Slots:

          LABEL   STATE     MANUFACTURER            MODEL        SERIAL
slot-000 SIM 0   ok        Sun Microsystems, Inc. 3753633      2029QTF-1004QC1271
slot-001 SIM 1   absent    Sun Microsystems, Inc. -             -

There will likely be a problem reported with the JBOD:

pt7420-020:maintenance hardware>
pt7420-020:maintenance problems> show

Problems:

COMPONENT    DIAGNOSED            TYPE            DESCRIPTION
problem-000 2011-3-9 14:41:17    Major Defect    The JBOD '1031QAQ042' is
                                                  configured incorrectly

pt7420-020:maintenance problems> select problem-000 show

Properties:

                          uuid = 8c16786a-04e4-c7f9-e9b1-f4c059c4edd9
                          code = AK-8000-H7
                     diagnosed = 2011-3-9 14:41:17
                   phoned_home = never
                      severity = Major
                          type = Defect
                           url = http://sun.com/msg/AK-8000-H7
                   description = The JBOD '1031QAQ042' is configured
                                 incorrectly.
                        impact = The disks contained within the enclosure
                                 cannot be used as part of a storage pool.
                      response = None.
                        action = Correct the disk configuration to conform to a
                                 valid configuration.

Components:

component-000 100% 1031QAQ042: hc://:product-id=SUN-Storage-J4410:chassis-id=1031QAQ042:serial=1031QAQ042:part=Storage-J4410:revision=3524/ses-enclosure=2 (degraded)
                     Manufacturer: Sun Microsystems, Inc.
                     Part number: Storage-J4410
                     Model: Sun Disk Shelf (SAS-2)
                     Revision: 3524

pt7420-020:maintenance problems>

Impact

The Diskshelf JBOD will be running in a degraded state.

Having a SIM card in a JBOD Diskshelf attached to an S7000 appliance at firmware revision 3420 while its peer SIM card is at firmware revision 3524 can lead to the Appliance CLI posting a message that lists the affected chassis as "faulted".

It will also show that the SIM card with 3520 FW loaded on it is reported as "absent", and the "maintenance problems> show" will indicate the that "The JBOD 'xxxxxxxxxx' is configured incorrectly". In certain conditions the entire JBOD will not be noticeable from the "HW View".

Changes

Contributing Factors

There are 3 Scenarios leading to how the firmware inconsistency could happen:

1) At the time of an installation or initial upgrade to Q3.1.1 the installation/upgrade
is interrupted leaving the SIM cards in different states.

2) A FRU replacement that has FW 3520 on it is installed into a chassis where
the other SIM card is already running Q3.1.1 (FW 3524).

3) A Customer is running at a version of AK build prior to Q3.1.1 and a SIM FRU
is swapped into their JBOD that has 3524 on it.

Cause

Root Cause

The cause of this is the incompatibility between SIM FW 3520 and 3524.

The formal fix is To Be Determined. Running the sesupgrade command from the Solaris Shell is the workaround for now.

Solution

Workaround

Upgrading the SIM with FW 3520 to FW 3524 using the manual sesupgrade command will alleviate this incompatibility.

IMPORTANT! It is imperative that all customers upgrade their Amber Road appliance to a minimum level of Q3.1.1 (aka 2010.08.17.1.1) so that this issue will not reoccur.

Resolution

Note: The following should only be performed by trained Oracle service personnel.

To rectify this problem, if the customer is not already at Q3.1.1, upgrade their appliance first.

IMPORTANT: WAIT FOR ALL HARDWARE UPDATES TO FULLY COMPLETE BEFORE PROCEDING WITH THIS FAB!

Check the versions of your system, including the SIM FW in the Diskshelf. If you still have the FW mismatched between SIMs, execute the rest of this FAB.

To manually upgrade the SIMs in your system, perform the following steps:

1-1) drop into the Solaris shell, identify that there is a downlevel SIM, then manually update.

pt7420-020:maintenance problems> confirm shell
Executing raw shell; "exit" to return to appliance shell ...

+-----------------------------------------------------------------------------+
| You are entering the operating system shell. By confirming this action in |
| the appliance shell you have agreed that THIS ACTION MAY VOID ANY SUPPORT |
| AGREEMENT. If you do not agree to this -- or do not otherwise understand |
| what you are doing -- you should type "exit" at the shell prompt. EVERY   |
| COMMAND THAT YOU EXECUTE HERE IS AUDITED, and support personnel may use    |
| this audit trail to substantiate invalidating your support contract. The |
| operating system shell is NOT a supported mechanism for managing this      |
| appliance, and COMMANDS EXECUTED HERE MAY DO IRREPARABLE HARM.             |
|                                                                             |
| NOTHING SHOULD BE ATTEMPTED HERE BY UNTRAINED SUPPORT PERSONNEL UNDER ANY |
| CIRCUMSTANCES. This appliance is a non-traditional operating system       |
| environment, and expertise in a traditional operating system environment   |
| in NO WAY constitutes training for supporting this appliance. THOSE WITH |
| EXPERTISE IN OTHER SYSTEMS -- HOWEVER SUPERFICIALLY SIMILAR -- ARE MORE    |
| LIKELY TO MISTAKENLY EXECUTE OPERATIONS HERE THAT WILL DO IRREPARABLE      |
| HARM. Unless you have been explicitly trained on supporting this          |
| appliance via the operating system shell, you should immediately return    |
| to the appliance shell.                                                    |
|                                                                             |
| Type "exit" now to return to the appliance shell.                          |
+-----------------------------------------------------------------------------+

1-2) pt7420-020# sesupgrade /dev/scsi/ses/*

                        device: /dev/scsi/ses/c0t500163600024BE7Dd0
                        vendor: SUN
                       product: Storage J4410
                      revision: 3524
                        serial: 1031QAQ03A
                current status: none

                        device: /dev/scsi/ses/c0t500163600024CC7Dd0
                        vendor: SUN
                       product: Storage J4410
                      revision: 3524
                        serial: 1031QAQ03A
                current status: none

                        device: /dev/scsi/ses/c0t500163600025643Dd0
                        vendor: SUN
                       product: Storage J4410
                      revision: 3524
                        serial: 1031QAQ030
                current status: none

                        device: /dev/scsi/ses/c0t5001636000256CFDd0
                        vendor: SUN
                       product: Storage J4410
                      revision: 3520
                        serial: 1031QAQ042
                current status: none

                        device: /dev/scsi/ses/c0t50016360002572FDd0
                        vendor: SUN
                       product: Storage J4410
                      revision: 3524
                        serial: 1031QAQ042
                current status: none

                        device: /dev/scsi/ses/c0t50016360002576FDd0
                        vendor: SUN
                       product: Storage J4410
                      revision: 3524
                        serial: 1020QAQ007
                current status: none

                        device: /dev/scsi/ses/c0t5001636000257DBDd0
                        vendor: SUN
                       product: Storage J4410
                      revision: 3524
                        serial: 1031QAQ029
                current status: none

                        device: /dev/scsi/ses/c0t500163600025803Dd0
                        vendor: SUN
                       product: Storage J4410
                      revision: 3524
                        serial: 1020QAQ007
                current status: none

                        device: /dev/scsi/ses/c0t50016360002590BDd0
                        vendor: SUN
                       product: Storage J4410
                      revision: 3524
                        serial: 1031QAQ021
                current status: none

                        device: /dev/scsi/ses/c0t50016360002593BDd0
                        vendor: SUN
                       product: Storage J4410
                      revision: 3524
                        serial: 1031QAQ021
                current status: none

                        device: /dev/scsi/ses/c0t5001636000259DFDd0
                        vendor: SUN
                       product: Storage J4410
                      revision: 3524
                        serial: 1031QAQ029
                current status: none

                        device: /dev/scsi/ses/c0t5001636000259EBDd0
                        vendor: SUN
                       product: Storage J4410
                      revision: 3524
                        serial: 1031QAQ030
                current status: none

                        device: /dev/scsi/ses/c0t500163600026423Dd0
                        vendor: SUN
                       product: Storage J4410
                      revision: 3524
                        serial: 1020QAQ01A
                current status: none

                        device: /dev/scsi/ses/c0t500163600026433Dd0
                        vendor: SUN
                       product: Storage J4410
                      revision: 3524
                        serial: 1020QAQ01A
                current status: none

                        device: /dev/scsi/ses/c0t500163600026633Dd0
                        vendor: SUN
                       product: Storage J4410
                      revision: 3524
                        serial: 1031QAQ022
                current status: none

                        device: /dev/scsi/ses/c0t50016360002664FDd0
                        vendor: SUN
                       product: Storage J4410
                      revision: 3524
                        serial: 1031QAQ022
                current status: none

pt7420-020#

Verify the specific device:

1-3) pt7420-020# sesupgrade /dev/scsi/ses/c0t5001636000256CFDd0

                        device: /dev/scsi/ses/c0t5001636000256CFDd0
                        vendor: SUN
                       product: Storage J4410
                      revision: 3520
                        serial: 1031QAQ042
                current status: none

pt7420-020#

Change to the firmware repository and manually update the SIM:

1-4) pt7420-020# cd /usr/lib/ak/fw/SUN-Storage-J4410-3524/
1-5) pt7420-020# ls

J4410.3524.fw manifest.xml   plugin.so

1-6) pt7420-020# sesupgrade -m 6 -b 5 -i J4410.3524.fw /dev/scsi/ses/c0t5001636000256CFDd0

                        device: /dev/scsi/ses/c0t5001636000256CFDd0
                        vendor: SUN
                       product: Storage J4410
                      revision: 3520
                        serial: 1031QAQ042
                current status: none

      downloading 424224 bytes: ok
                        status: completed (available)

Wait 3 minutes for the upgrade to complete and the SIM to reset and then verify the upload was successful:

1-7) pt7420-020# sesupgrade /dev/scsi/ses/c0t5001636000256CFDd0

                        device: /dev/scsi/ses/c0t5001636000256CFDd0
                        vendor: SUN
                       product: Storage J4410
                      revision: 3524
                        serial: 1031QAQ042
                current status: none

NOTE: Please be sure to wait 3 minutes between upgrades of each SIM.

Please repeat steps 1-1 through 1-7 for each SIM this has the mismatched FW residing on it.

**If the bui or cli does not show correct jbod information for any of the effected jbods it might require a restart of akd** If so please do the following from the appliance cli:

2-1) mall-maguro1:> maintenance
2-2) small-maguro1:maintenance> show
2-3) small-maguro1:maintenance> system
2-4) small-maguro1:maintenance system> restart
2-5) This will restart the management system. Are you sure? (Y/N)

At this point you will experience temporary loss of the bui or management cli (approximately 2 minutes). Once the akd restarts please check to make sure proper information about the affected jbod is now displayed (correct number of paths, correct sim firmware etc).

If for any reason the SIM seems to fail the reboot after the firmware update you might have to reseat the SIM card to do a cold reboot. After this you need to check the firmware version and maybe have to reapply the manual update.

Identification of Affected Parts (how to):

Once the akd restarts please check to make sure proper information about the affected jbod is now displayed (correct number of paths , correct SIM firmware etc. Access the SIMs via the BUI after insertion of the SIM to asses if the BUI can properly read that SIM and FW.

Contacts:

Contributor: Andy.Laker@oracle.com
Responsible Engineer: Christopher.Burgess@oracle.com
Responsible Manager: Cheri.Williams@oracle.com
Business Unit Group: NWS (Network Storage)

For information about FAB documents, its release processes, implementation strategies and billing information, go to the following URL:

   https://sunspace.sfbay.sun.com/display/Onestop/FAB%20(Field%20Action%20Bulletin)

In addition to the above you may email:

   * FAB-Manager_US@Oracle.COM

Attachments

This solution has no attachment