Sun Microsystems, Inc.  Sun System Handbook - ISO 3.4 June 2011 Internal/Partner Edition
   Home | Current Systems | Former STK Products | EOL Systems | Components | General Info | Search | Feedback

Asset ID: 1-73-1307357.1
Update Date:2011-03-29
Keywords:

Solution Type  FAB (standard) Sure

Solution  1307357.1 :   A SIM card in a JBOD Diskshelf attached to an S7000 at f/w 3420 while its peer SIM card is at f/w 3524 can lead to the Appliance CLI posting a message that lists the affected chassis as "faulted".  


Related Items
  • Sun Storage 7310 Unified Storage System
  •  
  • Sun Storage 7410 Unified Storage System
  •  
  • Sun ZFS Storage 7120
  •  
  • Sun ZFS Storage 7320
  •  
  • Sun ZFS Storage 7420
  •  
Related Categories
  • GCS>Sun Microsystems>Sun FAB>Standard>Reactive
  •  




In this Document
  Symptoms
  Changes
  Cause
  Solution


Oracle Confidential (PARTNER). Do not distribute to customers
Reason: FABs available to Internals and Partners only

Applies to:

Sun ZFS Storage 7120 - Version: Not Applicable to Not Applicable - Release: N/A to N/A
Sun Storage 7310 Unified Storage System - Version: Not Applicable to Not Applicable   [Release: N/A to N/A]
Sun ZFS Storage 7320 - Version: Not Applicable to Not Applicable   [Release: N/A to N/A]
Sun Storage 7410 Unified Storage System - Version: Not Applicable to Not Applicable   [Release: N/A to N/A]
Sun ZFS Storage 7420 - Version: Not Applicable to Not Applicable   [Release: N/A to N/A]
Information in this document applies to any platform.
__________

SUNBUG: 7020272

Affected Parts:

542-0194 - Disk Shelf 24x3.5" SAS-2 Interface Module (SIM)

Symptoms

When a SIM with 3520 is in the same JBOD with a SIM that has 3524, the Appliance BUI will not detect that SIM and all associated devices down that SIM's path will not go online.

An example of out put would look like this.

From the appliance CLI this situation may look like the SIM is absent:

pt7420-020:maintenance hardware> list


  NAME                 STATE     MANUFACTURER              MODEL                   SERIAL     
chassis-000  pt7420-020  ok        Oracle                  Sun ZFS Storage 7420    1027DF0013 
chassis-001  1031QAQ03A  ok        Sun Microsystems, Inc.  Sun Disk Shelf (SAS-2)  1031QAQ03A 
chassis-002  1031QAQ030  ok        Sun Microsystems, Inc.  Sun Disk Shelf (SAS-2)  1031QAQ030 
chassis-003  1031QAQ042  faulted   Sun Microsystems, Inc.  Sun Disk Shelf (SAS-2)  1031QAQ042 
chassis-004  1020QAQ007  ok        Sun Microsystems, Inc.  Sun Disk Shelf (SAS-2)  1020QAQ007 
chassis-005  1031QAQ029  ok        Sun Microsystems, Inc.  Sun Disk Shelf (SAS-2)  1031QAQ029 
chassis-006  1031QAQ021  ok        Sun Microsystems, Inc.  Sun Disk Shelf (SAS-2)  1031QAQ021 
chassis-007  1020QAQ01A  ok        Sun Microsystems, Inc.  Sun Disk Shelf (SAS-2)  1020QAQ01A 
chassis-008  1031QAQ022  ok        Sun Microsystems, Inc.  Sun Disk Shelf (SAS-2)  1031QAQ022 

pt7420-020:maintenance hardware> select chassis-003 select slot show

Slots:


          LABEL   STATE     MANUFACTURER            MODEL        SERIAL
slot-000  SIM 0   ok        Sun Microsystems, Inc.  3753633      2029QTF-1004QC1271
slot-001  SIM 1   absent    Sun Microsystems, Inc.  -             -


There will likely be a problem reported with the JBOD:

pt7420-020:maintenance hardware>
pt7420-020:maintenance problems> show

Problems:


COMPONENT    DIAGNOSED            TYPE            DESCRIPTION
problem-000  2011-3-9 14:41:17    Major Defect    The JBOD '1031QAQ042' is
                                                  configured incorrectly


pt7420-020:maintenance problems> select problem-000 show

Properties:

                          uuid = 8c16786a-04e4-c7f9-e9b1-f4c059c4edd9
                          code = AK-8000-H7
                     diagnosed = 2011-3-9 14:41:17
                   phoned_home = never
                      severity = Major
                          type = Defect
                           url = http://sun.com/msg/AK-8000-H7
                   description = The JBOD '1031QAQ042' is configured
                                 incorrectly.
                        impact = The disks contained within the enclosure
                                 cannot be used as part of a storage pool.
                      response = None.
                        action = Correct the disk configuration to conform to a
                                 valid configuration.


Components:

component-000  100%  1031QAQ042: hc://:product-id=SUN-Storage-J4410:chassis-id=1031QAQ042:serial=1031QAQ042:part=Storage-J4410:revision=3524/ses-enclosure=2 (degraded)
                     Manufacturer: Sun Microsystems, Inc.
                     Part number: Storage-J4410
                     Model: Sun Disk Shelf (SAS-2)
                     Revision: 3524


pt7420-020:maintenance problems>

Impact

The Diskshelf JBOD will be running in a degraded state.

Having a SIM card in a JBOD Diskshelf attached to an S7000 appliance at firmware revision 3420 while its peer SIM card is at firmware revision 3524 can lead to the Appliance CLI posting a message that lists the affected chassis as "faulted".

It will also show that the SIM card with 3520 FW loaded on it is reported as "absent", and the "maintenance problems> show" will indicate the that "The JBOD 'xxxxxxxxxx' is configured incorrectly". In certain conditions the entire JBOD will not be noticeable from the "HW View".

Changes

Contributing Factors

There are 3 Scenarios leading to how the firmware inconsistency could happen:

1) At the time of an installation or initial upgrade to Q3.1.1 the installation/upgrade
    is interrupted leaving the SIM cards in different states.

2) A FRU replacement that has FW 3520 on it is installed into a chassis where
    the other SIM card is already running Q3.1.1 (FW 3524).

3) A Customer is running at a version of AK build prior to Q3.1.1 and a SIM FRU
    is swapped into their JBOD that has 3524 on it.

Cause

Root Cause

The cause of this is the incompatibility between SIM FW 3520 and 3524.

The formal fix is To Be Determined. Running the sesupgrade command from the Solaris Shell is the workaround for now.

Solution

Workaround

Upgrading the SIM with FW 3520 to FW 3524 using the manual sesupgrade command will alleviate this incompatibility.

IMPORTANT! It is imperative that all customers upgrade their Amber Road appliance to a minimum level of Q3.1.1 (aka 2010.08.17.1.1) so that this issue will not reoccur.

Resolution

Note: The following should only be performed by trained Oracle service personnel.

To rectify this problem, if the customer is not already at Q3.1.1, upgrade their appliance first.

IMPORTANT: WAIT FOR ALL HARDWARE UPDATES TO FULLY COMPLETE BEFORE PROCEDING WITH THIS FAB!

Check the versions of your system, including the SIM FW in the Diskshelf. If you still have the FW mismatched between SIMs, execute the rest of this FAB.

To manually upgrade the SIMs in your system, perform the following steps:

1-1) drop into the Solaris shell, identify that there is a downlevel SIM, then manually update.

pt7420-020:maintenance problems> confirm shell
Executing raw shell; "exit" to return to appliance shell ...

+-----------------------------------------------------------------------------+
|  You are entering the operating system shell.  By confirming this action in |
|  the appliance shell you have agreed that THIS ACTION MAY VOID ANY SUPPORT  |
|  AGREEMENT.  If you do not agree to this -- or do not otherwise understand  |
|  what you are doing -- you should type "exit" at the shell prompt.  EVERY   |
|  COMMAND THAT YOU EXECUTE HERE IS AUDITED, and support personnel may use    |
|  this audit trail to substantiate invalidating your support contract.  The  |
|  operating system shell is NOT a supported mechanism for managing this      |
|  appliance, and COMMANDS EXECUTED HERE MAY DO IRREPARABLE HARM.             |
|                                                                             |
|  NOTHING SHOULD BE ATTEMPTED HERE BY UNTRAINED SUPPORT PERSONNEL UNDER ANY  |
|  CIRCUMSTANCES.  This appliance is a non-traditional operating system       |
|  environment, and expertise in a traditional operating system environment   |
|  in NO WAY constitutes training for supporting this appliance.  THOSE WITH  |
|  EXPERTISE IN OTHER SYSTEMS -- HOWEVER SUPERFICIALLY SIMILAR -- ARE MORE    |
|  LIKELY TO MISTAKENLY EXECUTE OPERATIONS HERE THAT WILL DO IRREPARABLE      |
|  HARM.  Unless you have been explicitly trained on supporting this          |
|  appliance via the operating system shell, you should immediately return    |
|  to the appliance shell.                                                    |
|                                                                             |
|  Type "exit" now to return to the appliance shell.                          |
+-----------------------------------------------------------------------------+


1-2) pt7420-020# sesupgrade /dev/scsi/ses/*

                        device: /dev/scsi/ses/c0t500163600024BE7Dd0
                        vendor: SUN
                       product: Storage J4410
                      revision: 3524
                        serial: 1031QAQ03A
                current status: none

                        device: /dev/scsi/ses/c0t500163600024CC7Dd0
                        vendor: SUN
                       product: Storage J4410
                      revision: 3524
                        serial: 1031QAQ03A
                current status: none

                        device: /dev/scsi/ses/c0t500163600025643Dd0
                        vendor: SUN
                       product: Storage J4410
                      revision: 3524
                        serial: 1031QAQ030
                current status: none

                        device: /dev/scsi/ses/c0t5001636000256CFDd0
                        vendor: SUN
                       product: Storage J4410
                      revision: 3520
                        serial: 1031QAQ042
                current status: none

                        device: /dev/scsi/ses/c0t50016360002572FDd0
                        vendor: SUN
                       product: Storage J4410
                      revision: 3524
                        serial: 1031QAQ042
                current status: none

                        device: /dev/scsi/ses/c0t50016360002576FDd0
                        vendor: SUN
                       product: Storage J4410
                      revision: 3524
                        serial: 1020QAQ007
                current status: none

                        device: /dev/scsi/ses/c0t5001636000257DBDd0
                        vendor: SUN
                       product: Storage J4410
                      revision: 3524
                        serial: 1031QAQ029
                current status: none

                        device: /dev/scsi/ses/c0t500163600025803Dd0
                        vendor: SUN
                       product: Storage J4410
                      revision: 3524
                        serial: 1020QAQ007
                current status: none

                        device: /dev/scsi/ses/c0t50016360002590BDd0
                        vendor: SUN
                       product: Storage J4410
                      revision: 3524
                        serial: 1031QAQ021
                current status: none

                        device: /dev/scsi/ses/c0t50016360002593BDd0
                        vendor: SUN
                       product: Storage J4410
                      revision: 3524
                        serial: 1031QAQ021
                current status: none

                        device: /dev/scsi/ses/c0t5001636000259DFDd0
                        vendor: SUN
                       product: Storage J4410
                      revision: 3524
                        serial: 1031QAQ029
                current status: none

                        device: /dev/scsi/ses/c0t5001636000259EBDd0
                        vendor: SUN
                       product: Storage J4410
                      revision: 3524
                        serial: 1031QAQ030
                current status: none

                        device: /dev/scsi/ses/c0t500163600026423Dd0
                        vendor: SUN
                       product: Storage J4410
                      revision: 3524
                        serial: 1020QAQ01A
                current status: none

                        device: /dev/scsi/ses/c0t500163600026433Dd0
                        vendor: SUN
                       product: Storage J4410
                      revision: 3524
                        serial: 1020QAQ01A
                current status: none

                        device: /dev/scsi/ses/c0t500163600026633Dd0
                        vendor: SUN
                       product: Storage J4410
                      revision: 3524
                        serial: 1031QAQ022
                current status: none

                        device: /dev/scsi/ses/c0t50016360002664FDd0
                        vendor: SUN
                       product: Storage J4410
                      revision: 3524
                        serial: 1031QAQ022
                current status: none


pt7420-020#

Verify the specific device:

1-3) pt7420-020# sesupgrade /dev/scsi/ses/c0t5001636000256CFDd0

                        device: /dev/scsi/ses/c0t5001636000256CFDd0
                        vendor: SUN
                       product: Storage J4410
                      revision: 3520
                        serial: 1031QAQ042
                current status: none


pt7420-020#

Change to the firmware repository and manually update the SIM:

1-4) pt7420-020# cd /usr/lib/ak/fw/SUN-Storage-J4410-3524/
1-5) pt7420-020# ls                
     
J4410.3524.fw  manifest.xml   plugin.so

1-6) pt7420-020# sesupgrade -m 6 -b 5 -i J4410.3524.fw /dev/scsi/ses/c0t5001636000256CFDd0

                        device: /dev/scsi/ses/c0t5001636000256CFDd0
                        vendor: SUN
                       product: Storage J4410
                      revision: 3520
                        serial: 1031QAQ042
                current status: none


      downloading 424224 bytes: ok
                        status: completed (available)

Wait 3 minutes for the upgrade to complete and the SIM to reset and then verify the upload was successful:

1-7) pt7420-020# sesupgrade /dev/scsi/ses/c0t5001636000256CFDd0

                        device: /dev/scsi/ses/c0t5001636000256CFDd0
                        vendor: SUN
                       product: Storage J4410
                      revision: 3524
                        serial: 1031QAQ042
                current status: none


NOTE: Please be sure to wait 3 minutes between upgrades of each SIM.

Please repeat steps 1-1 through 1-7 for each SIM this has the mismatched FW residing on it.

**If the bui or cli does not show correct jbod information for any of the effected jbods it might require a restart of akd** If so please do the following from the appliance cli:

2-1) mall-maguro1:> maintenance
2-2) small-maguro1:maintenance> show
2-3) small-maguro1:maintenance> system
2-4) small-maguro1:maintenance system> restart
2-5) This will restart the management system. Are you sure? (Y/N)

At this point you will experience temporary loss of the bui or management cli (approximately 2 minutes). Once the akd restarts please check to make sure proper information about the affected jbod is now displayed (correct number of paths, correct sim firmware etc).

If for any reason the SIM seems to fail the reboot after the firmware update you might have to reseat the SIM card to do a cold reboot. After this you need to check the firmware version and maybe have to reapply the manual update.

Identification of Affected Parts (how to):

Once the akd restarts please check to make sure proper information about the affected jbod is now displayed (correct number of paths , correct SIM firmware etc. Access the SIMs via the BUI after insertion of the SIM to asses if the BUI can properly read that SIM and FW.

Contacts:

Contributor: Andy.Laker@oracle.com
Responsible Engineer: Christopher.Burgess@oracle.com
Responsible Manager: Cheri.Williams@oracle.com
Business Unit Group: NWS (Network Storage)

For information about FAB documents, its release processes, implementation strategies and billing information, go to the following URL:

   https://sunspace.sfbay.sun.com/display/Onestop/FAB%20(Field%20Action%20Bulletin)

In addition to the above you may email:

   * FAB-Manager_US@Oracle.COM


Attachments
This solution has no attachment
  Copyright © 2011 Sun Microsystems, Inc.  All rights reserved.
 Feedback