Sun Fire T1000 Server
 

GCS>Sun Microsystems>Sun FAB>Hardware Remediation>Mandatory
 

• Part No: 375-3357-02/03
• Part Description: PCI Express - Dual Ultra-320 SCSI (Rhea HBA)

• Xoption Number: SG-XPCIE2SCSIU320Z
• Xoption Description: Dual-Channel Ultra320 LVD SCSI PCI Express Adapter

Dual Ultra-320 SCSI Rhea card (Sun p/n 375-3357-02/03), when installed in a Sun Fire T1000, generates excessive RTO and LUP events as a result of PCI-Express specification violation.  These events may result in lost data, data corruption and system hangs.

This issue only arises when a -02 or -03 Rhea card (p/n 375-3357) is plugged into a T1000 system.

The easiest way to expose the issue is to run the storage tests included in SunVTS.  If a customer has a revision -02 or -03 Rhea card, there will be immediate warnings about excessive RTO and LUP events causing the message queue to overflow and ereports to drop.

Excessive RTO and LUP events which may result in lost data, data corruption and system hangs.

With a load on the HBA using SunVTS HBA test or applications doing read and/or writes to the storage device(s) attached to the HBA, the appearance of the error messages and the hang condition accelerates.

The "Dropping ereports" messages would appear first, followed by a system hang.

Sun Fire T1000 systems will show the following error messages to the console and saved to messages files:

"SC Alert: Dropping ereports, message queue is full."
...
Nov 01 08:05:06: 00060029: "Dropping ereports, message queue is full."
Nov 01 08:05:08: 00060029: "Dropping ereports, message queue is full."
Nov 01 08:05:09: 00060029: "Dropping ereports, message queue is full."
...

A dump of the fma logs after rebooting the hung machine shows PCI-E retraining (Replay Timeout RTO) and PCI-E link up (LUP) events when the hang occurs:

Nov 01 12:40:30.357937920 ereport.io.fire.pec.rto
Nov 01 12:40:30.404282880 ereport.io.fire.pec.rto
Nov 01 12:05:03.389229600 ereport.io.fire.pec.lup
Nov 01 12:05:03.389229600 ereport.io.fire.pec.lup
Nov 01 12:40:43.300991360 ereport.io.fire.pec.rto
Nov 01 12:40:44.359243200 ereport.io.fire.pec.rto
...

A bug has been identified where the Rhea card causes a lot of RTO and LUP events when installed in T1000 servers.  The root cause is that the Rhea card does not return the expected ACK DLLP on the PCI Express link back to the Fire chip on the T1000 within the correct number of clocks and violates the PCI Express specification.  No other SPARC or AMD systems have shown sensitivity to this issue.

Corrective action was made available in Manufacturing via a dash roll or the Rhea card to -04 per ECO# WO_35411 as of March 1, 2007.

Corrective action was made available in Services via GSAP# 3881 as of March 13, 2007.

Implementation Target Completion Dates:

AMER: September 12, 2008
APAC: September 12, 2008
EMEA: September 12, 2008

Replacement Time Estimate: 30 minutes

Hot Swappable: No

Proactively contact customers showing both the Rhea HBA and a Sun Fire T1000 on the same Sales Order and proactively replace the 375-3357-02 or -03 with a 375-3357-04 (or above).

For all other customers owning a Rhea HBA with part number 375-3357-02 or -03, replace upon failure or at customer request with part number 375-3357-04 (or above), but only after ascertaining the replacement Rhea HBA is to be installed into a T1000.

For Rhea HBAs installed in other than a T1000, no action is required.

For the Mandatory portion of this FCO, the Customer List is available on SunFIT;

   http://sunfit/

In addition, a Sun Legal approved Customer Letter is available via the below URL;

  http://sdpsweb.central/FIN_FCO/FCO/A0279-1/SPE/Rhea_HBA-Customer_Letter.sxw

How to Determine if a Rhea HBA is Installed:

To determine if a Rhea HBA is installed on a T1000 system, the following command can be run (look for PCI-E slot 0 for the presence of LSI,1030):

$ prtdiag -v
Location    Type  Slot     Path
------------------------- ---------
MB/PCIE0   PCIE      0     /pci@780/pci@0/scsi@8
scsi-pci1000,30  LSI,1030
MB/PCIE0   PCIE      0     /pci@780/pci@0/scsi@8,1
scsi-pci1000,30  LSI,1030
...

If a Rhea HBA is present, it's dash level must then be visually inspected.  Only dash levels -02 and -03 are impacted by this issue.

•  Added "How to Determine..." section.  Added readiness statement under "Hardware Remediation and Material Availability Details" section.   Added Hotswap statement in "Resolution" section.  Added additional details in "Symptoms" section.

Sun Alert 102991 is being published to inform customers of this issue/resolution.

For questions or feedback on this asset, send email to FCO_279_Discussion@sunwebcollab.east.sun.com

All Regions were materially ready at the time of this asset's publication.

• Other: Sun Alert 102991

This solution has no attachment