Sun Microsystems, Inc.  Sun System Handbook - ISO 3.4 June 2011 Internal/Partner Edition
   Home | Current Systems | Former STK Products | EOL Systems | Components | General Info | Search | Feedback

Asset ID: 1-73-1001031.1
Update Date:2010-07-06
Keywords:

Solution Type  FAB (standard) Sure

Solution  1001031.1 :   Emulex LP10000 PCI-X HBA Operating in 33MHz Mode Will Report PCI Bus Errors Resulting in the HBA not Being Accessible  


Related Items
  • Sun Fire E6900 Server
  •  
  • Sun Fire 6800 Server
  •  
  • Sun Fire E4900 Server
  •  
  • Sun Fire 4800 Server
  •  
  • Sun Fire V1280 Server
  •  
  • Sun Fire E2900 Server
  •  
Related Categories
  • GCS>Sun Microsystems>Sun FAB>Standard>Reactive
  •  

PreviouslyPublishedAs
201358


Product
Sun Fire 4800 Server
Sun Fire 6800 Server
Sun Fire E6900 Server
Sun Fire E2900 Server
Sun Fire V1280 Server
Sun Fire E4900 Server

Bug Id
<SUNBUG: 6425501>

Part
  • Part No: 540-6509
  • Part Description: PCI-X IO Board
Part
  • Part No: 541-1486
  • Part Description: PCI-X IB_SSC Board
Part
  • Part No: 541-0382
  • Part Description: PCI+ IB_SSC Board

Impact

Storage devices connected to Emulex LP10000 PCI-X HBA (Sun PN# 375-3305 and PN# 375-3304) will not be accessible due to PCI bus errors.  In some cases, it may not be possible to install Solaris on systems/domains with Emulex LP10000 HBAs running in 33MHz mode because of PCI bus errors.  


Contributing Factors

This issue is seen in the following configurations:

  • Emulex LP10000 cards that are installed in 33MHz-only slots of a PCI-X IO Assy.
  • Emulex LP10000 cards that are installed in a slot that shares a PCI bus with a slot that has a 33MHz card in a PCI-X IO Assy and PCI-X IB_SSC Assy. For example, a 33MHz card installed in slot 6, and a LP10000 card installed in slot 7 of a PCI-X IO Assembly.

Note: This issue is NOT seen with Emulex LP10000 HBAs installed in the older PCI IO Assy or the PCI IB_SSC Assy.

Please refer to the Sun System Handbook and related Info Docs for the detailed IO PCI bus layout of the various affected IO Assemblies.  

From "prtdiag" output of an affected system/domain:

/N0/IB6/P1  PCI   25   B    5    100  100  2,0  ok    SUNW,emlxs-pci10df,fc00/fp (fp)   LP10000DC-S
/N0/IB6/P1  PCI   25   B    5    100  100  2,1  ok    SUNW,emlxs-pci10df,fc00/fp (fp)   LP10000DC-S
/N0/IB6/P1  PCI   25   A    6    33   100  2,0  ok    pci-pci1011,24.3/pci108e,1000     pci-bridge
/N0/IB6/P1  PCI   25   A    6    33   100  0,0  ok    pci108e,1000-pci108e,1000.1
/N0/IB6/P1  PCI   25   A    6    33   100  0,1  ok    SUNW,hme-pci108e,1001             SUNW,cheerio
/N0/IB6/P1  PCI   25   A    6    33   100  4,0  ok    SUNW,isptwo-pci1077,1020/sd (blo+ QLGC,ISP1040B
-> /N0/IB6/P1  PCI   25   A    7    33   100  1,0  ok    SUNW,emlxs-pci10df,fc00/fp (fp)   LP10000DC-S  <---
-> /N0/IB6/P1  PCI   25   A    7    33   100  1,1  ok    SUNW,emlxs-pci10df,fc00/fp (fp)   LP10000DC-S  <---
========================= Active Boards for Domain ===========================
Board        Receptacle    Occupant
FRU Name   Type         Status        Status        Condition Info
---------  -----------  -----------   ------------  --------- ----------------------------------------
/N0/SB2    CPU_V3       connected     configured    ok        powered-on, assigned
/N0/SB4    CPU_V2       connected     configured    ok        powered-on, assigned
-> /N0/IB6    PCI-X_I/O_B  connected     configured    ok        powered-on, assigned <--

 

Note that the LP10000 card installed in slot 7 is running at 33MHz (In the above scenario, the PCI bus is shared by a 33MHz Sun Swift card in slot 6 and LP10000 card in slot 7, hence the LP10000 card also runs at 33MHz as observed in the above "prtdiag" output). Also note that the IO Assembly in slot IB6 is a PCI-X IO Assembly.


Symptoms

When this issue occurs, the following type of errors are seen on the console (also recorded in /var/adm/messages):

Aug 23 11:34:36 qame10-a scsi: WARNING: /ssm@0,0/pci@18,700000/SUNW,emlxs@2/fp@0,0/ssd@w500000e010763ff1,0 (ssd24):
Aug 23 11:34:36 qame10-a        SCSI transport failed: reason 'timeout': retrying command
Aug 23 11:34:36 qame10-a pcisch: WARNING: pcisch0 (pci@18,700000): PCI fault log start:
Aug 23 11:34:36 qame10-a pcisch: PCI iommu error
Aug 23 11:34:36 qame10-a pcisch: pcisch0: Error 1 on IOMMU TLB entry 2:
Aug 23 11:34:36 qame10-a        Context=0 not Writable not Streamable
Aug 23 11:34:36 qame10-a        PCI Page Size=8k Address in page c1320000
Aug 23 11:34:36 qame10-a pcisch: Memory: Valid not Cacheable Page Frame=0
Aug 23 11:34:36 qame10-a pcisch: pcisch0 (pci@18,700000): PBM AFSR=0x0.00000000
Aug 23 11:34:36 qame10-a pcisch: dwordmask=0 bytemask=0
Aug 23 11:34:36 qame10-a pcisch: pcisch0 (pci@18,700000): PCI primary error (0):
Aug 23 11:34:36 qame10-a pcisch: pcisch0 (pci@18,700000): PCI secondary error (0):
Aug 23 11:34:36 qame10-a pcisch: pcisch0 (pci@18,700000): PBM AFAR 0.00000000:
Aug 23 11:34:36 qame10-a pcisch: WARNING: pcisch0: PCI config space CSR=0xaa0<signaled-target-abort>Aug 23 11:34:36 qame10-a pcisch: pcisch0 (pci@18,700000): PCI fault log end.
Aug 23 11:34:36 qame10-a pcisch: Scrubbing PCI iommu TLB entries
Aug 23 11:34:36 qame10-a pcisch: No fatal PCI bus error(s)
Aug 23 11:39:07 qame10-a scsi: WARNING: /ssm@0,0/pci@19,600000/SUNW,emlxs@1/fp@0,0/ssd@w2100000c509f1d80,0 (ssd59):
Aug 23 11:39:07 qame10-a        offline
Aug 23 11:39:13 qame10-a scsi: WARNING: /ssm@0,0/pci@18,700000/SUNW,emlxs@2/fp@0,0/ssd@w500000e010763ff1,0 (ssd24):
Aug 23 11:39:13 qame10-a        offline
Aug 23 11:39:13 qame10-a scsi: WARNING: /ssm@0,0/pci@18,700000/SUNW,emlxs@2/fp@0,0/ssd@w500000e010763ff1,0 (ssd24):

Root Cause

The current PCI nexus driver is incompatible in a configuration where the Emulex LP10000 HBA runs at 33MHz.


Workaround

Avoid installing an Emulex LP10000 HBA in:

  • 33MHz slots (Slot 0 and Slot 1 of PCI-X IO Assy)
  • Slots where the PCI bus is shared by another slot that has a 33MHz HBA installed

Resolution

For Solaris 10, install Kernel Update 118833-23 (or later) to fix this issue.
For Solaris 8 and Solaris 9, the fix is not available yet, but will be included in a future release of the "pcipsy" driver patch for Solaris 8 and Solaris 9.


Previously Published As
102578
Internal Comments


If a 33Mhz PCI card is installed in a slot that is 33Mhz/66Mhz/100Mhz compatible, and if an Emulex LP10000 PCI-X card is installed in a slot that shares the same PCI bus as the 33Mhz card, then the LP10000 card is also forced to run at 33Mhz.



Refer to the following Infodoc for a detailed view of the PCI IO slot layout for various IO Assemblies:




  • Infodoc 86460 for IO layout on Midrange systems



In any case, it is not recommended to install a 100Mhz PCI-X HBA in a 33Mhz slot or install in a slot that shares the PCI bus with another 33Mhz slot for obvious performance reasons.


Related Information
  • Other: Info Doc 86460

Internal Contributor/submitter
Nikhil.Suvarna@sun.com

Internal Eng Business Unit Group
SSG ES (Enterprise Systems)

Internal Eng Responsible Engineer
Anthony.Flores@sun.com

Internal Services Knowledge Engineer
Sean.Hassall@sun.com

Internal Resolution Patches
118833-23

Internal Kasp FAB Legacy ID
102578

Internal Sun Alert & FAB Admin Info
Critical Category:
Significant Change Date: 2006-09-27
Avoidance: Patch, Workaround
Responsible Manager: Harry.Huynh@sun.com
Original Admin Info: null

Product_uuid
29d3a694-0a18-11d6-92da-df959df44cdd|Sun Fire 4800 Server
29da7938-0a18-11d6-8a41-9ed1ad6d6779|Sun Fire 6800 Server
4fe39727-0599-11d8-84cb-080020a9ed93|Sun Fire E6900 Server
63ff680d-0598-11d8-84cb-080020a9ed93|Sun Fire E2900 Server
6a74b2f9-bbd8-4b2c-870d-b6b73d6e224f|Sun Fire V1280 Server
bed24aa9-0598-11d8-84cb-080020a9ed93|Sun Fire E4900 Server

Attachments
This solution has no attachment
  Copyright © 2011 Sun Microsystems, Inc.  All rights reserved.
 Feedback