---------------------------------------------------------------------
- Sun Proprietary/Confidential: Internal Use Only -
---------------------------------------------------------------------  
                            FIELD INFORMATION NOTICE
                  (For Authorized Distribution by SunService)

FIN #: I0665-1

Create Date: Apr/18/01

Keywords: 

PCI Bus configurations on Ultra 80 system based boards

Top FIN/FCO Report: No

Products Reference: PCI card configuration

Product Category: Server / Documentation

Product Affected: 

Mkt_ID   Platform   Model   Description   Serial Number
------   --------   -----   -----------   -------------
Systems Affected
----------------

  -      A27        ALL     Ultra 80            -
  -      A33        ALL     Enterprise 420R     -
  -      N14        ALL     Netra t 1405        -
  -      N15        ALL     Netra t 1400        -


X-Options Affected
------------------

X1034      -         -      Quad Fast Ethernet card   -
X1141      -         -      GBE/P Ethernet card       -

Parts Affected: 

Part Number   Description                    Model
-----------   -----------                    -----
501-4366      ASSY PCI CARD TESTED             -
501-5406      ASSY PCI CARD W/LUCENT TESTED    -
501-4373      ASSY GEM-PCI CARD MMF TESTED     -

References: 

BugID:  4424939 - U80 panics during Psycho reset. 
        4393940 - Netra T 1405/U80 panics intermittently.
        4342026 - Psycho intermittently violates PCI spec for signaling 
                  64-bit bus @ end of reset.
        4339479 - U80 getting panic on PCI bus 3 error(s).

ESC:    528472

DOC:    805-5938: GBE/P Product Note.
 
MANUAL: 805-2785: GBE/P Installation and User Guide.
        805-1797: QFE Installation and User Guide.
        806-0576: Netra 140x Service and Reference Manual.
        805-6618: Ultra 80 Service Manual.
        806-1080: E420R Service Manual.

Issue Description: 

Incorrect slot positioning of the GBE/P related to the QFE board can
cause panics on Ultra 80 systems.  Mixture of a QFE and a GBE/P on the
same bus, would cause the system may suffer a SERR error and panic the
system. 

Ultra 80, E420R, and Netra t 1400/1405 systems may panic at boot time
and drop to the "ok" prompt with certain PCI card configurations.  The
problem occurs when a QFE card and another 64-bit capable card such as
GBE/P Ethernet card are installed in adjacent 64-bit slots in the 33MHz
PCI bus of a U80 or related platform.

Typical failure:
----------------
 
     Solaris 5.7 Version Generic_106541-04,

     panic[cpu1]/thread=2a100003d60:
     pci0: PCI SERRpci0: PBM detected parity error.

     Fast Data Access MMU Miss
     ok 

Failing Configuration:
----------------------

      ---------------------------------------------------- 
     | Board location | slot |  bus | Mhz | mode  | Card  |
     |----------------------------------------------------|
     | J1801          |  1   |  A   | 66  | 64bit | GBE/P |
     | J2001          |  2   |  B   | 33  | 64bit | QFE   |
     | J1901          |  3   |  B   | 33  | 64bit | GBE/P |
     | J4701          |  4   |  B   | 33  | 32bit |   -   |
      ----------------------------------------------------

The probability of the problem occurring is very low.  However if it 
does occur, all service will be lost until the machine is rebooted.  
This failure has been reproduced on Solaris 2.6 and Solaris 7.  Due to 
improvements in Solaris 8, the fault is detected and the operating system 
panics but does not drop to the ok prompt.  It reboots successfully.

The system halts at the ok prompt since the OS did not successfully
panic, passing on handling of a fast data access MMU miss to OBP when
at TL=0x2.  OBP can only handle MMU misses from TL=0. MMU misses from
TL=0x2 should not have been triggered by Solaris.

The problem occurs due to a RESET timing bug in the Psycho chip.
During the RESET cycle, the PCI bus is sized, meaning any card
installed in that bus at RESET time will configure itself as either
being in a 64-bit slot (if REQ64 is asserted) or in a 32-bit slot (if
REQ64 is not asserted).  Psycho MAY de-assert REQ64 too early due to an
internal timing problem; when this occurs, the PCI cards may
incorrectly configure themselves as being installed in a 32-bit PCI
slot instead of a 64-bit slot.  In effect, the second PCI card (i.e.,
GBE) gets configured as a 32-bit card in this case.  

When the GBE card sees a 64-bit PCI transfer on the bus,
corresponding to the first QFE DMA data transfer after the driver is
loaded, GBE drives the upper 32 Address/Data lines to 0.  This can
happen even before GBE has been configured and the GBE driver gets
loaded.  This contends with the QFE data being sent to Psycho, and
causes an apparent parity error.  The parity error is caused by bus
contention from the second PCI card (GBE) and not from QFE itself (QFE
generated parity is, in fact, correct).

It is not possible to fix this problem without a respin to this ASIC
and will not be done. The only way to avoid this issue is through
correct PCI card placement.

Implementation: 

---
        |   |   MANDATORY (Fully Pro-Active)
         ---    
         
  
         ---
        |   |   CONTROLLED PRO-ACTIVE (per Sun Geo Plan) 
         --- 
         
                                
         ---
        | X |   REACTIVE (As Required)
         ---

Corrective Action: 

An Authorized Enterprise Field Service Representative may avoid the
above mentioned problems by following the recommendations as shown
below.

When installing this mixture of PCI QFE and GBE/P cards particular
attention must be made to slot assignment.  The slot number is the
physical position as printed on the printed circuit board and not the
Unix identification.

The following guideline should be adhered to: 

   1. Install QFE card in slot 1 only.  

   2. Install GBE/P cards in either slots 2 and 3.  Do not install
      GBE/P card in slot 1.

   3. Any 64-bit card can be used in a 32-bit slot (slot 4) provided 
      there is no mechanical interference from adjacent components 
      next to the slot.

The preferred slot configuration is shown in the following table:   

    ---------------------------------------------------- 
   | Board location | slot |  bus | Mhz | mode  | Card  |
   |====================================================|
   | J1801          |  1   |  A   | 66  | 64bit | QFE   |
   | J2001          |  2   |  B   | 33  | 64bit | GBE/P |
   | J1901          |  3   |  B   | 33  | 64bit | GBE/P |
   | J4701          |  4   |  B   | 33  | 32bit |   -   |
    ----------------------------------------------------

In addition to the physical PCI card positioning, system performance
tuning may be undertaken to obtain maximum throughput for this
configuration.  Typical setting are shown below but any changes are
dependent on the customers requirements.  Refer to chapter 3 of the
GBE/P Installation and User Guide (805-2785), and chapter 3 of the QFE
Installation and User Guide (805-1797) for detailed information about
setting device driver parameters.

     ndd -set /dev/ge instance 0
     ndd -set /dev/ge adv_pauseTX 1
     ndd -set /dev/ge adv_1000autoneg_cap 1
     ndd -set /dev/ge instance 2
     ndd -set /dev/ge adv_pauseTX 1
     ndd -set /dev/ge adv_1000autoneg_cap 1

     sq_max_size=500 (/etc/system),

Comments: 

-----------------------------------------------------------------------------

Implementation Footnote: 

i)   In case of MANDATORY FINs, Enterprise Services will attempt to    
     contact all affected customers to recommend implementation of 
     the FIN. 
   
ii)  For CONTROLLED PROACTIVE FINs, Enterprise Services mission critical    
     support teams will recommend implementation of the FIN  (to their  
     respective accounts), at the convenience of the customer. 

iii) For REACTIVE FINs, Enterprise Services will implement the FIN as the   
     need arises.
----------------------------------------------------------------------------
All released FINs and FCOs can be accessed using your favorite network 
browser as follows:
 
SunWeb Access:
-------------- 
* Access the top level URL of http://sdpsweb.ebay/FIN_FCO/

* From there, select the appropriate link to query or browse the FIN and
  FCO Homepage collections.
 
SunSolve Online Access:
-----------------------
* Access the SunSolve Online URL at http://sunsolve.Corp/

* From there, select the appropriate link to browse the FIN or FCO index.

Supporting Documents:
---------------------
* Supporting documents for FIN/FCOs can be found on Edist.  Edist can be 
  accessed internally at the following URL: http://edist.corp/.
  
* From there, follow the hyperlink path of "Enterprise Services Documenta- 
  tion" and click on "FIN & FCO attachments", then choose the appropriate   
  folder, FIN or FCO.  This will display supporting directories/files for 
  FINs or FCOs.
   
Internet Access:
----------------
* Access the top level URL of https://infoserver.Sun.COM
--------------------------------------------------------------------------
General:
--------
* Send questions or comments to finfco-manager@Sun.COM
---------------------------------------------------------------------------