Autonegotiation does not work between Sun[TM] Gigabit Ethernet (GBE) cards and Cisco 4003 fiber linecards

Asset ID:	1-72-1003118.1
Update Date:	2011-05-26
Keywords:

Solution Type Problem Resolution Sure

Solution 1003118.1 : Autonegotiation does not work between Sun[TM] Gigabit Ethernet (GBE) cards and Cisco 4003 fiber linecards

Applies to:

Sun PCI Express Dual GbE MMF-UTP ExpModule - Version: Not Applicable and later   [Release: N/A and later ]
Sun Fire V1280 Server - Version: Not Applicable and later    [Release: N/A and later]
Sun Fire V880 Server - Version: Not Applicable and later    [Release: N/A and later]
Sun Fire V890 Server - Version: Not Applicable and later    [Release: N/A and later]
Sun Gigabit Ethernet SBus Adapter - Version: Not Applicable and later    [Release: N/A and later]
Oracle Solaris on x86 (32-bit)
Oracle Solaris on SPARC (64-bit)
Oracle Solaris on x86-64 (64-bit)
Oracle Solaris on SPARC (32-bit)
All Platforms

Symptoms

Autonegotiation does not work reliably with Cisco 4003 fiber linecards.

This affects Cisco Catalyst 4000 fiber linecards WS-X4418 and WS-X4424-SX-MT. The issue is described in Sun Bug ID 4733516 and Cisco Bug IDs CSCdm38405 and CSCdz02928 . Cisco Bug Toolkit access is needed to query these IDs.

Reported symptoms :

After boot, Sun[TM] Gigabit Ethernet network cards lose link to Cisco switch.. After unplug and replug of the network cable, it works again.
Ports 0 and 1, 4 and 5, etc. show Link Up with enabled Autonegotiation, but Link Down when the Sun system reboots, re-initializing (unplug/re-plug) helps. Ports 2 and 3, 6 and 7, etc show Link Down with enabled Autonegotiation, re-initializing does not help.
The Uplink port (Port 1) always negotiates successfully and port 2 negotiates fine most of the time. The other ports (3-18) fail. For testing, this can be reproduced by connecting the failing ports (3-18) directly (e.g 4 connected with 5, Spanning Tree disabled).

One or all of the above symptoms may be seen.

The failure is believed to be due to Cisco failing to restart the autonegotiation state machine when the link is lost. Hence, autonegotiation works once only. Removing the laser light seems to cause a more substantial reset of the Cisco's port. Turning autonegotiation off and on again at the Cisco has also been seen to recover the link; this indicates that a software solution on the Cisco may be possible.

Cause

The mentioned Cisco linecards do not work reliably in Autonegotiation mode. Cisco confirmed in Cisco case D393465 that this is a Cisco issue, which only Cisco Engineering could fix.
The related Cisco bugs have the status "Closed" (In Cisco's terms this is equivalent to Sun's "will not fix". Fixed Bugs have the status "Resolved").

Solution

Workaround: Disable autonegotiation on both the Cisco switch and the Sun Gigabit Ethernet card and force them to 1000 Mbits/full duplex.

For the Solaris system :

The commands for the ge card (Gigabit Ethernet 2.x) are:

ndd -set /dev/ge instance 0  (settings for ge0)
ndd -set /dev/ge adv_1000hdx_cap 0 (no half-duplex)
ndd -set /dev/ge adv_pauseTX 1 (will send pause frames)
ndd -set /dev/ge adv_pauseRX 1 (will receive pause frames)
ndd -set /dev/ge adv_1000autoneg_cap 0 (turn auto-negotiation off)

The commands for the ce fiber card (Sun GigaSwift 1.0) are:

ndd -set /dev/ce instance 0  (settings for ce0)
ndd -set /dev/ce adv_1000hdx_cap 0 (no half-duplex)
ndd -set /dev/ce adv_pause_cap 1
ndd -set /dev/ce adv_asmpause_cap 0 (send/receive pause frames)
ndd -set /dev/ce adv_autoneg_cap 0 (turn auto-negotiation off)

Note: The commands above are independent of the version of Solaris[TM] Operating System running on the Sun machine.

The above is the workaround suggested by Cisco. As ndd is not the recommendation for static configuration changes to interfaces using the ce driver it is recommended that <Document: 1018207.1> Correct Use of the Driver Configuration File to Force the Sun[TM] GigaSwift Ethernet Adapter's Operational Parameters is used to make permanent changes. Also because adv_pause_cap and adv_asmpause_cap are not related to link establishment it is not believed to be necessary to change these two parameters to work around this issue.

Check the interface User's Guide or the Sun Server Networking page for detailed information.

On the Catalyst 4000 the commands are:

set port negotiation  disable (turn auto-negotiation off)
set port flowcontrol  receive desired (default value)
set port flowcontrol  send on (default value)

For newer switch revisions, an additional command is reported to be needed:

 set speed nonegotiate

Switch info (note the + sign at WS-X4013) :
1 2 1000BaseX (GBIC) Supervisor(active) WS-X4013+ JAE080413C2
2 18 1000BaseX (GBIC) WS-X4418-GB JAE07460BFY
6 48 10/100/1000BaseTX (RJ45) WS-X4448-GB-RJ45 JAB08020AY2
IOS (tm) Catalyst 4000 L3 Switch Software (cat4000-I9S-M), Version 12.1(20)EW1,
EARLY DEPLOYMENT RELEASE SOFTWARE (fc1)

See the Cisco web page for detailed info or Check Cisco User's Guide

Internal Comments
Advise customers to prove that ports 3-18 do not negotiate well among themselves to show it is not a Sun issue and strongly suggest to file a Cisco P1 bug to motivate Cisco to fix their autonegotiation bug(s).

For corrections/questions: mailto:martin.lorenz@oracle.com
Product Page Reference:
http://panacea.uk.oracle.com/twiki/bin/view/Netprod/Ge
http://panacea.uk.oracle.com/twiki/bin/view/Netprod/Ce

Sun Case 36712291 EscID 542305
Sun Case 37146025

Cisco, boot, networking, link, switch, down, force, plug, jumpstart

Previously Published As
41728 and 204273

References

<NOTE:1018207.1> - Correct Use of the Driver Configuration File to Force the Sun[TM] GigaSwift Ethernet Adapter's Operational Parameters

Attachments

This solution has no attachment