Sun Storage 6320 System
 
Sun Storage 6120 Array
 

GCS>Sun Microsystems>Sun Alert>Criteria Category>Availability
 
GCS>Sun Microsystems>Sun Alert>Release Phase>Resolved
 

The use of certain sscs(1M) commands, GUI actions or commands run from telnet(1) to manage a Sun StorEdge 6120/6320 Array attached via McData Fibre Channel (FC) switches (listed below), and with Host Bus Adapters (HBA) using the Sun QLC HBA driver (listed below), may cause loss of connectivity to a host(s). As a result, the use of these commands may cause multiple path failures, which could lead to a complete loss of host access to the array.

This issue can occur on the following platforms:

• Sun StorEdge 6120/6320 Arrays without firmware 3.2.7

when connected to the following Fibre Channel switch models:

• McData 4300/4500/6064/6140 switches running E/OS 06.02.00 or earlier

and the following Host Bus Adapters (HBA):

• PCI Dual FC Network Adapter+ - Option 6757
• PCI Single FC Host Adapter - Option 6799
• PCI Dual FC Network Adapter+ - Option 6727
• 2GB PCI Single FC Network Adapter - Option SG-XPCI1FC-QF2
• 2GB PCI Dual FC Network Adapter - Option SG-XPCI2FC-QF2
• Compact PCI (cPCI) Dual Channel FC - Option 6748

The described issue may occur in the configurations described above when the following sscs(1M) commands, or array/StorEdge 3900SL CLI commands are issued:

sscs(1M) commands:

• sscs modify volgroup
• sscs create volume
• sscs create initiator
• sscs create pool
• sscs modify array
• sscs add initgroup

StorEdge 6120/T3+ telnet(1) commands:

• lun perm
• hwwn
• volslice
• vol mount
• sys mp_support

StorEdge 3900SL Service Processor (SP) CLI commands: (the following menu options in the program "/opt/SUNWsecfg/runsecfg"):

• 3) Configure Sun StorEdge T3+ Array(s)
• 6) Modify Sun StorEdge T3+ Array Sys Parameters
• 8) Manage Sun StorEdge T3+ Array LUN Slicing
• 9) Manage Sun StorEdge T3+ Array LUN Masking

The following commands from the directory "/opt/SUNWsecfg/bin" on the Service Processor (SP):

• createt3group
• addtot3group
• delfromt3group
• rmt3group
• createt3slice
• rmt3slice
• modifyt3config
• savet3config
• modifyt3params
• sett3lunperm

Note: Equivalent StorEdge 3900SL/6320 GUI actions to these commands may also cause this issue to occur.

The following "read-only" commands will NOT trigger the described issue:

• lun perm list
• hwwn list
• hwwn listgrp
• volslice list

sscs(1M) "read-only" commands:

• sscs create volume
• sscs remove volume

StorEdge 6120 "read-only" telnet(1) commands:

• volslice
• lun perm
• hwwn

To determine if a system uses the Sun QLC HBA driver for its connection to an array, do the following:

Run the format(1M) command as the "root" user, and choose a LUN on each controller path for each 6120/6320 array from the output list:

# format
Searching for disks...done
AVAILABLE DISK SELECTIONS:
0. c0t0d0 <SUN18G cyl 7506 alt 2 hd 19 sec 248>
	  /ssm@0,0/pci@1a,700000/pci@2/SUNW,isptwo@4/sd@0,0
1. c0t6d0 <SUN18G cyl 7506 alt 2 hd 19 sec 248>
	  /ssm@0,0/pci@1a,700000/pci@2/SUNW,isptwo@4/sd@6,0
--> 2. c3t60003BA27CC6B00040C472BF000262B0d0 <SUN-T4-0301 cyl 11705 alt 2 hd 7 sec 128>
	  /scsi_vhci/ssd@g60003ba27cc6b00040c472bf000262b0...
	  [lines omitted] ...
 Specify disk (enter its number): ^D
#

In the above example, we see an STMS device path for a StorEdge 6120 device on "c3", so we need to know which HBA(s) are used to access that STMS device. This is done by using the "luxadm display" command.

The following steps need to be repeated for each different controller number on a system for any controller numbers which are used to connect to StorEdge 6120/6320 arrays:

1) As "root" user, run "luxadm display" on the STMS device path, in order to see the physical HBA(s) which are used to access that STMS device. This uses the STMS device name from format(1M) (in this example, that device name is c3t60003BA27CC6B00040C472BF000262B0d0) prepended with "/dev/rdsk" and adding the suffix "s2" (assuming the Solaris standard label with a slice 2 has been used on that device; if this is not the case in your environment, then add a slice number which is used on the STMS device):

# luxadm display /dev/rdsk/c3t60003BA27CC6B00040C472BF000262B0d0s2
DEVICE PROPERTIES for disk:
/dev/rdsk/c3t60003BA27CC6B00040C472BF000262B0d0s2
Vendor:               SUN
Product ID:           T4
Revision:             0301
Serial Num:           Unsupported
Unformatted capacity: 5121.812 MBytes
Write Cache:          Enabled
Read Cache:           Enabled
Minimum prefetch:   0x0
Maximum prefetch:   0x0
Device Type:          Disk device
Path(s):
/dev/rdsk/c3t60003BA27CC6B00040C472BF000262B0d0s2
/devices/scsi_vhci/ssd@g60003ba27cc6b00040c472bf000262b0:c,raw
-->   Controller
/devices/ssm@0,0/pci@1a,700000/SUNW,qlc@1/fp@0,0
Device Address              20030003ba27cc6b,6
Host controller port WWN    210000e08b0aadac
Class                       primary
State                       ONLINE
-->   Controller
/devices/ssm@0,0/pci@1a,700000/SUNW,qlc@1,1/fp@0,0
Device Address              20030003ba27cc63,6
Host controller port WWN    210100e08b2aadac
Class                       secondary
State                       STANDBY
#

2) From the above output of the "luxadm display" command, note the 2 physical HBA paths used for connecting to the StorEdge 6120 LUN:

/devices/ssm@0,0/pci@1a,700000/SUNW,qlc@1/fp@0,0
/devices/ssm@0,0/pci@1a,700000/SUNW,qlc@1,1/fp@0,0

Note how these device paths contain the string "SUNW,qlc". This means that these HBAs will be using the Sun QLC driver.

Should the described issue occur, "lun failover" messages and host messages from STMS reporting that LUNs are being offlined, and that the paths allowing access to those LUNs are now degraded due to the loss of one path, will be displayed in the array syslog, similar to the following:

    [date time hostname] scsi: [ID 243001 kern.info]
/ssm@0,0/pci@1a,700000/SUNW,qlc@1,1/fp@0,0 (fcp1):
[date time hostname] offlining lun=1f (trace=0), target=90100
(trace=2800004)
...
[date time hostname] Initiating failover for device ssd (GUID
60003ba27cc6b00040c473c3000525ab)
[date time hostname] mpxio: [ID 669396 kern.info]
/scsi_vhci/ssd@g60003ba27cc6b00040c473d600076246 (ssd0) multipath status
: degraded, path /ssm@0,0/pci@1a,700000/SUNW,qlc@1,1/fp@0,0 (fp1) to
target address: 20030003ba27cc63,1f is offline. Load balancing:
round-robin

Note: The above are examples only. On each system, the LUN numbers, target numbers and device paths will vary. To identify that this issue is being seen, check the target trace value ("trace=2800004" above) and the overall sequence of events, where many LUNs failover, and a path is reported to be "offline", after performing any of the commands shown in section 2.

There are two methods available to clear this condition. The first is to offline and online the affected port on the McData switch. Offlining/onlining the switch port can be done through the CLI or through the GUI. The second is to reset the SE6120 (which is not recommended since it takes a long time to reboot the array). The following describes the steps for offlining/onlining the switch port, first using the CLI and next using the GUI:

I. Port Online/Offline Using CLI:

1. Telnet to the switch (default login/passwd is: Administrator/password)
2. execute 'config port blocked <port#> true
3. execute 'config port blocked <port#> false

II. Port Online/Offline Using GUI:

1. Start the EFCM GUI (default login/passwd is: Administrator/password) or SANpilot.
2. Double click on the icon for the subject switch
3. Double click on the UPM card that controls the port to offline/online
4. Right click on the port and select Block Port (the box to the left should not be checked). Then select OK. Note the port LED turning off.
5. Right click on the port and select Block Port (the box to the left should be checked now). Then select OK. Note the port LED turning on.

Once the above steps are completed and the proper status is displayed on the switch, you should be able to execute the luxadm display command for each affected LUN, and see that the Primary Paths are in a state of "ONLINE" and the secondary paths are in a state of "STANDBY".

This issue is addressed on the following platforms:

• Sun StorEdge 6120/6320 Arrays with firmware 3.2.7 (as delivered in patch 116931-23 or later)

with:

• McData 4300/4500/6064/6140 switches running E/OS 8.02.00 (as delivered in patch 119554-01 or later)

• Updated Contributing Factors and Resolution sections
• State: Resolved

References

The following bugs are related to the issue and are driving changes in different areas:

Bug 5056021: 6120 array is doing LOGO with 000000 S_ID when hwwn is added .....

Bug 5089209: Host does not see luns after SE6210 volslice create/remove W/ McData Switch

Bug 6179794: introduce delay to completing hwwn/volslice/lun perm commands to avoid high frequency of link events

Bug 5072558: qlc responds with FLOGI after receiving LOGO

Sun Alert 57609: The Use of Certain sscs(1M) Commands, Array/StorEdge 3900SL CLI Commands, or Certain StorEdge 3900SL/6320 GUI Actions to Manage a Sun StorEdge 3900SL/6120/6320/T3+ Array, Attached via Certain Brocade Switches, May Cause Loss of Connectivity to a Host(s)

Bug 5109873: Host does not see luns after SE6210 volslice create/remove W/ McData Switch

Bug 5068068: Host offlines SE6x20 target when multiple hwwns are added on the target side

Info Doc 78347: Host does not see LUNs after SE6120(T4) volslice create/remove with McData Switch

Here is a summary of the problem from infodoc 78347:

"Host loses access to LUNs after a volslice create or remove is executed on the 6120 array. The configuration in which this issue shows up is when one zone is created on the McData switch with two hosts and both of the SE6120 host ports are connected to the same zone. When a volslice create/remove is executed, the frontend ISP2300 interface initiates and completes the NOS/OLS/LR/LRR handshake with the switch, then the SE6120 sends an FLOGI and before the switch sends the ACC, it sends RSCNs to the registered hosts indicating the SE6120 state change. As a result of receiving the RSCNs, the host sends the PLOGI to the SE6120, this is not acknowledged because it did not receive the ACC for the FLOGI. This condition seems to confuse the the HBA (which happens to Qlogic) and causes the host not to see the LUNs when a format is executed.

Note: This configuration has been tested without any issues on the Brocade switch. Also, the above described issue shows up with McData E/OS firmware revs 4.02.00 to 6.01.00 on all switch modules (4300, 4500, 6064, 6140)."

This is a summary of the issues from McData:

- storage port sending frames with bad S-ID

After the port bounces from a config change there is a 5 millisecond window between when the port goes offline and sends out a FLOGI. The trace shows that the storage receives a Plogi from the HBA's and sends an LS-ACC back with a source ID of 00-00-00 which is not routable. THe storage port should wait for the Flogi accept for its address before sending the LS_ACC.

- hba not properly recovering the sequence after the storage ports bounce

on the trace showing the failure, the storage port and the switch get synced up and stable, the HBA PLOGI's the array for the 2nd time (the first time fails due to the frames come in prior to the flogi acc and the storage port acc's with a bad S-ID), then PRLI's but never does a Report Luns or anything beyond the PRLI. Also, the HBA never ABTS the original Plogis sent before the flogi Acc.

- the switch is hitting a timing issue between the route tables being updated and the name server servicing requests so frames are routed to the storage port before the Flogi acc goes out (this is not against the spec). Also, the switch drops the Flogi from the second storage port due to using the software port state - PR#45428 - fixed in E/OS 7.0.

References

This solution has no attachment