Too many FC InvTxWord can cause SCSI Command Timeout and/or poor I/O performance

Asset ID:	1-72-1005122.1
Update Date:	2010-10-29
Keywords:

Solution Type Problem Resolution Sure

Solution 1005122.1 : Too many FC InvTxWord can cause SCSI Command Timeout and/or poor I/O performance

Related Items


Sun Storage 6320 System
 Sun Storage 6020 Array
 Sun Storage 6120 Array
 Sun Storage 6920 System

Related Categories


GCS>Sun Microsystems>Storage - Disk>Modular Disk - 6xxx Arrays

PreviouslyPublishedAs
207198

Symptoms
An FC link generating too many InvTxWord errors (for example, a few thousand per 5 minutes) can result in Host SCSI Command Timeout, and/or may result in poor I/O performance.

Resolution
InvTxWord indicates that the FC Transmitter sent a word which was not recognized by the receiver.

These InvTxWord errors do not represent any failure on their own. However, they indicate that some active component like FC port, GBIC or SFP in the FC Path is generating bad FC words.
It is normal to observe a few InvTxWord during boot-up when the speed is being negotiated. However, if too many InvTxWord are continuously being generated during I/O operation (such as 5000 InvTxWord in 5 mins), this has the potential of affecting I/O performance, and the host may see SCSI Command Timeouts.
---------
scsi: [ID 243001 kern.warning] WARNING: /scsi_vhci/ssd@g60003ba4e84460004109bdbe0004c876 (ssd10):
SCSI transport failed: reason 'timeout': retrying command
---------
In configurations using StorADE, InvTxWord is reported as follows:
---------
6120. Alarm loop.statistics
Internal Loop statistics changed on 'ctrl1' of
6120 sp0-array02 (ip=192.168.0.42):
'InvTxWord' on unit0-ctrl1-loop2 went from
'23063' to '31523'
---------
In the above StorADE event log, the 'unit0-ctrl1-loop2' represents 'host port' of 'controller 1' in 'unit 0' 6120, which is reporting the InvTxWord.
This indicates a problem in the FC connection between the 6120 host port and the switch port.
If StorADE is not installed, status InvTxWord can be cross-checked using the following commands:
On the T3B/6020/6120 (FW 3.x.x and above), the following command can be used repeatedly to check the FC link statistics:
linkstat u1ctr 2
linkstat u2ctr 2
Where
u1ctr : U1 controller
2     : Host port
u2ctr : U2 controller
On the host:
Make sure that I/O to Target/LUNs in question is active using command `iostat -xnd 2`. Run the following command repeatedly and observe if there is any increase in the number of InvTxWord being generated:
luxadm -e rdls /devices/<HBA>:devctl
From the above outputs, calculate the number of invalid transmission word generated over a span of 5 mins [from the "invalid word/INVTXWORD" column].
If there are around 5000 InvTxWord in 5mins, this has the potential of causing SCSI timeouts. However, a few thousand InvTxWord in 5mins can also affect I/O performance silently, without generating SCSI timeouts or any FC error messages in /var/adm/messages or in the 6120 syslog. In this case, the FC port, GBIC or SFP along the I/O path should be checked.

Product
Storage Automated Diagnostic Environment 2.1
Storage Automated Diagnostic Environment 2.2
Storage Automated Diagnostic Environment 2.4
Sun StorageTek 6320 System
Sun StorageTek 6120 Array
Sun StorageTek 6020 Array
Sun StorageTek 6920 System

Internal Comments
This document contains normalized content and is managed by the the Domain Lead(s) of the respective domains. To notify content owners of a knowledge gap contained in this document, and/or prior to updating this document, please contact the domain engineers that are managing this document via the “Document Feedback” alias(es) listed below:

storage-os-disk-mid-domain@sun.com

The Knowledge Work Queue for this article is KNO-STO-MIDRANGE_DISK
InvTxWord, non-actionable, 6120, 6320, 6920, storade, loop.statistics, ctrl1-loop2, ctrl2-loop2, audited
Previously Published As
82994

Change History
Date: 2007-08-14
User Name: 71396
Action: Approved
Comment: Performed final review of article.
No changes required.
Publishing.
Version: 8
Date: 2007-08-13
User Name: 71396
Action: Accept
Comment:
Version: 0
Date: 2007-08-10
User Name: 26402
Action: Approved
Comment: Made several changes and simplifications to improve readability.

Attachments

This solution has no attachment