ST9990V and ST9985V data may become inconsistent on SATA OPEN-V CVS volumes with SATA-E Mode where CVS Volumes were created from the SVP by specifying "Set remaining space as volume".

Asset ID:	1-73-1022240.1
Update Date:	2010-08-05
Keywords:

Solution Type FAB (standard) Sure

Solution 1022240.1 : ST9990V and ST9985V data may become inconsistent on SATA OPEN-V CVS volumes with SATA-E Mode where CVS Volumes were created from the SVP by specifying "Set remaining space as volume".

Impact

Data becomes inconsistent and should not be used. HDDs or parity groups become blocked.

Contributing Factors

This issue may occur when ALL the following conditions below are met;

1. Any of the Storage Subsystems listed in the above Product section are being used.

2. Any of the versions of microcode range listed below are being used.

     V06: 60-06-10-00/10-M151
     V06: 60-06-05-00/00-M143 - 60-06-10-00/00-M148
     V05: 60-05-10-00/00-M110** - 60-05-17-00/00-M150

   ** Note: 60-05-10-00/00-M110 was the first version supporting SATA-E.

3. The Protection Level: SATA-E attribute is set on the SATA parity group.
   This can only be set through the Service Processor(SVP) when the parity group is installed.

   Note: All SATA parity groups in a system shipping from Factory (since August, 2009)
         have SATA-E set on as a default.

   The following procedures from the SVP can cause this problem when they
   are used to set this configuration:

Loading a configuration file during New Machine Installation using Define Configuration & Install(DCI) <Never use DCI as a standalone procedure>
Make Volume (SVP only).
Importing a configuration file using the Offline SVP Operation of the offline Configuration tool.

4. Emulation type: OPEN-V.

5. In SVP Variable Volume Size Define window (during the process of parity group installation
   through SVP), when a SATA-E volume is configured as a Variable Size (CVS) volume and
   is created by selecting "Set remaining space as volume" in the Variable Volume Size Define
   dialog box.

   Please refer to the figure described in step 5 of Condition of Occurrence section at below
   HDS alert link;

     http://se9990.eng/tech_docs/fab/SATA_issue/USPV_USPVM_063502R4.pdf

   Note: There will not be a problem if it is created by specifying LDEV capacity (Specify size
        and number) dialog box, instead of selecting "Set remaining space as volume".

Symptoms

The following phenomena may occur on a volume where the SATA parity group has the SATA-E (Enhanced) attribute set.

Phenomenon 1: Data inconsistency may occur in response to a read I/O from the host.
Phenomenon 2: Errors may occur in response to a read IO from the host resulting in
              a SATA disk drive or parity group being blocked by the microcode.
Phenomenon 3: If a drive fails and must be spared out, the operation may fail and
              the drive or parity group may be blocked by the microcode.

Note: The volumes that are affected by this problem are SATA-E CVS volumes configured
      only via the SVP by specifying "Set remaining space as volume".

Root Cause

In order for SATA-E to function, it requires that a Volume Management Area (VMA) be created by the microcode at the end of the volume when the SATA-E parity group is initialized during creation. This microcode bug allows the host to write over this area by mistake. Note that the host may not yet have written enough data to overwrite this management area at the end of the volume.

Note that just loading fixed microcode does not correct this problem. See the Resolution section for the procedures that must be followed to detect the condition and also to correct it.

For more information on SATA-E, please refer to the following Maintenance Manual Sections starting on these pages...

ST9990V THEORY03-341 and THEORY03-3350
ST9985V THEORY03-341 and THEORY03-3480

For ST9900 Maintenance Manuals reference the below (internal only) URL;

http://pts-storage.west/products/T99x0/documentation.html

Corrective Action

Workaround:

When the Service Representative is setting up the SATA-E and is creating the Variable Size volume, do not use "Set remaining space as volume", but instead specify the LDEV capacity (Specify size and number).

Resolution:

1. Actions to take if the problem has occurred:

Under the direction of the backline support perform the following when the Conditions of Occurrence are met as mentioned in contributing factor section above:

a) If RAID Group is blocked (Phenomenon 2 or 3 mentioned in impact section)

   i) Perform the LDEV recover y (Normal Restore) to the blocked RAID Group.
   ii) Perform the LDEV blockade to the invalid LDEV.
       Note: If you do not block the invalid LDEV, and Correction Copy runs because
             of HDD replacement, the HDD (RAID Group) may be blocked again.
   iii) Recover the HDD by performing a dummy-replacement.
   iv) Delete the invalid LDEV by performing Volume to Space.
   v) Install LDEV again by performing LDEV Install (Install CV) and restore
      the data from the backup.

b) If only HDD is blocked (Phenomenon 2 or 3)

   i) Perform the LDEV blockade to the invalid LDEV.
   ii) Recover the HDD by performing a dummy-replacement.
   iii) Delete the invalid LDEV by performing Volume to Space.
   iv) Install LDEV again by performing LDEV Install (Install CV) and restore
       the data from the backup.

c) If HDD is not blocked (Phenomenon 1 or 2)

   i) Perform the LDEV blockade to the invalid LDEV.
   ii) Delete the invalid LDEV by performing Volume to Space.
   iii) Install LDEV again by performing LDEV Install (Install CV) and restore
        the data from the backup.

2. Actions to take if the above mentioned problem has NOT occurred:

Follow the below guidelines:

For HiTrack connected units you can check the below URL to review the SATA drive install base and Microcode levels (60-05-10-00/00 and above) as of March 1, 2010.

http://se9990.eng/tech_docs/fab/SATA_issue/ST9900SATA_Installbase.ods

For non HiTrack systems you will need to do a physical inspection of any units containing SATA drives.

I) Identification of the problem (invalid condition of ldev) :

Use one of the two methods, running a script or doing a manual check by following the details in "identification of the problem" in SOLUTION section via the below link;

http://se9990.eng/tech_docs/fab/SATA_issue/USPV_USPVM_063502R4.pdf

SATA check script with instructions to run the program is available at below link;

http://se9990.eng/tech_docs/fab/SATA_issue/satascript_063655.zip

Please download and run the program in SVP by following the instructions.

II) Replace the micro-program with the modified one listed below.

    For V06: 60-06-12-00/00-M152 or above
    For V05: 60-05-18-00/00-M154 or above

Note that even if the micro-program is replaced, the invalid condition of the LDEV is not removed. When the SATA-E attribute is defined, check if there is an invalid condition in the LDEV by using the method described in "Identification of The problem" above.

III) If it is determined that an invalid condition on an LDEV exists, perform the following operations:

   a. Delete the invalid LDEV by performing Volume to Space.
   b. After that, install the LDEV again by performing LDEV Install (Install CV)
      and restore the data from the backup.

Additional details:

1) SATA-E mode is a default microcode setting when configuring SATA parity groups starting with 60-05-10-00/00-M110. Therefore, SATA-E has been shipping enabled from the IDC/EDC distribution centers for all SATA parity groups where SATA-E is supported in the microcode. Note that the distribution centers do not configure any CVS volumes as part of their build process. The volumes that are affected by this problem are SATA-E CVS volumes configured only via the SVP by a service representative at the customer site by specifying "Set remaining space as volume".

2) SATA Parity Group Setting on machines shipped from Distribution Centers:

As of February 24, 2010 the Distribution Centers no longer ship SATA parity groups configured with SATA-E Mode as a default.

SATA--W/V: 24-February-2010 (60-06-10-00/00-m148) < All current and future shipments >
SATA-E : 07-August 2009 (60-05-10-00/00-M110) < through > 24-February-2010 (60-06-10-00/00-m148)
SATA-W/V: 06-December 2007 (60-02-05-00/00-M018) < through > 07-August 2009 (60-04-15-00/00-M105)

Comments

Although this FAB is being published with a Controlled Proactive implementation, no Customer List will be provided in SunFIT and there is no requirement to track implementation. However, a SATA Drive Installed Base report can be found via an (internal only) link under step 2 in the Resolution section above.

Subscribe to ST9900 up-to-date Alerts by refering the below link:

http://sejsc.ebay/alerts_via_alias.html

For ST9900 Maintenance Manuals go to;

http://pts-storage.west/products/T99x0/documentation.html

For information about FAB documents, its release processes, implementation strategies and billing information, go to the following URL:

http://tns.central/fab

For Sun Authorized Service Providers go to:

http://tns.central/SPE/fab/

In addition to the above you may email:

FAB-Manager@sun.com

Internal Contributor/submitter
Suresh.Gummanur@Sun.COM

Internal Eng Responsible Engineer
Suresh.Gummanur@Sun.COM Responsible Manager: Tejinder.Singh@Sun.COM

Internal Services Knowledge Engineer
Joe.Davis@Sun.COM

Internal Sun Alert & FAB Admin Info
03-Mar-2010: Completed draft and sent to Extended Review.
05-Mar-2010: On-hold awaiting final updates from submitter.
08-Mar-2010: Received final updates from submitter - sending to Publish.

Attachments

This solution has no attachment