Document Audience:INTERNAL
Document ID:I0854-3
Title:Invalid Device structure found after disk removal when under Volume Manager 3.2 Control.
Copyright Notice:Copyright © 2005 Sun Microsystems, Inc. All Rights Reserved
Update Date:2004-07-27

---------------------------------------------------------
            - Sun Proprietary/Confidential: Internal Use Only -
---------------------------------------------------------------------  
                        FIELD INFORMATION NOTICE
               (For Authorized Distribution by SunService)
FIN #: I0854-3
Synopsis: Invalid Device structure found after disk removal when under Volume Manager 3.2 Control.
Create Date: Jul/27/04
SunAlert: No
Top FIN/FCO Report: Yes
Products Reference: Sun Fire V880/V480, E3500 Servers, A5x00 StorEdge Array
Product Category: Server / SW Admin
Product Affected: 
Systems Affected:
-----------------  
Mkt_ID   Platform   Model   Description             Serial Number
------   --------   -----   -----------             -------------
  -        A30       ALL    Sun Fire V880                 -
  -        A37       ALL    Sun Fire V480                 -
  -        E3500     ALL    Ultra Enterprise 3500         -


X-Options Affected:
-------------------
Mkt_ID   Platform   Model   Description             Serial Number
------   --------   -----   -----------             -------------
  -       A5x00      ALL    A5x00 Storage Array           -
Parts Affected: 
Part Number   Description           Model
-----------   -----------           -----
     -             -                  -
References: 
BugId:   4630477 - bogus device in "vxdisk list" output after replacing 
                   a disk drive.

PatchId: 113201-03: VxVM 3.2: general patch for Solaris 2.6; 7; and 8.
         112385-05: VxVM 3.2s9: general patch for Solaris 9.
                
ESC:     534731- bogus device in `vxdisk list` output after replacing a 
                 disk drive.

URL:     http://sdn.sfbay/cgi-bin/escweb?-I024271?-M534731?-P1
Issue Description: 
--------------------------------------------------------------------------
| Change History from FIN I0854-2 to FIN I0854-3                           |
| ==============================================                           |
|    Date Modified: July 27, 2004                                          |
|                                                                          |
|      Updated Sections: Corrective Action                                 |
|                                                                          |
|      CORRECTIVE ACTION: Eliminated temporary workaround fix and replaced |
|                         with permanent patch fix.                        |
|                                                                          |
|                                                                          |
| Change History from I0854-1 on Dec 17, 2002                              |
| ===========================================                              |
|    Date Modified: Nov/24/2003                                            |
|                                                                          |
|    Updates:  CORRECTIVE ACTION:                                          |
|                                                                          |
|    CORRECTIVE ACTION: . Added a procedure for 'disk replacement for Root |
|                         Disk', and 'V880 disk replacement'.              |
|                                                                          |
|                       . Added a procedure for Manual replacement on      |
|                         Volume Manager Disk w/CLI on 'Mirrored boot disk'|
|                         and 'Standard mirrored data disk'.               |
  -------------------------------------------------------------------------

After removing an FCAL  disk under VM 3.2 control from the internal
disk sub-system of a V880, V480, E3500, or from an A5X00, an 'Invalid
device structure' error message can be seen.  This causes the
replacement of the disks to fail.  When this happens, a reboot is
needed to allow replacement of the disk to proceed.  Having to reboot
the system nullifies the advantage of disk hotswap, causing unnecessary
downtime.

The following is an example of removing a disk under control of VM3.2.
Invalid device can be seen that's left behind:

   # vxdiskadm

     Volume Manager Support Operations
     Menu: VolumeManager/Disk

       1      Add or initialize one or more disks
       2      Encapsulate one or more disks
       3      Remove a disk
       4      Remove a disk for replacement
       5      Replace a failed or removed disk
       6      Mirror volumes on a disk
       7      Move volumes from a disk
       8      Enable access to (import) a disk group
       9      Remove access to (deport) a disk group
      10      Enable (online) a disk device
      11      Disable (offline) a disk device
      12      Mark a disk as a spare for a disk group
      13      Turn off the spare flag on a disk
      14      Unrelocate subdisks back to a disk
      15      Exclude a disk from hot-relocation use
      16      Make a disk available for hot-relocation use
      17      Prevent multipathing/Suppress devices from VxVM's view
      18      Allow multipathing/Unsuppress devices from VxVM's view
      19      List currently suppressed/non-multipathed devices
      20      Change the disk naming scheme
      21      Get the newly connected/zoned disks in VxVM view
 
      list   List disk information

       ?      Display help about menu
      ??      Display help about the menuing system
       q      Exit from menus

      Select an operation to perform: 4

    Remove a disk for replacement
    Menu: VolumeManager/Disk/RemoveForReplace

Use this menu operation to remove a physical disk from a disk group,
while retaining the disk name.  This changes the state for the disk
name to a "removed" disk.  If there are any initialized disks that are
not part of a disk group, you will be given the option of using one of
these disks as a replacement.

    Enter disk name [,list,q,?] list

       Disk group: rootdg

       DM NAME         DEVICE       TYPE     PRIVLEN  PUBLEN   STATE

       dm disk01       c1t11d0s2    sliced   4711     35358848 -
       dm disk02       c1t13d0s2    sliced   4711     35358848 -

    Enter disk name [,list,q,?] disk01

    The following volumes will lose mirrors as a result of this operation:

       vol01

       No data on these volumes will be lost.

    The following devices are available as replacements:

       c1t2d0
        
    Choose one of these disks now, to replace disk01.
    Select "none" if you do not wish to select a replacement disk.

    Choose a device, or select "none"
    [,none,q,?] (default: c1t2d0) none

The requested operation is to remove disk disk01 from disk group
rootdg.  The disk name will be kept, along with any volumes using the
disk, allowing replacement of the disk.

    Select "Replace a failed or removed disk" from the main menu
    when you wish to replace the disk.

  Continue with operation? [y,n,q,?] (default: y) 

    Removal of disk disk01 completed successfully.

  Remove another disk? [y,n,q,?] (default: n) 
  
    Volume Manager Support Operations
    Menu: VolumeManager/Disk

      1      Add or initialize one or more disks
      2      Encapsulate one or more disks
      3      Remove a disk
      4      Remove a disk for replacement
      5      Replace a failed or removed disk
      6      Mirror volumes on a disk
      7      Move volumes from a disk
      8      Enable access to (import) a disk group
      9      Remove access to (deport) a disk group
     10      Enable (online) a disk device
     11      Disable (offline) a disk device
     12      Mark a disk as a spare for a disk group
     13      Turn off the spare flag on a disk
     14      Unrelocate subdisks back to a disk
     15      Exclude a disk from hot-relocation use
     16      Make a disk available for hot-relocation use
     17      Prevent multipathing/Suppress devices from VxVM's view
     18      Allow multipathing/Unsuppress devices from VxVM's view
     19      List currently suppressed/non-multipathed devices
     20      Change the disk naming scheme
     21      Get the newly connected/zoned disks in VxVM view
 
     list   List disk information

      ?      Display help about menu
     ??      Display help about the menuing system
      q      Exit from menus

   Select an operation to perform: q

   Goodbye.

     # luxadm remove_device /dev/rdsk/c1t11d0s2

WARNING!!! Please ensure that no filesystems are mounted on these device(s).
           All data on these devices should have been backed up.

The list of devices being used (either busy or reserved) by the host:
     1: Box Name:    "dak"  slot 9

Please enter 's' or  to Skip the "busy/reserved" device(s) or
     'q' to Quit and run the subcommand with
     -F (force) option. [Default: s]: 

   # luxadm remove_device -F /dev/rdsk/c1t11d0s2 --> Had to Force removal
     of device.

WARNING!!! Please ensure that no filesystems are mounted on these device(s).
           All data on these devices should have been backed up.

  The list of devices which will be removed is:
  
     1: Box Name:    "dak" slot 9
        Node WWN:    2000002037d9ff50
        Device Type:Disk device
        Device Paths:
        /dev/rdsk/c1t11d0s2

  Please verify the above list of devices and then enter 'c' or  to
  Continue or 'q' to Quit. [Default: c]:
 
    stopping:  Drive in "dak" slot 9....Done
    offlining: Drive in "dak" slot 9....Done

  Hit  after removing the device(s).
  Jun 6 08:51:52 eis-dak-f picld[233]:
 
  Device DISK9 removed
  Jun  6 08:51:52 eis-dak-f picld[233]: Device DISK9 removed

  Drive in Box Name "dak" slot 9
  Logical Nodes being removed under /dev/dsk/ and /dev/rdsk:
  Logical Nodes being removed under /dev/dsk/ and /dev/rdsk:
        c1t11d0s0
        c1t11d0s1
        c1t11d0s2
        c1t11d0s3
        c1t11d0s4
        c1t11d0s5
        c1t11d0s6
        c1t11d0s7

   # format
     Searching for disks...done

     AVAILABLE DISK SELECTIONS:

       0. c1t0d0 
          /pci@8,600000/SUNW,qlc@2/fp@0,0/ssd@w2100002037f3d422,0
       1. c1t1d0 
          /pci@8,600000/SUNW,qlc@2/fp@0,0/ssd@w2100002037bd2c91,0
       2. c1t2d0 
          /pci@8,600000/SUNW,qlc@2/fp@0,0/ssd@w2100002037d9ff6a,0
       3. c1t3d0 
          /pci@8,600000/SUNW,qlc@2/fp@0,0/ssd@w2100002037d9ff44,0
       4. c1t4d0 
          /pci@8,600000/SUNW,qlc@2/fp@0,0/ssd@w2100002037d9ff4c,0
       5. c1t5d0 
          /pci@8,600000/SUNW,qlc@2/fp@0,0/ssd@w2100002037e6057a,0
       6. c1t8d0 
          /pci@8,600000/SUNW,qlc@2/fp@0,0/ssd@w2100002037d9fd70,0
       7. c1t9d0 
          /pci@8,600000/SUNW,qlc@2/fp@0,0/ssd@w2100002037d9ff5d,0
       8. c1t10d0 
          /pci@8,600000/SUNW,qlc@2/fp@0,0/ssd@w2100002037d9ff56,0
       9. c1t11d0 
          /pci@8,600000/SUNW,qlc@2/fp@0,0/ssd@w2100002037d9ff50,0
      10. c1t12d0 
          /pci@8,600000/SUNW,qlc@2/fp@0,0/ssd@w2100002037bde2dc,0
      11. c1t13d0 
          /pci@8,600000/SUNW,qlc@2/fp@0,0/ssd@w2100002037d9ff46,0

    Specify disk (enter its number): 1
    selecting c1t1d0
    [disk formatted]

    FORMAT MENU:

        disk       - select a disk
        type       - select (define) a disk type
        partition  - select (define) a partition table
        current    - describe the current disk
        format     - format and analyze the disk
        repair     - repair a defective sector
        label      - write label to the disk
        analyze    - surface analysis
        defect     - defect list management
        backup     - search for backup labels
        verify     - read and display labels
        save       - save new disk/partition definitions
        inquiry    - show vendor, product and revision
        volname    - set 8-character volume name
        !     - execute , then return
        quit

  format> q

  # devfsadm -C

  # cd /dev/rdsk

  # ls -al c1t11*

    lrwxrwxrwx   1 root     root          74 May 22 08:10 c1t11d0s0 -> 
    ../../devices/pci@8,600000/SUNW,qlc@2/fp@0,0/ssd@w2100002037d9ff50,0:a,raw
    lrwxrwxrwx   1 root     root          74 May 22 08:10 c1t11d0s1 -> 
    ../../devices/pci@8,600000/SUNW,qlc@2/fp@0,0/ssd@w2100002037d9ff50,0:b,raw
    lrwxrwxrwx   1 root     root          74 May 22 08:10 c1t11d0s2 -> 
    ../../devices/pci@8,600000/SUNW,qlc@2/fp@0,0/ssd@w2100002037d9ff50,0:c,raw
    lrwxrwxrwx   1 root     root          74 May 22 08:10 c1t11d0s3 -> 
    ../../devices/pci@8,600000/SUNW,qlc@2/fp@0,0/ssd@w2100002037d9ff50,0:d,raw
    lrwxrwxrwx   1 root     root          74 May 22 08:10 c1t11d0s4 -> 
    ../../devices/pci@8,600000/SUNW,qlc@2/fp@0,0/ssd@w2100002037d9ff50,0:e,raw
    lrwxrwxrwx   1 root     root          74 May 22 08:10 c1t11d0s5 -> 
    ../../devices/pci@8,600000/SUNW,qlc@2/fp@0,0/ssd@w2100002037d9ff50,0:f,raw
    lrwxrwxrwx   1 root     root          74 May 22 08:10 c1t11d0s6 -> 
    ../../devices/pci@8,600000/SUNW,qlc@2/fp@0,0/ssd@w2100002037d9ff50,0:g,raw
    lrwxrwxrwx   1 root     root          74 May 22 08:10 c1t11d0s7 -> 
    ../../devices/pci@8,600000/SUNW,qlc@2/fp@0,0/ssd@w2100002037d9ff50,0:h,raw

  All above Devices should NOT exist.......

  # luxadm insert_device

  Please hit  when finished adding Fibre Channel Enclosure(s)/Device(s): 
     Jun  6 08:54:49 eis-dak-f picld[233]: Device DISK9 inserted
     Jun  6 08:54:49 eis-dak-f picld[233]: Device DISK9 inserted

  Waiting for Loop Initialization to complete...
  New Logical Nodes under /dev/dsk and /dev/rdsk :

        c1t11d0s0
        c1t11d0s1
        c1t11d0s2
        c1t11d0s3
        c1t11d0s4
        c1t11d0s5
        c1t11d0s6
        c1t11d0s7

 No new enclosure(s) were added!!

  # vxdctl enable
    Jun  6 08:56:37 eis-dak-f vxdmp: NOTICE: vxvm:vxdmp: enabled path 118/0x48 
         belonging to the dmpnode 239/0x10
    Jun  6 08:56:37 eis-dak-f vxdmp: NOTICE: vxvm:vxdmp: enabled path 118/0x48 
         belonging to the dmpnode 239/0x10
    Jun  6 08:56:37 eis-dak-f vxdmp: NOTICE: vxvm:vxdmp: enabled dmpnode 
239/0x10
    Jun  6 08:56:37 eis-dak-f vxdmp: NOTICE: vxvm:vxdmp: enabled dmpnode 
239/0x10
    Jun  6 08:56:41 eis-dak-f vxdmp: NOTICE: vxvm:vxdmp: disabled path 118/0x208 
         belonging to the dmpnode 239/0x8
    Jun  6 08:56:41 eis-dak-f vxdmp: NOTICE: vxvm:vxdmp: disabled path 118/0x208 
         belonging to the dmpnode 239/0x8
    Jun  6 08:56:41 eis-dak-f vxdmp: NOTICE: vxvm:vxdmp: disabled dmpnode 
239/0x8
    Jun  6 08:56:41 eis-dak-f vxdmp: NOTICE: vxvm:vxdmp: disabled dmpnode 
239/0x8

  # vxdisk list    

    DEVICE       TYPE      DISK         GROUP        STATUS

    c1t0d0s2     sliced    -            -            error
    c1t1d0s2     sliced    -            -            error
    c1t2d0s2     sliced    -            -            online
    c1t3d0s2     sliced    -            -            error
    c1t4d0s2     sliced    -            -            online
    c1t5d0s2     sliced    -            -            error
    c1t8d0s2     sliced    -            -            online
    c1t9d0s2     sliced    -            -            online
    c1t10d0s2    sliced    -            -            online
    c1t11d0s2    sliced    -            -            error
    c1t11d0s2    sliced    -            -            error --> Invalid Device
    c1t12d0s2    sliced    -            -            error
    c1t13d0s2    sliced    disk02       rootdg       online
    -            -         disk01       rootdg       removed was:c1t11d0s2

The root cause of the problem is that if the VxVM is not disabled,
then the drive device node is dangling and should be removed from the
device tree.  After the new drive is inserted, a new node is created
on the device tree and will not see the dangling device node.
Implementation: 
---
        |   |   MANDATORY (Fully Proactive)
         ---    
         
  
         ---
        | X |   CONTROLLED PROACTIVE (per Sun Geo Plan) 
         --- 
         
                                
         ---
        |   |   REACTIVE (As Required)
         ---
Corrective Action: 
The following recommendation is provided as a guideline for authorized
SUN Services Field Representatives who may encounter the above
mentioned issue.

This issue has now been fixed in 113201-03 (For Solaris 6,7,8) and 
112385-05 (Solaris 9).  The appropriate patch need to be applied to avoid 
issue.
 
Once patches are appllied, the workaround in bug 4630477 should not be 
required and normal disk replacemnt procedure should now be followed to 
replace a failed disk.
Comments: 
Using 'vxdisk rm' command in conjunction with luxadm -e offline will
allow replacement of disks to succeed without having to reboot the
system, maintaining uptime and availablity, and allowing hotswap of
disks to remain a viable solution.  The advantage to using this
procedure is that whether the disk was pulled prior to taking any VM
action, or when using the procedure outlined in above steps, disk
replacement is successful without needing a reboot.  In reference to
SRDB 17003 using option 11 to offline a disk through VM, in some cases
this will work, and in other cases where the disk has physically been
pulled first, option 11 to offline disk won't cleanup any opens left.
Thus we will have the duplicate enties.

============================================================================
Implementation Footnote: 
i)   In case of MANDATORY FINs, Enterprise Services will attempt to    
     contact all affected customers to recommend implementation of 
     the FIN. 
   
ii)  For CONTROLLED PROACTIVE FINs, Enterprise Services mission critical    
     support teams will recommend implementation of the FIN  (to their  
     respective accounts), at the convenience of the customer. 

iii) For REACTIVE FINs, Enterprise Services will implement the FIN as the   
     need arises.
----------------------------------------------------------------------------
All released FINs and FCOs can be accessed using your favorite network 
browser as follows:
 
SunWeb Access:
-------------- 
* Access the top level URL of http://sdpsweb.ebay/FIN_FCO/

* From there, select the appropriate link to query or browse the FIN and
  FCO Homepage collections.
 
SunSolve Online Access:
-----------------------
* Access the SunSolve Online URL at http://sunsolve.Corp/

* From there, select the appropriate link to browse the FIN or FCO index.

Internet Access:
----------------
* Access the top level URL of https://infoserver.Sun.COM
--------------------------------------------------------------------------
General:
--------
* Send questions or comments to finfco-manager@sdpsweb.EBay
--------------------------------------------------------------------------
Statusactive