Sun Microsystems, Inc.  Sun System Handbook - ISO 3.4 June 2011 Internal/Partner Edition
   Home | Current Systems | Former STK Products | EOL Systems | Components | General Info | Search | Feedback

Asset ID: 1-71-1019720.1
Update Date:2011-05-25
Keywords:

Solution Type  Technical Instruction Sure

Solution  1019720.1 :   How to Replace the Motherboard or Service Processor in a Logical Domain (LDOM) Environment  


Related Items
  • Sun Blade T6300 Server Module
  •  
  • Sun Netra T5220 Server
  •  
  • Sun Netra T5440 Server
  •  
  • Sun SPARC Enterprise T5140 Server
  •  
  • Sun Blade T6340 Server Module
  •  
  • SPARC T3-2
  •  
  • SPARC T3-4
  •  
  • Sun SPARC Enterprise T2000 Server
  •  
  • Sun Blade T6320 Server Module
  •  
  • Sun Netra T2000 Server
  •  
  • SPARC T3-1
  •  
  • Sun SPARC Enterprise T5440 Server
  •  
  • Sun SPARC Enterprise T1000 Server
  •  
  • Sun Fire T1000 Server
  •  
  • Sun SPARC Enterprise T5220 Server
  •  
  • SPARC T3-1B
  •  
  • Sun SPARC Enterprise T5240 Server
  •  
  • Solaris SPARC Operating System
  •  
  • Sun Fire T2000 Server
  •  
  • Sun SPARC Enterprise T5120 Server
  •  
Related Categories
  • GCS>Sun Microsystems>Operating Systems>Solaris Operating System
  •  
  • GCS>Sun Microsystems>Servers>NEBS-Certified Servers
  •  
  • GCS>Sun Microsystems>Servers>CMT Servers
  •  
  • GCS>Sun Microsystems>Servers>Blade Servers
  •  

PreviouslyPublishedAs
244546


Applies to:

Sun Netra T2000 Server - Version: Not Applicable and later   [Release: N/A and later ]
Sun Netra T5440 Server - Version: Not Applicable and later ]
Sun Fire T1000 Server - Version: Not Applicable and later ]
Sun SPARC Enterprise T5220 Server - Version: Not Applicable and later ]
SPARC T3-2 - Version: Not Applicable and later    [Release: N/A and later]
All Platforms

Goal

How to recover the LDOM environment following the replacement of the service processor in a CMT capable system. The LDOM configuration is stored on the service processor in CMT systems.The service processor may be integral to the motherboard or an individual FRU. When this part is replaced the LDOM configuration will be lost.

This procedure explains the steps needed to recover the Logical Domains in a basic LDOM configuration.It does not include procedures for recovery of  Solaris ZFS or Veritas  Storage management.

Solution

The following procedure should be followed in the event of a system board or SC failure when the system is used in an LDOM environment. For LDoms 2.0 environments, reference Chapter 11 of the Administration Guide under "Managing Configurations" http://download.oracle.com/docs/cd/E19608-01/html/821-1485/index.html .
  • Backup of the ldom environment before hardware replacement.

  • Identify the ldom names using the ldm list-domain command

Example:
              # ldm list-domain
           NAME         STATE    FLAGS   CONS    VCPU  MEMORY   UTIL  UPTIME
           primary          active      -n-cv       SP            8                4G         0.1%   1d 4h 52m
     guest1  active -----       8      4G            
  • Capture the LDOM environment
  • Log in as an authorized user or superuser
  • Use each of the following methods to ensure the required information is captured
  • Run the command ldm ls -e -p
  • Run the command ldm list-constraints -x ldom > ldom.xml for each of the ldom names identified earlier
  • If this is not available then the system configuration can be obtained from the file /var/opt/SUNWldm/ldom-db.xml

*See the Appendix to this document for examples of the outputs from these commands. These examples are used in this procedure.

The system will display an errors similar to the following:

Oct 14 12:15:18 svc.startd[7]: svc:/network/physical:default: Method "/lib/svc/method/net-physical" failed with exit status 96.
Oct 14 12:15:18 svc.startd[7]: network/physical:default misconfigured: transitioned to maintenance (see 'svcs -xv' for details)
Oct 14 12:15:34 svc.startd[7]: svc:/ldoms/vntsd:default: Method "/lib/svc/method/svc-vntsd" failed with exit status 1.
Oct 14 12:15:34 svc.startd[7]: svc:/ldoms/vntsd:default: Method "/lib/svc/method/svc-vntsd" failed with exit status 1.
Oct 14 12:15:35 svc.startd[7]: svc:/ldoms/vntsd:default: Method "/lib/svc/method/svc-vntsd" failed with exit status 1.
Oct 14 12:15:35 svc.startd[7]: ldoms/vntsd:default failed: transitioned to maintenance (see 'svcs -xv' for details)

It will still be possible to login to the system, however there will be no network access.

  • Login to the system as root

  • Move the virtual network interface from /etc/hostname.vsw0 to the physical network interface.
    In this example the physical interface is e1000g0

# mv /etc/hostname.vsw0 /etc/hostname.e1000g0
  • Configure the physical network interface
# ifconfig e1000g0 plumb
# ifconfig e1000g0 ip_of_e1000g0 netmask netmask_of_e1000g0 broadcast + up
  • Reboot the system and then login again.

In some instances the previous svc-vntsd failure messages may still be displayed. However, it will still be possible to login.

  • Manually re-create the Primary Domain.

The following examples of the commands use the parameters from the example files in the Appendix.
Substitute the true values from the saved files.

  • Recreate the Virtual Disk Server (vds)


<ldom>
<ldom_name>primary</ldom_name>
-
-
-
<vds>
<vds_instance>
<service_name>primary-vds0</service_name>
<vds_volumes>
<vds_volume>

# ldm add-vds primary-vds0 primary

  • Re-create the Virtual Console Concentrator service (vcc)

 
<vcc>
<vcc_instance>
<service_name>primary-vcc0</service_name>
<min_port>5000</min_port>
<max_port>5100</max_port>
</vcc_instance>
</vcc>

# ldm add-vcc port-range=5000-5100 primary-vcc0 primary
  • Re-create the cryptographic resource

<mau>
<number>1</number>
</mau>

# ldm set-mau 1 primary

  • Re-create the Virtual CPUs

<cpu>
<number>8</number>
</cpu>

# ldm set-vcpu 8 primary

  • Re-create the memory

<memory>
<size>4294967296</size>
</memory>

# ldm set-memory 4G primary

  • Save the configuration to the service controller
# ldm add-config initial
  • List the saved configs

# ldm list-config

factory-default [current]
initial [next]

  • Perform a re-configuration boot of the system

# touch /reconfigure
# shutdown -y -g0 -i6

  • Power off and then power on the platform and allow the system to boot
  • Restore the virtual network

Please remember the following are examples , your system may be using different names and values.

# ifconfig -a
# ifconfig vsw0 plumb
# ifconfig e1000g0 down unplumb
# ifconfig vsw0 ip_of_e1000g0 netmask netamsk_of_e1000g0 broadcast + up
# mv /etc/hostname.e1000g0 /etc/hostname.vsw0
  • Check the virtual network terminal server daemon is enabled.


# svcs -l vntsd

fmri svc:/ldoms/vntsd:default
name virtual network terminal server
enabled true
state online
next_state none
state_time Wed Oct 15 16:03:54 2008
logfile /var/svc/log/ldoms-vntsd:default.log
restarter svc:/system/svc/restarter:default
contract_id 57
dependency optional_all/error svc:/milestone/network (online)
dependency optional_all/none svc:/system/system-log (online)

If this is not enabled use the following command


# svcadm enable vntsd

  • Restore the virtual disk server to the guest domains

Please remember the following are examples. Your system may be using different values and names.

<vds>
<vds_instance>
<service_name>primary-vds0</service_name>
<vds_volumes>
<vds_volume>
<block_dev>/dev/dsk/c1t1d0s2</block_dev>
<vol_name>vol1</vol_name>
<vol_opts/>
</vds_volume>
</vds_volumes>
</vds_instance>
</vds>

# ldm add-vdsdev /dev/dsk/c1t1d0s2 vol1@primary-vds0

  • Finally Bind and Start the remaining domains.

In this example the guest domain is named guest1.

# ldm bind-domain guest1
# ldm start-domain guest1

 


Appendix:

Output from command ldm ls -e -p

 

VERSION 1.1
DOMAIN|name=primary|state=active|flags=normal,control,vio-service|cons=SP|ncpu=8|mem=4294967296|util=12.8|uptime=684412|softstate=Solaris running
VCPU
|vid=0|pid=0|util= 17|strand=100
|vid=1|pid=1|util=0.4|strand=100
|vid=2|pid=2|util=0.4|strand=100
vid=3|pid=3|util=0.4|strand=100
|vid=4|pid=4|util= 81|strand=100
|vid=5|pid=5|util=1.5|strand=100
|vid=6|pid=6|util=2.1|strand=100
|vid=7|pid=7|util=0.0|strand=100
MAU
|id=0|cpuset=0,1,2,3,4,5,6,7
MEMORY
|ra=0x8000000|pa=0x8000000|size=4294967296
IO
|dev=pci@0|alias=pci
|dev=niu@80|alias=niu
VCC|name=primary-vcc0|nclients=1|port-range=5000-5100
VSW|name=primary-vsw0|nclients=1|mac-addr=00:14:4f:fb:6f:f9|net-dev=e1000g0|dev=switch@0|mode=
VDS|name=primary-vds0|nclients=1
|vol=vol1|opts=|dev=/dev/dsk/c1t1d0s2
VLDC|name=primary-vldc3|nclients=7
VLDC|name=primary-vldc0|nclients=4
VLDCC|name=ds|service=primary-vldc0@primary|desc=domain-services
VLDCC|name=vldcc2|service=SP|desc=spfma
VLDCC|name=vldcc1|service=primary-vldc0@primary|desc=ldmfma
VCONS|type=SP
DOMAIN|name=guest1|state=active|flags=normal|cons=5000|ncpu=8|mem=4294967296|util=0.1|uptime=512444|softstate=Solaris running
VCPU
|vid=0|pid=8|util= 75|strand=100
|vid=1|pid=9|util= 49|strand=100
|vid=2|pid=10|util= 11|strand=100
|vid=3|pid=11|util=0.0|strand=100
|vid=4|pid=12|util=100|strand=100
|vid=5|pid=13|util=0.0|strand=100
|vid=6|pid=14|util=0.0|strand=100
|vid=7|pid=15|util=0.0|strand=100
MEMORY
|ra=0x8000000|pa=0x108000000|size=4294967296
VARIABLES
|auto-boot?=true
|boot-device=vdisk1
|keyboard-layout=US-English
VNET|name=vnet1|dev=network@0|service=primary-vsw0@primary|mac-addr=00:14:4f:f9:15:9e
VDISK|name=vdisk1|vol=vol1@primary-vds0|dev=disk@0|server=primary
VLDCC|name=ds|service=primary-vldc0@primary|desc=domain-services
VCONS|group=guest1|service=primary-vcc0@primary|port=5000

 


Output from ldm list-constraints command for control/primary domain

ldm list-constraints -x ldom


<?xml version="1.0"?>
<LDM_interface version="1.1" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:noNamespaceSchemaLocation="./schemas/combined-v3.xsd" xmlns:ovf="./schemas/envelope" xmlns:rasd="./schemas/CIM_ResourceAllocationSettingData" xmlns:vssd="./schemas/CIM_VirtualSystemSettingData" xmlns:gprop="./schemas/GenericProperty" xmlns:bind="./schemas/Binding">
<data version="3.0">
<Envelope>
<References/>
<Content xsi:type="ovf:VirtualSystem_Type" ovf:id="primary">
<Section xsi:type="ovf:ResourceAllocationSection_Type">
<Item>
<rasd:OtherResourceType>ldom_info</rasd:OtherResourceType>
<rasd:Address>00:14:4f:ab:e3:16</rasd:Address>
</Item>
</Section>
<Section xsi:type="ovf:VirtualHardwareSection_Type">
<Item>
<rasd:OtherResourceType>cpu</rasd:OtherResourceType>
<rasd:AllocationUnits>8</rasd:AllocationUnits>
</Item>
</Section>
<Section xsi:type="ovf:VirtualHardwareSection_Type">
<Item>
<rasd:OtherResourceType>mau</rasd:OtherResourceType>
<rasd:AllocationUnits>1</rasd:AllocationUnits>
</Item>
</Section>
<Section xsi:type="ovf:VirtualHardwareSection_Type">
<Item>
<rasd:OtherResourceType>memory</rasd:OtherResourceType>
<rasd:AllocationUnits>4G</rasd:AllocationUnits>
</Item>
</Section>
<Section xsi:type="ovf:VirtualHardwareSection_Type">
<Item>
<rasd:OtherResourceType>physio_device</rasd:OtherResourceType>
<gprop:GenericProperty key="name">pci@0</gprop:GenericProperty>
</Item>
</Section>
<Section xsi:type="ovf:VirtualHardwareSection_Type">
<Item>
<rasd:OtherResourceType>physio_device</rasd:OtherResourceType>
<gprop:GenericProperty key="name">niu@80</gprop:GenericProperty>
</Item>
</Section>
<Section xsi:type="ovf:VirtualHardwareSection_Type">
<Item>
<rasd:OtherResourceType>vsw</rasd:OtherResourceType>
<rasd:Address>auto-allocated</rasd:Address>
<gprop:GenericProperty key="service_name">primary-vsw0</gprop:GenericProperty>
<gprop:GenericProperty key="dev_path">e1000g0</gprop:GenericProperty>
</Item>
</Section>
<Section xsi:type="ovf:VirtualHardwareSection_Type">
<Item>
<rasd:OtherResourceType>vcc</rasd:OtherResourceType>
<gprop:GenericProperty key="service_name">primary-vcc0</gprop:GenericProperty>
<gprop:GenericProperty key="min_port">5000</gprop:GenericProperty>
<gprop:GenericProperty key="max_port">5100</gprop:GenericProperty>
</Item>
</Section>
<Section xsi:type="ovf:VirtualHardwareSection_Type">
<Item>
<rasd:OtherResourceType>vds</rasd:OtherResourceType>
<gprop:GenericProperty key="service_name">primary-vds0</gprop:GenericProperty>
</Item>
</Section>
<Section xsi:type="ovf:VirtualHardwareSection_Type">
<Item>
<rasd:OtherResourceType>vds_volume</rasd:OtherResourceType>
<gprop:GenericProperty key="vol_name">vol1</gprop:GenericProperty>
<gprop:GenericProperty key="block_dev">/dev/dsk/c1t1d0s2</gprop:GenericProperty>
<gprop:GenericProperty key="service_name">primary-vds0</gprop:GenericProperty>
</Item>
</Section>
</Content>
</Envelope>
</data>
</LDM_interface>
Output from the file /var/opt/SUNWldm/ldom-db.xml
<?xml version="1.0"?>
<ldom_database version="3.0">
<ldom>
<ldom_name>primary</ldom_name>
<cpu>
<number>8</number>
</cpu>
<mau>
<number>1</number>
</mau>
<memory>
<size>4294967296</size>
</memory>
<io>
<io_instance>
<iodevice>pci@0</iodevice>
<bypass_mode>0</bypass_mode>
</io_instance>
<io_instance>
<iodevice>niu@80</iodevice>
<bypass_mode>0</bypass_mode>
</io_instance>
</io>
<mac_address>0x00144fabe316</mac_address>
<vsw>
<vsw_instance>
<service_name>primary-vsw0</service_name>
<mac_address>0x00144ffb6ff9</mac_address>
<dev_path>e1000g0</dev_path>
<mode/>
</vsw_instance>
</vsw>
<network/>
<vds>
<vds_instance>
<service_name>primary-vds0</service_name>
<vds_volumes>
<vds_volume>
<block_dev>/dev/dsk/c1t1d0s2</block_dev>
<vol_name>vol1</vol_name>
<vol_opts/>
</vds_volume>
</vds_volumes>
</vds_instance>
</vds>
<disk/>
<vcc>
<vcc_instance>
<service_name>primary-vcc0</service_name>
<min_port>5000</min_port>
<max_port>5100</max_port>
</vcc_instance>
</vcc>
<console/>
<vdpcs/>
<vdpcc/>
<variables/>
<options/>
</ldom>
<ldom>
<ldom_name>guest1</ldom_name>
<cpu>
<number>8</number>
</cpu>
<mau><number>0</number>
</mau>
<memory>
<size>4294967296</size>
</memory>
<io/>
<mac_address>0x00144ffab8f0</mac_address>
<vsw/>
<network>
<network_instance>
<vnet_name>vnet1</vnet_name>
<service_name>primary-vsw0</service_name>
<mac_address>0x00144ff9159e</mac_address>
<mtu>1500</mtu>
</network_instance>
</network>
<vds/>
<disk>
<disk_instance>
<vdisk_name>vdisk1</vdisk_name>
<service_name>primary-vds0</service_name>
<vol_name>vol1</vol_name>
<timeout>-1</timeout>
</disk_instance>
</disk>
<vcc/>
<console>
<console_instance>
<service_name>primary-vcc0</service_name>
<group>guest1</group>
<port>5000</port>
</console_instance>
</console>
<vdpcs/>
<vdpcc/>
<variables>
<var>
<name>auto-boot?</name>
<value>true</value>
</var>
<var>
<name>boot-device</name>
<value>vdisk1</value>
</var>
<var>
<name>keyboard-layout</name>
<value>US-English</value>
</var>
</variables>
<options/>
</ldom>
</ldom_database>



To discuss this information further with Oracle experts and industry peers, we encourage you to review, join or start a discussion in the My Oracle Support Community, Oracle Solaris Installation, Booting and Patching Community.



Product
Sun Fire T2000 Server
Sun Fire T1000 Server
Sun Netra T2000 Server
Sun Netra T5220 Server
Sun Netra T5440 Server
Sun SPARC Enterprise T5220 Server
Sun SPARC Enterprise T5240 Server
Sun Blade T6300 Server Module
Sun Blade T6320 Server Module
Sun Blade T6340 Server Module
Sun SPARC Enterprise T5120 Server
Sun SPARC Enterprise T5140 Server
Sun SPARC Enterprise T2000 Server
Sun SPARC Enterprise T5440 Server
Sun SPARC Enterprise T1000 Server

Internal Comments
comments

This document contains normalized content and is managed by the the Domain Lead(s) of the respective domains. To notify content owners of a knowledge gap contained in this document, and/or prior to updating this document, please contact the domain engineers that are managing this document via the "Document Feedback" alias(es) listed below:

Domain Lead: dencho.kojucharov@sun.com

LDOM, ldom, CMT, recovery, "logical domain"

Attachments
This solution has no attachment
  Copyright © 2011 Sun Microsystems, Inc.  All rights reserved.
 Feedback