Sun Ray Hardware
 
Sun Ray Hardware
 
Sun Ray Software
 
Sun Ray Hardware
 
Sun Ray Hardware
 
Sun Ray Hardware
 

GCS>Sun Microsystems>Desktops>Desktop Virtualization>Sun Ray Hardware
 
GCS>Sun Microsystems>Desktops>Desktop Virtualization>Sun Ray Software
 

This guide explains the significance of the green newt cursor. The corresponding OSD status code for Sun Ray appliances running 2.0 firmware is 26 D.

Similarly, a Sun Ray appliance running 2.0 or higher firmware will display OSD icon 26 D if it has received all DHCP parameters, has successfully made initial connection to an authentication manager, and is now waiting for an X server to send display request to the appliance.

Note: if you are running the Sun Ray Server Software 2.0 or 3.0 in a remote LAN configuration, you must update the firmware before putting an appliance into the network. 1.x firmware does not support remote LAN configurations. A remote LAN configuration exists if Sun Ray appliances are not within the same subnet as the Sun Ray server.

On Solaris[TM], the Xsun is started by the dtlogin master process.
The dtlogin master process reads two configuration files:

    /etc/dt/config/Xservers
/etc/dt/config/Xconfig

Sun Ray replaces the above two files with links to /tmp/SUNWut/config/xconfig. When a new Sun Ray[TM] session is created, the authentication manager (utauthd) will spawn subprocesses which add the corresponding lines to Xservers and Xconfig, and subsequently a SIGHUP will be sent to the dtlogin master process to make it rescan Xservers and Xconfig.

On Linux, the gdm and Xnewt shipped with the SRSS must be used. The authentication manager still uses /tmp/SUNWut/config/xconfig/Xservers and /tmp/SUNWut/config/xconfig/Xconfig.

Related logfiles:
On Solaris:
- /var/dt/Xerrors (dtlogin and Xsun)
- /var/opt/SUNWut/log/messages and /var/opt/SUNWut/log/auth_log (utauthd)
- /var/adm/messages

On Linux (SRSS 3.0):
- /var/log/gdm (gdm and Xnewt)
- /var/opt/SUNWut/log/messages and /var/opt/SUNWut/log/auth_log (utauthd)
- /var/log/messages

Resolution:

If the green newt cursor, resp. OSD icon 26 D, is displayed for an extended period, there is no X server sending display rendering commands to the Sun Ray appliance. The problem can usually be traced back to
- a network problem, where packets are being dropped, or are blocked by a firewall. On a Linux Sun Ray server, turning on a local firewall with default settings blocks various ports essential for Sun Ray services.
- an overloaded system, where processes are behaving sluggishly.
- session startup failed
+ the dtlogin master process (resp. gdm on Linux) not running, or being unresponsive, or being unable to start the session's processes.
+ the X server crashing, looping, or being unable to access the display.
+ the subprocesses utauthd spawns to trigger session creation failed

To troubleshoot a green newt cursor, you need to consider:

    Are the patches up to date?
Is it only one session which shows the green newt/icon 26 D?
Are the daemons up and responsive?
Is there really a problem?
Is the problem caused by lack of system resources?
Is the problem caused by a misconfiguration?
Are the configuration files corrupt?

Are the patches up to date?

The following patches contain related bugfixes:
- the Sun Ray[TM] Server Software patch
- the dtlogin patch
- the Xsun patch
- On Solaris[TM] 8, the linker patch 109147-14 (4469684 and 4723552)
- On Solaris 8, the libraries patch 108993-25 (4479187).
- The ocfserv patch (if ocfserv is enabled in /etc/inet/inetd.conf).

Is it only one session which shows the green newt/icon 26 D?

If it is only one session which is stuck at the green newt/icon 26 D, some processes essential for that sessions are hanging, looping, or failed to start up. Typically, the user can resolve this problem by pressing Ctrl-Alt-Backspace twice to terminate the existing session.

Example:

In the example, the user with smartcard JavaBadge-nonPers.4090009c227b2d0a3c13 reports that he gets persistent icon 26 D.

The display number of the user's session is 18:
/tmp/SUNWut/config/displays % grep 9c227b2d0a3c13 *
18:TOKEN=JavaBadge-nonPers.4090009c227b2d0a3c13
18:TOKEN_SET=JavaBadge-nonPers.4090009c227b2d0a3c13
18:INSERT_TOKEN=JavaBadge-nonPers.4090009c227b2d0a3c13

ptree output shows that the session for display :18 is incomplete:
21552 /bin/ksh -p /opt/SUNWut/lib/utxinit /etc/opt/SUNWut/gulogin.start pseudo.
21555 /usr/openwin/bin/Xsun :18 -nobanner -terminate -auth /tmp/SUNWut/config
21573 /opt/SUNWut/lib/utxexec /etc/opt/SUNWut/gulogin.start pseudo.080020fe01
[21573 has no child processes]

ps output shows that the Xsun process is looping:
8 O root 21555 21552 13 89 20 ? 2698 Mar 09 ? 19440:33 /usr/openwin/bin/Xsun :18 -nobanner

Thus, in this example, one would kill the runaway Xsun process.

Are the daemons up and responsive?

- dtlogin master process

 + The pid of the dtlogin master process is logged in /var/dt/Xpid.
Check that this process is running. If the dtlogin master process crashes,
dtlogin child process will have parent process id 1, existing user
sessions will continue to run, but no new sessions
can be created. If neither dtlogin master process nor dtlogin child
processes are there, check that /etc/rc2.d/S99dtlogin exists.
Warning: on a Sun Ray server, you cannot just restart the
dtlogin master process. If dtlogin has crashed, you MUST reboot.
+ Check that all dtlogin child process are child processes of the
dtlogin master process. If there are dtlogin child processes with parent
process id 1, probably somebody restarted the dtlogin parent process,
which can cause green newt conditions on a Sun Ray[TM] server.
+ Check that the dtlogin master process is not leaking memory.
Normal size of the dtlogin master process is 5-6 MB.
+ Check /var/dt/Xerrors for messages indicating that dtlogin
received and processed a SIGHUP:

   Fri Apr 30 17:27:45 2004
info (pid 495): Rebuilding default language list from /usr/lib/locale

   Fri Apr 30 17:27:45 2004
info (pid 495): Rescanning both config and servers files

   Fri Apr 30 17:27:45 2004
info (pid 495): Rereading configuration file /etc/dt/config/Xconfig

   If you do not see such messages, then either the dtlogin master
(pid 495 here) did not receive a SIGHUP from the Sun Ray[TM] session
startup scripts, or it is in a dysfunctional state. If you think
that the dtlogin master process might be unresponsive, send it
a SIGHUP ("kill -HUP <dtlogin_pid>"), and then check whether lines
such as the above are appended to /var/dt/Xerrors.

- utauthd

 Note: a problem with utauthd will cause green newt conditions only if
the subprocesses spawned by utauthd somehow fail. If utauthd itself is
unresponsive, you normally get OSD icons 22 (2.0/3.0 firmware) resp. the
hourglass with three dashes (1.x firmware).
+ Check in ps output that exactly one utauthd is running.
+ Check in ps output that utauthd was started after the dtlogin
master process, and after utdsd (2.0/3.0) resp dsservd (1.x).
+ check in ptree (pstree on Linux) output that no subprocesses spawned
by utauthd are hanging. Example ptree of a failure because the loopback
interface was unresponsive:
63455 /opt/SUNWut/jre/bin/../bin/sparc/native_threads/java auth.utauthd.utauthd
20313 /bin/ksh -p /opt/SUNWut/lib/utdtsession -c gulogin add
20773 /bin/ksh -p /opt/SUNWut/lib/utdmsession -c 11 -z utdtsession
20782 /bin/ksh -p /opt/SUNWut/lib/utscrevent -c 11 -z utdtsession
20784 ksh -c echo 'CREATE_SESSION 11 # utdtsession' >/dev/tcp/127.0.0.1/7013
[... more of those ...]
+ Check /var/opt/SUNWut/log/messages for utauthd activity, and
for error messages indicating that other Sun Ray[TM] services were
unable to contact utauthd.
+ Telnet to utauthd callback port 7010, and enter "status".
You should get information about the Sun Ray appliances currently
in contact with utauthd.
+ If you suspect misbehaviour of a running utauthd, collect a
pstack and a java threaddump of the utauthd ("kill -3 <utauthd_pid>";
the utauthd java threaddump will be written into auth_log), and check
that all worker threads and callback thread are alive.
There should be one callback thread and eight worker threads.
As the number of worker threads is configurable, the following
command will tell you what your configuration is:
# grep workers /etc/opt/SUNWut/auth.props

- utdevmgrd

 + Check that one watchdog and one child utdevmgrd process are running.
+ Check /var/opt/SUNWut/log/messages for error messages indicating
utdevmgrd failure, such as
Apr 14 23:35:21 [10.21.3.224.2.2]  0x0.0xc46ce4f 0:3:ba:2a:54:c8 USB:
rdd: unable to contact device manager on 10.21.3.1:7011  retrying
+ Check /var/adm/messages for error messages from utdevmgrd

- utsessiond

 + Check that one watchdog and one child utsessiond process are running.

- utdsd/dsservd

 + Check that one utdsd (2.0/3.0) resp. dsservd (1.x) daemon is running.   

- ocfserv (only if enabled in the inetd.conf, only for 1.3 and 2.0)

 + Check whether there are utscrevent processes hanging. In 1.3 and 2.0,
utscrevent tries to connect to ocfserv if ocfserv is running.
If ocfserv is responsive, utscrevent will finish within a few seconds.

It also is recommended that coreadm is turned on with global-setid cores enabled, so that in case of a dtlogin or X server crash corefiles are collected:

   # coreadm -e global -e global-setid -g /var/cores/core.%f.%t
# mkdir /var/cores

Is there really a problem?

The Sun Ray administration model has several user session types:

    Default     --   normal user login
NSCM        --   non smart card mobility
Kiosk       --   anonymous user operation
utselect -L --   utselect in "login" mode
Register    --   user self-registration
Insert card --   user smart card required
Card error  --   unrecognized user smart card type
No entry    --   user's smart card token is blocked

The first two session types have normal login processes. The last four session types do not have a login process at all, but display an icon on the Sun Ray appliance monitor, possibly along with the green newt cursor. This icon indicates that the user must take other steps before successful login is possible.

These last four session types, their icons, and the appearance of the green newt cursor are not cause for alarm. The user can:

    - Insert a recognized smart card in the correct orientation
- Ask the Sun Ray[TM] administrator to grant access

For the SRSS 2.0, see pages 195ff. of the SRSS 2.0 Administration Guide for detailed information regarding Sun Ray appliance startup, and the icons displayed.

With the SRSS 1.3, the icons and the green newt are explained in Appendix A of the SRSS 1.3 Administration Guide, pages 95ff.

Is the problem caused by lack of system resources?

Sluggish Sun Ray server performance or excessive disk swapping is an indication that the Sun Ray[TM] server is under-provisioned. Under these circumstances, there might be insufficient virtual memory available to start an X Window server instance for a user's session. If after several retries the Xsun process does not start, the dtlogin daemon just gives up. With no X Window server running, the cursor remains a green newt.

Typical error messages in /var/adm/messages are
"WARNING: /tmp: File system full, swap space limit exceeded"
Typical error messages in /var/opt/SUNWut/log/messages are
"UNEXPECTED: SESSION_ERROR ... java.io.IOException:
Not enough space </opt/SUNWut/lib/utdtsession add>"
"UNEXPECTED: AuthRecord::CreateClient:/opt/SUNWut/lib/utdtsession
add :child error: write: No space left on device"

The solution in this situation is to add more memory and/or increase the size of the swap partition. The suggested Sun Ray server configuration includes approximately 50-100 MBytes of swap space per user. See the Sun Ray[TM] Server Software Installation Guide for additional hardware requirements.

Furthermore, if the system is overloaded, or utauthd resp. utsessiond are overloaded, utauthd and utsessiond might not be passing information efficiently to the X servers.

Is the problem caused by a misconfiguration?

- /etc/dt/config/Xconfig must link to /tmp/SUNWut/config/xconfig/Xconfig
- /etc/dt/config/Xservers must link to /tmp/SUNWut/config/xconfig/Xservers
- If there are any custom X configuration scripts in /etc/dt/config, they must contain the corresponding Sun Ray specific code from the default X configuration files in /usr/dt/config.
- Several subdirectories of /var/opt/SUNWut must link into /tmp/SUNWut/config.
- As /tmp/SUNWut contains important Sun Ray dynamic data, there may
be no script or cronjob which deletes files from /tmp/SUNWut.
- Nor may /tmp/.X11-pipe and /tmp/.X11-unix be touched by scripts.
- Some other process might block port 60XX which Xsun :XX needs. See also
<Document: 1003369.1> "Conflicts between Sun Rays[TM] and remote display applications"

Are the configuration files corrupt?

These configuration files are susceptible to corruption, especially if cronjobs or scripts touch /tmp, if the system ran out of swap space, or if /tmp is not mounted on a tmpfs.

    /etc/dt/config/Xservers (links to /tmp/SUNWut/config/xconfig/Xservers)
/etc/dt/config/Xconfig  (links to /tmp/SUNWut/config/xconfig/Xconfig)
/tmp/SUNWut/config/displays

The first two files are used by the dtlogin master process. When they are corrupt, the dtlogin daemon cannot properly start the Xsun. Without an X server running, the cursor remains a green newt. The files in the /tmp/SUNWut/config/displays directory contain information needed by the Xsun process at session startup.

Some symptoms for corruption of the configuration files are:

  + Error message in /var/opt/SUNWut/log/messages:

   Error: could not open file '/var/opt/SUNWut/displays/<display>'
(this message means that the file /tmp/SUNWut/config/displays/<display>
does not exist or that the link from /var/opt/SUNWut/displays
to /tmp/SUNWut/config/displays is broken)
+ Example error messages in /var/dt/Xerrors:
Fatal server error:
SunRay SessionId not specified!
Signal 10 received! (pid 28464)
[...]

   and

   Fatal server error:
error (<pid>): Server for <display> terminated unexpectedly 0

   error (pid <pid>): Server open attempt #8 failed for <display>,
giving up

   error (pid <pid>): Hung in XOpenDisplay(<display>)attempt #<nr>,
aborting.

   These messages mean that the Xsun died because it could not extract
the Sun Ray session ID from /tmp/SUNWut/config/displays/<display>,
either because the file does not exist or because it is corrupted.

    /etc/dt/config/Xservers should contain lines such as the following.

    Solaris version:
# BEGIN SUNRAY CONFIGURATION
:4 SunRay local@none /usr/openwin/bin/Xsun :8 -nobanner
.
.
:5 SunRay local@none /usr/openwin/bin/Xsun :9 -nobanner
# END SUNRAY CONFIGURATION

    Linux version:
# BEGIN SUNRAY CONFIGURATION
:21 SunRay local /usr/X11R6/bin/Xnewt :21
[...]
# END SUNRAY CONFIGURATION

    There also might be a few
# :<display> RESERVED
lines in there, where a display has been reserved for use in
a future Sun Ray session. A RESERVED entry might for example
indicate a session at the NSCM login GUI.
Note - These are simplified examples. Your output may have tens of
lines between the BEGIN SUNRAY CONFIGURATION and END SUNRAY
CONFIGURATION comments. 

    /etc/dt/config/Xconfig should contain lines such as the following:

    Solaris version:
# BEGIN SUNRAY CONFIGURATION
Dtlogin.*_4.environment: SUN_SUNRAY_TOKEN=pseudo.080020c0ec84
CORONA_TOKEN=pseudo.080020c0ec84
.
.
Dtlogin.*_5.environment: SUN_SUNRAY_TOKEN=user.1051355046-6093
CORONA_TOKEN=user.1051355046-6093
# END SUNRAY CONFIGURATION

    Linux version:
# BEGIN SUNRAY CONFIGURATION
DisplayManager.*_20.exportList: SUN_SUNRAY_TOKEN=MicroPayflex.000095af65000100 CORONA_TOKEN=MicroPayflex.000095af65000100
# END SUNRAY CONFIGURATION

    For each display number which currently is in use (4 and 5 in this
example), there must be one line in both Xservers and Xconfig, and
one corresponding file in /tmp/SUNWut/config/displays. All three
must be in sync. 

    % cat /tmp/SUNWut/config/displays/4
SESSION=labhost:7007:5380295458739376318  <---- session id
TOKEN=pseudo.080020c0ec84
SESSION_TYPE=default
TOKEN_SET=pseudo.080020c0ec84
CALLBACK_COOKIE=8891015337650688598
DISPLAY=4  <---- the display number yet again.
INSERT_TOKEN=pseudo.080020c0ec84

    The tokens in the displays file must match the token in the
corresponding line in Xconfig. 

This does not have any impact on existing Sun Ray[TM] sessions. Sun Ray 1.3 and 2.0 uses ocfserv when it is enabled, but does not need it for normal operations.

If the utauthd is hanging, or has crashed, restart utauth without terminating existing sessions:
Procedure for SRSS 1.x:
# /opt/SUNWut/sbin/utpolicy -i soft
Procedure for SRSS 2.0:
# /opt/SUNWut/sbin/utrestart

If the dtlogin master process has crashed, or is unresponsive, schedule an outage to reboot the Sun Ray server.

OSD - On Screen Display

On Linux systems which do not provide the NPTL, multithreading is implemented by spawning a separate process for each thread. 20 utauthd processes on SLES 8.1 or JDS 2 are normal, they just represent the different utauthd threads.

currency review 4/16/10

This solution has no attachment