VM won't start and now is in a Stuck State

From time to time Vm's can get stuck in a state where you cannot start, shutdown, reboot or even force a reboot or shutdown. When this happens from the console type: xe-toolstack-restart (note that your XenCenter will loose connection when you do this. Nothing will however happen to your running VMs)

Reconnect with XenCenter (or let it reconnect by itself) and you should see your VM no longer in a stuck state

Appliance lock Regkeys

When using the Citrix Appliance lock if you ever need to disable the Windows logon settings to stop using the PNAAgent at log on change the following reg key from

HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows NT\CurrentVersion\Winlogon
"Shell"="C:\Program Files\Citrix\ICA Client\pnagent.exe"

To

HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows NT\CurrentVersion\Winlogon
"Shell"="Explorer.exe"

This will enable Windows Explorer to start during Windows logon and not the PNAAgent.

XenServer Storage Test VM

Please check out www.kaztechsolutions.co.uk for more of my technical posts, alternately please call us on 01932 268289. 

PerformanceVM is a XenServer virtual machine that helps troubleshoot performance related issues such as poor performance casued by storage I/O and network I/O. The VM built on Debian Linux equips with following test utilities and is accessible via a web based user interface:

Disk I/O performance utility
It can be used to conduct the following disk I/Os to measure: sequential read/writes and random read/writes with various specified block sizes.

Network I/O performance utility
It is essentially a modified version of netperf. Additional information about netperf can be found at http://www.netperf.org. Netperf runs on the backend and provides end-to-end request/response round trip latency and TCP/UPD throughout tests.

download the VM from the following locations http://xenbits.xensource.com/export/1/perfmon/PerfVM1_0.zip and Import the VM in to XenCenter.


Running PerformanceVM

Access the console
1. SSH into the PerformanceVM’s IP address.
2. Enter the 'root' user name and password. The root user log on credentials are:
User name: root
Password: xensupport

Start the web server
1. Start the web service using the script and the IP address of the virtual machine:
/start.sh [XenPOOLMASTERAddress]
2. When prompted for a password, enter:
The password for [XenPOOLMASTERAddress]
3. A prompt to inflate thin provisioned VDIs will be presented. Answer yes or no.

Note: If the Disk I/O test returns values that are off of the graph, then the above steps
probably need to be run in order to accurately reflect the performance.

4. In a web browser, enter http://[PerfVMAddress ]:8888/ to start testing.

Note: If XenServer hasn’t reported the VM’s IP address yet to XenCenter, the scripts will fail to run.

Network I/O Performance Utility
Running the Network utility requires that Netserver be installed on a remote system. This can be downloaded from the Network I/O utility page. Netserver can be installed on a Linux VM or dom0 at a remote site, where the destination testing will occur. Start Netserver before running any network performance tests.

Disk I/O performance utility
Running the Disk Utility will show real time performance of the specified disk. Access to Dom0 is required to access the performance data, but Dom0 is not modified. It can be run on master or slave machines.

Additional results are provided and will show up to the last 10 minutes of a test period. This will include various other counters along with the Disk results. Final results are graphed on a separate results page.

How to increase XenServer HA TimeOut

Use the following two commands from CLI to increase the HA timeout on your XenServer, first you have to disbale HA then re-enable HA via the CLI with the timeout switch. I've found that 120 seconds works best but it all depends on your environment and what suits your needs best.

xe pool-ha-enable heartbeat-sr-uuids=ad1983ed-66cb-8fd7-1ef5-66a41c9d4345 ha-config:timeout=XX

Obviously change the SR UUID to the UUID of your own SR and where =XX change this to the number of seconds to wait until HA kicks in by default, it is 30 seconds.

NOTE: to get your list of SR UUID's type xe sr-list in the console

Separating the Roles of DDC's in a XenDesktop Farm

Each Desktop Delivery Controller machine in the farm can potentially become the farm master in an election process. This election process can be influenced by settings on the various server machines, and one (or more) machines can be indicated as preferred as being farm master, and other machines only take on the farm master role if the preferred machines are unavailable. This preference indication is achieved by use of registry entries on the various server machines. Each machine can be configured to have one of four possible settings:

• Master – servers with this setting are preferentially chosen as the farm master. Often, only one server in the farm would have this setting.

• Backup - servers with this setting are preferentially chosen as the farm master when the master server is unavailable.

• Member - servers with this setting are normally not the farm master, but can assume the farm master role when none of the master or backup servers are available.

• Slave Only - servers with this setting are not eligible to be the farm master.

To configure a server machine to one of these settings, perform the following steps.

Caution! This procedure requires you to edit the registry. Using Registry Editor incorrectly can cause serious problems that may require you to reinstall your operating system. Citrix cannot guarantee that problems resulting from the incorrect use of Registry Editor can be solved. Use Registry Editor at your own risk. Back up the registry before you edit it.

1. Set the HKLM\Software\Citrix\IMA\RUNTIME\UseRegistrySetting DWORD registry entry to 1

2. Set the desired master ranking value in the HKLM\Software\Citrix\IMA\RUNTIME\MasterRanking DWORD registry entry to one of the following values:

• 1 indicates ‘Master’
• 2 indicates ‘Backup’
• 3 indicates ‘Member’
• 4 indicates ‘Slave Only’

3. Restart the server

Enabling better PVS Logs

ere how to turn on better logging:

Console log file: “c:\documents and settings\all users\application data\citrix\provisioning server\eeapi.log”
Config file: “c:\program files\citrix\provisioning server console\enterpriseaccess_log.config”

Stream process log file: “c:\documents and settings\all users\application data\citrix\provisioning server\stream.log”
Config file: “c:\program files\citrix\provisioning server\stream_log.config”

Soap server log file: “c:\documents and settings\all users\application data\citrix\provisioning server\soapserver.log”
Config file: “c:\program files\citrix\provisioning server\soapserver_log.config”

MCLI log file: “c:\documents and settings\all users\application data\citrix\provisioning server\mcli.log”
Config file: “c:\program files\citrix\provisioning server\mcli_log.config”

In each case, the .config file controls the level output according to the “level value” entry towards the bottom of the file:




The three levels are ERROR, DEBUG and TRACE with ERROR being the default value installed with the product. DEBUG will be the most useful for diagnosing customer issues. The corresponding service does not require restarting after the log level has been changed.

Get some logs and we can start to troubleshoot.

Change Pool Master via CLI

First disable high availability:
xe pool-ha-disable


Now list your XenServer hosts:
xe host-list


Using the list above, designate a new pool master by supplying the uuid associated with the desired host:
xe pool-designate-new-master host-uuid=


You’ll probably lose connection to the pool at this point, but that’s normal behavior. Once XenCenter reconnects (this should happen automatically), it’s safe to re-enable high availability:
xe pool-ha-enable

XenServer CLI - Commands

Please check out www.kaztechsolutions.co.uk for more of my technical posts, alternately please call us on 01932 268289. 

Since there is so many command within XenServer sometimes its hard to remember which command you require. This is just a list of regular commands that I use but please feel free to point out your favourite commands.

To find anything with the word "Server1" in messages logs. Obviously change the word to what you require but if you'r having a problem with a server named Server1 of instances you can enter the command below to filter out any information with Server1 in the logs.

grep Server1 /var/log/messages

To list your interfaces so you can identify the right UUID for your network device.

xe pif-list

To list full device details for the network device. TIP: once you have the UUID's of all of your interfaces simply enter the first couple of characters of the UUID interface and then press tab to fully populate the UUID.

xe pif-param-list uuid=(UUID of pif)

In XenServer 5 FP 1 you have the option to use vSwiches instead of the traditional bridging technology, the default install of XenServer does not have this enabled so you will have to run the following command to enable it. NOTE: you have to restart your XenServer after you have entered this command.

xe-switch-network-backend openvswitch


With XenServer it recommended that turn off auto-negotiation, set duplex to Full, and speed to 1000 using the Command Line Interface (CLI). To do this first find the UUID of the storage interface using the following command.

xe pif-list

Next set the parameters of the physical interface (PIF) using the following commands.

xe pif-param-set uuid= other-config:ethtool-autoneg=”off”
xe pif-param-set uuid= other-config:ethtool-speed=1000
xe pif-param-set uuid= other-config:ethtool-duplex=”full”

Restart the XAPI service because it controls networking on the XenServer, an alternative option is to reboot the XenServer. If you are in a pool, the above changes must be done on each storage PIF of the XenServers.

With XenServer its sometimes necessary to restart the iSCSI service on a XenServer, to do so run the following command from the CLI

service open-iscsi restart

Some times in XenCenter you'll get a VM that's in a "halted" state and you cannot even perform a force shutdown/reboot and even a xe-toolstack-restart will not shutdown the VM. To run a force restart from a CLI command run the following commands.

xe vm-shutdown -u root vm-name=vm_name
xe vm-shutdown -u root vm-id=vm_UUID

NOTE: you can change the shutdown option to either boot or retart to change the function you want.

xe help –all|more Show a list of XenServer CLI commands
• xsconsole Runs up the XenServer text based console
• xe-toolstack-restart Restarts the XenServer management tools
• ls –l List files in a Directory
• less /var/log/dmesg Display Boot Messages from Linux
• xe host-dmesg Xen Hypervisor Boot messages
• tail –f /var/log/xensource.log Look at xapi messages as they happen
• tail –f /var/log/xensource.log | grep xxx Look at xapi messages only for vm uuid xxx
• tail –f [log name] > [target filename] Send output to a file for analysis later
• cat /etc/xensource-inventory Display XenSource Inventory info
• xen-bugtool –yes Build a status report when xapi is down
• xe-backup-metadata -d –u [uuid of SR] Back up Pool metadata for all VMs
• tcpdump –i [inf] –vvv –w [filename] Get a Packet trace from [inf]. E.g. Inf=eth0, xenbr0, vif2.0 etc.
• top List the top processes running in Dom0
• xentop List top Xen processes
• mpstat 5 Processor stats in Dom0
• vmstat 2 Virtual memory in Dom0
• netstat –s Networking statistics
• iostat -d 2 6 Storage traffic stats
• list_domains Lists VMs that are running
• fdisk –l List the disk partitions
• hdparm –t /dev/sda3 Device read times for sda3 (normally local SR)
• pvs Show local and remote LVHD SRs
• ll /dev/disks/by-id Look at disk partitions
• lvs List logical volumes (virtual disks)
• vgs List LVM volume groups
• cd /var/run/sr-mount Look at NFS SRs
• df -h Shows how much disk space you have left
• dd if=/dev/sdb of=/dev/null iflag=direct bs=1M count=512 Read data performance from sdb.
• dd if=/dev/zero of=/dev/sdb oflag=direct bs=1M count=4096 Write performance on sdb. * Don’t use on disks with VMs on them!
• ifconfig Show info on NICs, virtual switches and vNICs
• brctl show Show info on virtual switches
• ethtool eth0 Info for NIC eth0
• mii-tool Info on NIC bonding
• iscsiadm -m discovery –type sendtargets –port 192.168.250.14 Discover iSCSI targets available to this server
• iscsiadm –m session Open iscsi sessions
• history Lists the history of commands you’ve used
• !136 Executes command #136 in the history
• history -w history-list.txt Writes the history info to a text file

Database move failed because content index in a failed state

1st restart Microsoft Exchange Search Indexer service on the node that has the active DB

2nd use Update-MailboxDatabaseCopy - catalogonly -identity dbname\servername on the node that has the passive DB

Add Permissions for Client Users to Access Public Folder Content

http://technet.microsoft.com/en-us/library/aa998834.aspx

When adding client permissions, you can either use predefined permission roles (which consist of specific access rights) or you can customize permissions by manually applying the available access rights. To specify the permissions for the client user, you can use the Add-PublicFolderClientPermission cmdlet or the AddUsersToPFRecursive.ps1 user management script.

Looking for other management tasks related to public folder permissions? Check out Managing Public Folder Permissions.

Note:
If a client user already has a specific access right to a public folder, you can't add the same access right again. Therefore, if you use the AddUsersToPFRecursive.ps1 script, and the user already has one of the access rights that you're trying to grant, a warning will appear stating that the current access rights will be removed before new access rights are granted.

Use the Shell to add public folder permissions for a client user

You need to be assigned permissions before you can perform this procedure. To see what permissions you need, see the "Public folder client permissions" entry in the Mailbox Permissions topic.

Note:
You can't use the EMC to add public folder permissions for a client user.

This example adds Publishing Editor permissions for the user Kim to access the public folder West Coast.

Add-PublicFolderClientPermission -Identity "\Marketing\West Coast" -AccessRights PublishingEditor -User Kim

For detailed syntax and parameter information, see Add-PublicFolderClientPermission.

Use the AddUsersToPFRecursive.ps1 script to add client access rights to a public folder

You need to be assigned permissions before you can perform this procedure. To see what permissions you need, see the "Public folder client permissions" entry in the Mailbox Permissions topic.

This example adds Reviewer permissions for the user David to access the top-level public folder Sales and all of the public folders under it.

AddUsersToPFRecursive.ps1 -TopPublicFolder "\Sales" -User "David" -Permission Reviewer

For more information about how to use public folder management scripts, see Scripts for Managing Public Folders in the Exchange Management Shell.

Powershell Commands With Exchange 2003

Get-WMIObject -namespace root\MicrosoftExchangeV2 -class Exchange_Mailbox -comput
er ExchServer | sort-object MailboxDisplayName | format-table MailboxDisplayName,Servername,StorageGroupName,StoreName,Siz
e,SentItems,TotalItems, DeletedMessageSizeExtended

vpxd.cfg Advanced Configuration

The vpxd.cfg is an XML formatted file which can be modified to alter the behaviour of the VMware vCenter Server.

The vpxd.cfg file is located by default at %ALLUSERPROFILE%\Application Data\VMware\VMware VirtualCenter\vpxd.cfg

• On Windows Server 2008, this would generally be C:\ProgramData\VMware\VMware VirtualCenter\vpxd.cfg
• On Windows Server 2003, this would generally be C:\Documents and Settings\All Users\Application Data\VMware\VMware VirtualCenter\vpxd.cfg

Remember to restart the VMware VirtualCenter Server service in the Server Manager for changes to vpxd.cfg to take effect.

Tag:  blockingTimeoutSeconds

Nested In:  vmomi, soapStubAdapter

What It Does:  Defines the timeout value in seconds for SOAP layer blocking.  Use cases for increasing: slow connections, low bandwidth, or high latency between virtual infrastructure components.  Read more here and here.

Example:

< vmomi> 

< soapStubAdapter> 

< blockingTimeoutSeconds> 10800< /blockingTimeoutSeconds> 

< /soapStubAdapter> 

< /vmomi> 

Tag:  calls

Nested In:  trace, vmomi

What It Does:  Unknown.  Read more here.

Example:

< trace> 

< vmomi> 

< calls> true< /calls> 

< /vmomi> 

< /trace> 

Tag:  cipherList

Nested In:  vmacore, ssl

What It Does:  Reverts to cipher suites used in previous versions of vCenter Server (2.5u3 and earlier) for browser/SSL compatibility issues.  Read more here.

Example:

< vmacore> 

< ssl> 

< cipherList> DEFAULT< /cipherList> 

< /ssl> 

< /vmacore> 

Tag:  compressOnRoll

Nested In:  log

What It Does:  Defines whether or not vCenter Server vpxd log files are rolled up and compressed into .gz files.  Read more here.

Example:

< log> 

< compressOnRoll> false< /compressOnRoll> 

< /log> 

Tag:  cpuFeatureMask

Nested In:  guestOSDescriptor, esx-2-x-x, all-versions, all-guests

What It Does:  Masks CPU features to force VMotion compatibility between hosts. VMware neither supports nor recommends modifying the VMotion constraints for CPU features.  Read more here.

Example:

< guestOSDescriptor> 

< esx-2-x-x> 

< all-versions> 

< all-guests> 

< cpuFeatureMask> Elements and mask definition go in here< /cpuFeatureMask> 

< /all-guests> 

< /all-versions> 

< /esx-2-x-x> 

< /guestOSDescriptor> 

Tag:  directory

Nested In:  log

What It Does:  Defines the location for the vCenter logs.  Read more here.

Example:

< log> 

< directory> D:\VC_Logs< /directory> 

< /log> 

Tag:  dontstartconsolidation

Nested In:  vcp2v

What It Does:  May resolve an issue where the Consolidation button is missing in the Virtual Infrastructure Client.  Read more here.

Example:

< vcp2v> 

< dontstartconsolidation> true< /dontstartconsolidation> 

< /vcp2v> 

Tag:  filterOverheadLimitIssues

Nested In:  vpxd

What It Does:  Unknown.

Example:

< vpxd> 

< filterOverheadLimitIssues> true< /filterOverheadLimitIssues> 

< /vpxd> 

Tag:  hostRescanFilter

Nested In:  unknown

What It Does:  Defines the behavior of mass ESX(i) host rescans of vmHBAs.  Read more here.

Example:

< hostRescanFilter> true< /hostRescanFilter> 

Tag:  IoMax

Nested In:  vmacore, threadpool

What It Does:  Unknown but my guess is it defines the maximum I/O for the vpxd.exe process (vCenter Server service). Influenced by TaskMax.

Example:

< vmacore> 

< threadpool> 

< IoMax> 200< /IoMax> 

< /threadpool> 

< /vmacore> 

Tag:  level

Nested In:  log

What It Does:  Defines the logging level for vCenter logs.  Read more here.

Example:

 < log> 

< level> trivia< /level> 

< /log> 

Tag:  logLevel

Nested In:  trace, vmomi

What It Does:  Enables debug logging level for vmomi?  Read more here.

Example:

< trace> 

< vmomi> 

< logLevel> verbose< /logLevel> 

< /vmomi> 

< /trace> 

Tag:  loglevel

Nested In:  nfc

What It Does:  Enables debug logging level for the NFC process.  Read more here.

Example:

< nfc> 

< loglevel> debug< /loglevel> 

< /nfc> 

Tag:  managedIP

Nested In:  unknown

What It Does:  Defines the managed IP address used in vCenter Server Heartbeat.  Read more here.

Example:

< managedIP> 10.10.0.1< /managedIP> 

Tag:  maxCostPerHost

Nested In:  ResourceManager

What It Does:  Defines the number of simultaneous VM migrations (both hot and cold) per ESX(i) host.  Read more here.

Example:

< ResourceManager> 

< maxCostPerHost> 8< /maxCostPerHost> 

< /ResourceManager> 

Tag:  maxFileNum

Nested In:  log

What It Does:  Defines the maximum number of log files for vCenter logs.  Read more here.

Example:

< log> 

< maxFileNum> 50< /maxFileNum> 

< /log> 

Tag:  maxFileSize

Nested In:  log

What It Does:  Defines the maximum log file size in Bytes and thus rollover interval for vCenter logs.  Read more here.

Example:

< log> 

< maxFileSize> 10485760< /maxFileSize> 

< /log> 

Tag:  name

Nested In:  log

What It Does:  Defines the log file prefix name for vCenter logs.  Read more here.

Example:

< log> 

< name> vpxd< /name> 

< /log> 

Tag:  notRespondingTimeout

Nested In:  heartbeat

What It Does:  Defines the heartbeat timeout value in seconds between ESX(i) hosts and vCenter Server.  Use case would be to increase the value if remote ESX(i) hosts frequently go into a not responding state in vCenter Server due to WAN bandwidth or latency issues.  Read more here.

Example:

< heartbeat> 

< notRespondingTimeout> 60< /notRespondingTimeout> 

< /heartbeat> 

Tag:  portReserveTimeout

Nested In:  dvs

What It Does:  Defines the timeout value in minutes for unused dvPort reservations.  Lowering the value temporarily is helpful for unlocking dvPorts to remove a vDS or dvPort group.  Read more here.

Example:

< dvs> 

< portReserveTimeout> 10< /portReserveTimeout> 

< /dvs> 

Tag:  serializeadds

Nested In:  vpxd, das

What It Does:  Unknown but if I had to guess I’d say it defines the behavior of how the HA agent is installed on cluster hosts.

Example:

< vpxd> 

< das> 

< serializeadds> true< /serializeadds> 

< /das> 

< /vpxd> 

Tag:  slotCpuMinMHz

Nested In:  vpxd, das

What It Does:  Defines the minimum CPU calculation of a HA cluster slot size when there are no CPU reservations. Read more here.

Example:

< vpxd> 

< das> 

< slotCpuMinMHz> 256< /slotCpuMinMHz> 

< /das> 

< /vpxd> 

Tag:  slotMemMinMB

Nested In:  vpxd, das

What It Does:  Defines the minimum memory calculation of a HA cluster slot size when there are no memory reservations. Read more here.

Example:

< vpxd> 

< das> 

< slotMemMinMB> 0< /slotMemMinMB> 

< /das> 

< /vpxd> 

Tag:  sspiProtocol

Nested In:  unknown

What It Does:  Defines the authentication mechanism used with passthrough authentication between the Virtual Infrastructure Client and vCenter Server.  Read more here.

Example:

< sspiProtocol> Kerberos< /sspiProtocol> 

Tag:  TaskMax

Nested In:  vmacore, threadpool

What It Does:  Defines the number of worker threads for the vpxd.exe process (vCenter Server service). Influences IoMax.

Example:

< vmacore> 

< threadpool> 

< TaskMax> 30< /TaskMax> 

< /threadpool> 

< /vmacore> 

Tag:  timeout

Nested In:  task

What It Does:  Defines the timeout value in seconds for long tasks.  Read more here.

Example:

< task> 

< timeout> 10800< /timeout> 

< /task> 

Tag:  verbose

Nested In:  trace, db

What It Does:  Enables database tracing.  Enables database logging in the vpxd log.  Read more here and here.

Example:

< trace> 

< db> 

< verbose> true< /verbose> 

< /db> 

< /trace> 

Tag:  verbosity

Nested In:  trace, vmomi

What It Does:  Unknown.  Read more here.

Example:

< trace> 

< vmomi> 

< verbosity> verbose< /verbosity> 

< /vmomi> 

< /trace> 

Tag:  verboseObjectSize

Nested In:  trace, vmomi

What It Does:  Unknown.  Read more here.

Example:

< trace> 

< vmomi> 

< verboseObjectSize> 40< /verboseObjectSize> 

< /vmomi> 

< /trace> 

Tag:  VMOnVirtualIntranet

Nested In:  migrate, test, CompatibleNetworks

What It Does:  Setting to false enables VMotion for VMs connected to an internal vSwitch. Setting to false will turn off the internal vSwitch restriction on VMotion events. Useful for servers behind a firewall virtual appliance deployed in bridged networking mode.  Read more here.

Example:

< migrate> 

< test> 

< CompatibleNetworks> 

< VMOnVirtualIntranet> false< /VMOnVirtualIntranet> 

< /CompatibleNetworks> 

< /test> 

< /migrate> 

Tag:  VMOverheadGrowthLimit

Nested In:  cluster

What It Does:  Defines the growth rate cap in terms of MB per minute for VM memory overhead at the cluster level. Can be adjusted to resolve high CPU utilization in guest VMs introduced in ESX(i) 3.5 and vCenter 2.5.  Read more here.

Example:

< cluster> 

< VMOverheadGrowthLimit> 5< /VMOverheadGrowthLimit> 

< /cluster> 

 

Slightly related, the vCenter Server process (vpxd.exe) can be launched at a command prompt on the vCenter Server (instead of starting as a service) for troubleshooting purposes.  The executable is located at:

< Install Directory> \VMware\Infrastructure\VirtualCenter Server> vpxd.exe

Usage: vpxd.exe [FLAGS]

Flags:

-r Register VMware VirtualCenter Server

-u Unregister VMware VirtualCenter Server

-s Run as a standalone server rather than a Service

-c Print vmdb schema to stdout

-b Recreate database repository

-f cfg Use the specified file instead of the default vpxd.cfg

-l licenseKey Store license key in ldap and assign it to VirtualCenter

-e feature Set the feature to be in use for VirtualCenter. This option takes only one feature at a time.

-p Reset the database password

-v Print the version number to stdout

Helpful ESX file paths

Check the following file (could be multiple secure logs followed with a .1, .2 etc) for logging in details in to the Service Console.

/var/log/secure

Check the following file if you like to see the iSCSI session, connection information.

/proc/scsi/vmkiscsi/1

If you are having VMKernel issues look in the following file at the exact same time that the error happen.

/var/log/vmkernel