XenServer - Lost interfaces

Please check out www.kaztechsolutions.co.uk for more of my technical posts, alternately please call us on 01932 268289. 

I recently ran in to an issue where in my XenServer test environment i seemed to lose all 4 of the NICs that was in my XenServer and the XenServer host could not be seen in XenCenter.

Looking at it closer when in XSCONSOLE and you looked at the management interface section you would get a message back saying "no interfaces present" and in the CLI if you were to run a xe pif-list you would get a message back saying "Host is still booting".

If you were to also run a ifconfig on the XenServer host would would even get back a list of all eth's, bond, xapi and xebr's.

I found out that running the xe pool-emergency-transition-to-master would resolve this issue. Once you have run this command wait around 30 seconds and then restart the host and it was now able to be seen within XenCenter.

Deleting Multiple VM's from XenCenter

First use excel (or your choice of word editor) and create a list of all your VM's you would like to delete plus the added XE command options

xe vm-uninstall vm=VM1 force=true
xe vm-uninstall vm=VM2 force=true
xe vm-uninstall vm=VM3 force=true
xe vm-uninstall vm=VM4 force=true
xe vm-uninstall vm=VM5 force=true
xe vm-uninstall vm=VM6 force=true
xe vm-uninstall vm=VM7 force=true
xe vm-uninstall vm=VM8 force=true
xe vm-uninstall vm=VM9 force=true

Next open up a putty session to a Xen Host and copy all of the above next directly in to the putty session.

NOTE: Please be aware that the VM names are CaSe Sensitive.

As the text is processed you'll see each VM being delete and you can also view the VM's being deleted in XenCenter as each line of command runs in the putty session.

[root@XENHOST ~]# xe vm-uninstall vm=VM1 force=true
The following items are about to be destroyed
VM : e995f197-5730-9cfc-3491-c2768e02236f (VM1)
All objects destroyed
[root@XENHOST ~]# xe vm-uninstall vm=VM2 force=true
The following items are about to be destroyed
VM : ec6d123d-a3d5-aa11-9439-f4c44927a0fa (VM2)
All objects destroyed

Obviously be double sure that you have the correct list of VM's before you run this command.

VM won't start and now is in a Stuck State

From time to time Vm's can get stuck in a state where you cannot start, shutdown, reboot or even force a reboot or shutdown. When this happens from the console type: xe-toolstack-restart (note that your XenCenter will loose connection when you do this. Nothing will however happen to your running VMs)

Reconnect with XenCenter (or let it reconnect by itself) and you should see your VM no longer in a stuck state

Change Pool Master via CLI

First disable high availability:
xe pool-ha-disable


Now list your XenServer hosts:
xe host-list


Using the list above, designate a new pool master by supplying the uuid associated with the desired host:
xe pool-designate-new-master host-uuid=


You’ll probably lose connection to the pool at this point, but that’s normal behavior. Once XenCenter reconnects (this should happen automatically), it’s safe to re-enable high availability:
xe pool-ha-enable

XenServer CLI - Commands

Please check out www.kaztechsolutions.co.uk for more of my technical posts, alternately please call us on 01932 268289. 

Since there is so many command within XenServer sometimes its hard to remember which command you require. This is just a list of regular commands that I use but please feel free to point out your favourite commands.

To find anything with the word "Server1" in messages logs. Obviously change the word to what you require but if you'r having a problem with a server named Server1 of instances you can enter the command below to filter out any information with Server1 in the logs.

grep Server1 /var/log/messages

To list your interfaces so you can identify the right UUID for your network device.

xe pif-list

To list full device details for the network device. TIP: once you have the UUID's of all of your interfaces simply enter the first couple of characters of the UUID interface and then press tab to fully populate the UUID.

xe pif-param-list uuid=(UUID of pif)

In XenServer 5 FP 1 you have the option to use vSwiches instead of the traditional bridging technology, the default install of XenServer does not have this enabled so you will have to run the following command to enable it. NOTE: you have to restart your XenServer after you have entered this command.

xe-switch-network-backend openvswitch


With XenServer it recommended that turn off auto-negotiation, set duplex to Full, and speed to 1000 using the Command Line Interface (CLI). To do this first find the UUID of the storage interface using the following command.

xe pif-list

Next set the parameters of the physical interface (PIF) using the following commands.

xe pif-param-set uuid= other-config:ethtool-autoneg=”off”
xe pif-param-set uuid= other-config:ethtool-speed=1000
xe pif-param-set uuid= other-config:ethtool-duplex=”full”

Restart the XAPI service because it controls networking on the XenServer, an alternative option is to reboot the XenServer. If you are in a pool, the above changes must be done on each storage PIF of the XenServers.

With XenServer its sometimes necessary to restart the iSCSI service on a XenServer, to do so run the following command from the CLI

service open-iscsi restart

Some times in XenCenter you'll get a VM that's in a "halted" state and you cannot even perform a force shutdown/reboot and even a xe-toolstack-restart will not shutdown the VM. To run a force restart from a CLI command run the following commands.

xe vm-shutdown -u root vm-name=vm_name
xe vm-shutdown -u root vm-id=vm_UUID

NOTE: you can change the shutdown option to either boot or retart to change the function you want.

xe help –all|more Show a list of XenServer CLI commands
• xsconsole Runs up the XenServer text based console
• xe-toolstack-restart Restarts the XenServer management tools
• ls –l List files in a Directory
• less /var/log/dmesg Display Boot Messages from Linux
• xe host-dmesg Xen Hypervisor Boot messages
• tail –f /var/log/xensource.log Look at xapi messages as they happen
• tail –f /var/log/xensource.log | grep xxx Look at xapi messages only for vm uuid xxx
• tail –f [log name] > [target filename] Send output to a file for analysis later
• cat /etc/xensource-inventory Display XenSource Inventory info
• xen-bugtool –yes Build a status report when xapi is down
• xe-backup-metadata -d –u [uuid of SR] Back up Pool metadata for all VMs
• tcpdump –i [inf] –vvv –w [filename] Get a Packet trace from [inf]. E.g. Inf=eth0, xenbr0, vif2.0 etc.
• top List the top processes running in Dom0
• xentop List top Xen processes
• mpstat 5 Processor stats in Dom0
• vmstat 2 Virtual memory in Dom0
• netstat –s Networking statistics
• iostat -d 2 6 Storage traffic stats
• list_domains Lists VMs that are running
• fdisk –l List the disk partitions
• hdparm –t /dev/sda3 Device read times for sda3 (normally local SR)
• pvs Show local and remote LVHD SRs
• ll /dev/disks/by-id Look at disk partitions
• lvs List logical volumes (virtual disks)
• vgs List LVM volume groups
• cd /var/run/sr-mount Look at NFS SRs
• df -h Shows how much disk space you have left
• dd if=/dev/sdb of=/dev/null iflag=direct bs=1M count=512 Read data performance from sdb.
• dd if=/dev/zero of=/dev/sdb oflag=direct bs=1M count=4096 Write performance on sdb. * Don’t use on disks with VMs on them!
• ifconfig Show info on NICs, virtual switches and vNICs
• brctl show Show info on virtual switches
• ethtool eth0 Info for NIC eth0
• mii-tool Info on NIC bonding
• iscsiadm -m discovery –type sendtargets –port 192.168.250.14 Discover iSCSI targets available to this server
• iscsiadm –m session Open iscsi sessions
• history Lists the history of commands you’ve used
• !136 Executes command #136 in the history
• history -w history-list.txt Writes the history info to a text file