Changing CPU and memory settings on XenServer VMs

You can change the CPU and memory settings of XenServer virtual machines using the API. XenServer allows you to set the priority of the virtual machine CPU (called weight), set a limit on the amount of CPU the virtual machine can use (called cap) and set the amount of memory the virtual machine gets.

CPU priority

You can set the weight parameter to a number between 1 and 65536. The default value is 256, meaning normal priority. A virtual machine with a weight of 512 will get twice as much CPU as a virtual machine with a weight of 256 on a contended XenServer host. The following code sets the weight to 128, meaning that it will get half as much CPU as a normal virtual machine.

Map<String, String> vcpuParameters = new HashMap<String, String>();
vcpuParameters.put("weight", "128");
vm.setVCPUsParams(connection, vcpuParameters);

CPU limit

You can set the cap parameter to a percentage number. The default value is 0, meaning there is no cap. The value 100 represents one virtual CPU, so half a virtual CPU can be set by using the value 50. The following code sets the cap to 35, meaning that it will not get more than 35 percent of one virtual CPU, even if the physical CPU is not used to its top.

Map<String, String> vcpuParameters = new HashMap<String, String>();
vcpuParameters.put("cap", "35");
vm.setVCPUsParams(connection, vcpuParameters);

Memory

You can set the amount of memory assigned to a virtual machine in bytes. The following code sets the amount of memory to 1 GB (1,073,741,824 bytes).

vm.setMemoryStaticMax(connection, new Long(1073741824));

Installing XenServer Tools on Ubuntu 10.04

XenServer supports a lot of Linux and Windows operating systems out of the box, but Ubuntu isn’t one of them. This means that running Ubuntu is slower, because it uses HVM (hardware-assisted virtualisation) instead of PV (paravirtualization). This article will explain how to create a paravirtualized Ubuntu 10.04 VM with the XenServer Tools installed. Many thanks go to the author of the article Ubuntu 10.04 LTS paravirtualised on Citrix XenServer.

We start with creating and installing a VM with HVM.

  • Create a VM from the Other install media template
  • Attach the Ubuntu 10.04 ISO to the DVD drive of the VM
  • Start the VM
  • At the disk partitioning stage make sure to replace Ext4 with Ext3, or create a specific boot partition with Ext3
  • Select at least the OpenSSH server package to be able to log in to the VM remotely
  • Finish installation and boot the VM

We now create a new console and edit the boot settings.

  • Connect to the VM using SSH
  • Create a new console for Xen
sudo cp /etc/init/tty1.conf /etc/init/hvc0.conf
sudo vi /etc/init/hvc0.conf
  • Replace all occurrences of tty1 with hvc0
  • Read /boot/grub/grub.cfg
sudo vi /boot/grub/grub.cfg
  • Copy the contents of menuentry near the bottom to a temporary text file, to be used as input for the makepv.sh script that we will use later on
menuentry 'Ubuntu, with Linux 2.6.32-21-server' --class ubuntu --class gnu-linux --class gnu --class os {
        recordfail
        insmod ext2
        set root='(hd0,1)'
        search --no-floppy --fs-uuid --set 02899ea9-1876-4e7b-8ef8-2b09b598cedb
        linux   /boot/vmlinuz-2.6.32-21-server root=UUID=02899ea9-1876-4e7b-8ef8-2b09b598cedb ro quiet
        initrd  /boot/initrd.img-2.6.32-21-server
}
  • Make special note of /boot/vmlinuz…, root=UUID=… and /boot/initrd.img
  • Shutdown the VM

We now convert the VM to PV.

  • Connect to the XenServer host with SSH
  • Copy the makepv.sh script (download from here) to the XenServer host and make it executable
chmod +x makepv.sh
  • Run the makepv.sh script, replacing my-vm-name with the actual name of your VM
makepv.sh my-vm-name

Finally we install the XenServer tools.

  • Boot the VM and log in with SSH
  • In XenCenter, attach xs-tools.iso to the DVD drive of the VM
  • Mount xs-tools.iso, install the correct XenServer Tools package (replace amd64 with i386 if necessary) and unmount xs-tools.iso
sudo mount /dev/cdrom1 /mnt
sudo dpkg -i /mnt/Linux/xe-guest-utilities_5.5.0-466_amd64.deb
sudo umount /mnt
  • In XenCenter, detach xs-tools.iso from the DVD drive of the VM (this ensures that XenServer does not complain about too many bootable devices)
  • Reboot the VM and log in with SSH
  • Make sure the services run at boot time
sudo update-rc.d -f xe-linux-distribution remove
sudo update-rc.d xe-linux-distribution defaults
  • Reboot the VM for the last time
  • Restart XenCenter to be able to log in to the console of the VM

Getting CPU, memory, disk and network metrics from XenServer

In a previous article we looked at getting CPU and memory metrics from XenServer. As noted in that article, as of version 5.5 of XenServer, the preferred way of getting virtual machine metrics is through HTTP calls to get RRD XML files. We showed how to revert to the old way of doing things, but in this article we will look into the new, preferred way. This has the added benefit that disk and network metrics are also available, along with a history of metrics.

The first thing to note is that the metrics are not available through the XenServer API. The RRD XML files are stored on the physical machines that currently host the virtual machines. For example, if you have two physical machines (P1 and P2) with two virtual machines running on each of them (V1a, V1b, V2a and V2b), you need to query P1 for the metrics of V1a and V1b and query P2 for the metrics of V2a and V2b. If a virtual machine is not running at the moment, you can get the (old) metrics from the master in the pool.

Each physical machine has an HTTP interface at the following location:

http://machine_name_or_ip/rrd_updates?start=1234567890

The parameter start tells the server to only give metrics starting at this timestamp (seconds since January 1st 1970). The call requires a username and password, which will be asked if you connect through a webbrowser. The following code does this in Java:

Connection connection = new Connection(new URL(MASTER_HOST));
Session.loginWithPassword(connection, USERNAME, PASSWORD, APIVersion.latest().toString());

Map hostRecords = Host.getAllRecords(connection);
for (Host host : hostRecords.keySet()) {
    URL url = new URL("http://" + host.getAddress(connection) + "/rrd_updates?start=" + (System.currentTimeMillis() / 1000 - TIME_WINDOW));
    URLConnection urlConnection = url.openConnection();
    String encoding = new BASE64Encoder().encode((USERNAME + ":" + PASSWORD).getBytes());
    urlConnection.setRequestProperty ("Authorization", "Basic " + encoding);

    String rrdXportData = IOUtils.toString(urlConnection.getInputStream());
}

The code needs two libraries:

  • commons-io-1.4.jar. This package contains the code to get a string containing the data of an inputstream.
  • xenserver-5.5.0-1.jar. This package contains the code to connect to the XenServer pool.

The following constants are used in this piece of code and need to be filled in:

  • MASTER_HOST. This is the IP address or hostname of the master host in the pool.
  • TIME_WINDOW. XenServer will return RRD updates of the last TIME_WINDOW seconds.
  • USERNAME and PASSWORD. These are the credentials for connecting to the master host and the other physical hosts in the pool.

At the end of this piece of code we have the string rrdXportData containing the RRD XML data. Now we need to parse this data to get the metrics we want. The following listing contains the most important parts of this string (edited for clarity):

<xport>
   <meta>
      <start>1273342925</start>
      <step>5</step>
      <end>1273342980</end>
      <rows>13</rows>
      <columns>31</columns>
      <legend>
         <entry>AVERAGE:vm:19ef51bd-2cbc-50d1-3fa2-ad8878699203:cpu0</entry>
         <entry>AVERAGE:vm:19ef51bd-2cbc-50d1-3fa2-ad8878699203:vif_4_tx</entry>
         ...
      </legend>
   </meta>
   <data>
      <row>
         <t>1273342980</t>
         <v>0.0002</v>
         <v>0.0</v>
         ...
      </row>
      <row>
         <t>1273342975</t>
         <v>0.0003</v>
         <v>0.0</v>
         ...
      </row>
      ...
   </data>
</xport>

The following Java code will parse this RRD XML string and make it available in the four variables shown at the top. The variable metricsTimelines contains a hashmap of string to double[]. The string is the metric name (e.g. AVERAGE:vm:19ef51bd-2cbc-50d1-3fa2-ad8878699203:cpu0) and the double[] contains the values for this metric from startTime to endTime with a step size of step. Note that the value at endTime is at position 0 and the value at  startTime is at the end of the array.

int endTime = 0;
HashMap<String, double[]> metricsTimelines = null;
int startTime = 0;
int step = 0;

try {
   DocumentBuilderFactory domFactory = DocumentBuilderFactory.newInstance();
   domFactory.setNamespaceAware(true);
   DocumentBuilder builder = domFactory.newDocumentBuilder();
   StringReader stringReader = new StringReader(rrdXportData);
   InputSource inputSource = new InputSource(stringReader);
   Document doc = builder.parse(inputSource);
   stringReader.close();

   ArrayList<ArrayList<String>> dataRows = new ArrayList<ArrayList<String>>();
   ArrayList<String> legends = new ArrayList<String>();
   NodeList xportChildNodes = doc.getDocumentElement().getChildNodes();
   for (int i = 0; i < xportChildNodes.getLength(); i++) {
      Node xportChildNode = xportChildNodes.item(i);
      if (xportChildNode.getNodeName().equals("meta")) {
         NodeList metaChildNodes = xportChildNode.getChildNodes();
         for (int j = 0; j < metaChildNodes.getLength(); j++) {
            Node metaChildNode = metaChildNodes.item(j);
            if (metaChildNode.getNodeName().equals("step")) {
               step = new Integer(metaChildNode.getTextContent()).intValue();
            } else if (metaChildNode.getNodeName().equals("start")) {
               startTime = new Integer(metaChildNode.getTextContent()).intValue();
            } else if (metaChildNode.getNodeName().equals("end")) {
               endTime = new Integer(metaChildNode.getTextContent()).intValue();
            } else if (metaChildNode.getNodeName().equals("legend")) {
               NodeList legendChildNodes = metaChildNode.getChildNodes();
               for (int k = 0; k < legendChildNodes.getLength(); k++) {
                  Node legendChildNode = legendChildNodes.item(k);
                  legends.add(k, legendChildNode.getTextContent());
               }
            }
         }
      } else if (xportChildNode.getNodeName().equals("data")) {
         NodeList dataChildNodes = xportChildNode.getChildNodes();
         for (int j = 0; j < dataChildNodes.getLength(); j++) {
            Node rowNode = dataChildNodes.item(j);
            NodeList rowChildNodes = rowNode.getChildNodes();
            ArrayList<String> dataRow = new ArrayList<String>();
            for (int k = 1; k < rowChildNodes.getLength(); k++) {
               Node rowChildNode = rowChildNodes.item(k);
               dataRow.add(k - 1, rowChildNode.getTextContent());
            }
            dataRows.add(dataRow);
         }
      }
   }

   int nrDataRows = dataRows.size();
   int nrLegends = legends.size();

   metricsTimelines = new HashMap<String, double[]>();
   for (int i = 0; i < nrLegends; i++) {
      metricsTimelines.put(legends.get(i), new double[nrDataRows]);
   }
   for (int i = 0; i < nrLegends; i++) {
      for (int j = 0; j < nrDataRows; j++) {
         double[] values = metricsTimelines.get(legends.get(i));
         values[j] = new Double(dataRows.get(j).get(i)).doubleValue();
      }
   }
} catch (Exception e) {
   e.printStackTrace();
}

Using XenServer in a routed IP network


Hetzner is a hosting company in Germany where you can rent dedicated root servers per month. Using their KVM-over-IP setup, it is possible to install operating systems that are not available by default, such as XenServer by Citrix. With the machine, you get four public IP addresses. One of these addresses is given to you when the machine is assigned to you, the other three are assigned to you on request.

There is a problem with the public IP addresses Hetzner gives you. Within the Hetzner network, there is a hard connection between the MAC address of your machine and the IP addresses you have been given. This is a problem for virtual machines that are bridged onto the network by the host machine. These virtual machines have their own MAC address and the Hetzner network will drop packets from these unknown MAC addresses.

There is a solution to this problem: have the host machine route IP packets from the virtual machines to the network and vice versa. We will use the following addresses in the example below:

IP addresses 188.40.109.204 (host), 188.40.109.250 (VM)
Netmask 255.255.255.192
Gateway 188.40.109.193 (host)
DNS 213.133.100.100

We start with the configuration of the host machine (running XenServer). The first file is the configuration file of eth0: /etc/sysconfig/network-scripts/ifcfg-eth0

XEMANAGED=yes
DEVICE=eth0
ONBOOT=no
TYPE=Ethernet
HWADDR=40:61:86:be:ce:88 (replace with MAC address of host)
BRIDGE=xenbr0

Notice that this file does not contain any IP configuration. The second file is the configuration file of xenbr0: /etc/sysconfig/network-scripts/ifcfg-xenbr0

XEMANAGED=yes
DEVICE=xenbr0
ONBOOT=no
TYPE=Bridge
DELAY=0
STP=off
PIFDEV=eth0
BOOTPROTO=none
IPADDR=188.40.109.204 (replace with IP address of host)
NETMASK=255.255.255.192
GATEWAY=188.40.109.193 (replace with gateway of host)
DNS1=213.133.100.100
DNS2=213.133.99.99
DNS3=213.133.98.98

Now we need to enable IP forwarding on the host machine. We start with the sysctl configuration file: /etc/sysctl.conf

net.ipv4.ip_forward=1
net.ipv4.conf.all.send_redirects=0
net.ipv4.conf.default.send_redirects=0
net.ipv4.conf.lo.send_redirects=0
net.ipv4.conf.xenbr0.send_redirects=0

The first line tells the machine to perform IP forwarding. The four lines after that tell the machine to disable sending ICMP redirects. The last file we need to edit is the firewall configuration file: /etc/sysconfig/iptables. Add this text below the line -A RH-Firewall-1-INPUT -i lo -j ACCEPT:

-A RH-Firewall-1-INPUT -i xenbr0 -o xenbr0 -j ACCEPT

Now reboot the machine and continue with the IP configuration of the virtual machine running on XenServer. There is only one file we need to edit here, which is the configuration file of eth0: /etc/sysconfig/network-scripts/ifcfg-eth0

DEVICE=eth0
BOOTPROTO=none
ONBOOT=yes
HWADDR=8e:35:1e:3b:12:aa (replace with MAC address of VM)
IPADDR=188.40.109.250 (replace with IP address of VM)
NETMASK=255.255.255.192
GATEWAY=188.40.109.204 (replace with IP address of host)

The gateway in this configuration is crucial: it needs to be the IP address of the host itself, not the gateway of the host.

Getting XenServer VM metrics in Java

XenServer is a product of Citrix that can run virtual machines on a set of physical hosts called a pool. There is an API that allows you to create programs that interact with the XenServer pool. There are some function calls that return metrics of the virtual machines and the physical hosts.

Let us have a look at retrieving the CPU load of a virtual machine (VM). This can be accomplished with the following API call (assuming that vm is of type com.xensource.xenapi.VM and connection is of type com.xensource.xenapi.Connection):

vm.getMetrics(connection).getVCPUsUtilisation(connection)

However, as many have noted, the answer is always 0.0. As of XenServer version 5.5, the way these metrics are stored has changed. The preferred way to get VM metrics now, is to make a direct HTTP call to the physical hosts. This call will return RRD XML files. This makes things more difficult because now there are multiple calls to make, one to the XenServer API and one call per physical host machine. And one needs to dig into these files to get the CPU load, instead of having the value returned in a single call.

There is a way to make the XenServer API behave like it did in version 5.0 and prior. We start by logging into the console of a physical machine and entering:

xe host-list

Note the UUID values of all the physical host machines. Now enter the following command for each UUID:

xe host-param-set uuid=<uuid found in previous command> other-config:rrd_update_interval=1

Now reboot all physical host machines and the VM metric command will work like a charm.

Top cloud computing providers

Although there are lots of different letters you can put in front of “as a service”, there are typically three that everybody agrees about. The following picture sums these up:

In this article we will look into some of the top providers in these three areas.

Infrastructure as a Service (IaaS) providers

Amazon

One of the biggest and most well known in this category is Amazon with its Elastic Compute Cloud (EC2). It of course started out as a company selling books, CD’s and DVD’s through the internet, but they realized that a lot of their computing resources were doing nothing most of the time, just waiting for the holiday season to hit its peak. Renting this capacity to others through the internet turned out to be a very good business model. Their current portfolio is really big and diverse, which makes it a bit daunting to start using it.

Rackspace

Another big player in this area is Rackspace. They started out as a regular hosting provider, renting out space in their racks, providing managed hosting and dedicated servers. In the last year they have moved into the cloud business as well and are doing just fine. One of the things might just be that their offering is more understandable than Amazon’s and their lowest prices are a lot better as well.

Joyent

Third on our list is Joyent. This company has been delivering cloud computing services even before the term was there to describe it. Their portfolio is quite big, and it is not limited to IaaS either. They do seem to lack in public relations, because they are not as well known as the first two IaaS players.

GoGrid

Fourth on our list is GoGrid. Like the other IaaS providers, GoGrid offers virtual machines to be rented by the hour, but they also offer dedicated servers for specific tasks that tend to be slow on virtual machines, such as databases. They call this setup a hybrid cloud, but this seems to be confusing because this term is also used for a mixture of a private and a public cloud.

Terremark

The final name on our list of IaaS providers is Terremark. They target two different types of users: the type that wants to have resources available quickly without a lot of hassle, and the type that wants to have a solid and secure environment that can run enterprise applications. For the second type of user, they provide dedicated resource pools and integration with private networks.

Platform as a Service (PaaS) providers

Google

The company with probably the biggest cloud of its own, also provides cloud services in the form of PaaS. Google App Engine is a framework that allows you to build software in Python or Java that is automatically distributed and scaled as necessary on the cloud. If you have the luxury of writing new applications in one of these two languages and want to adhere to the Google way of programming, this is one to take a look at.

Microsoft

Another really big name in this area is Microsoft. With their Azure platform, they want to give Visual Studio programmers the ability to build software for the cloud as easily as for real servers in the datacenter. There are currently two types of cloud service: Windows Azure, the operating system as an online service, and SQL Azure, a fully relational cloud database solution. If you are familiar with the Microsoft development tools and want to take the cloud for a spin, this is where yo start.

Force.com

Last on our list of PaaS providers is force.com. Created by salesforce.com, the company that grew big by providing software on demand. They started out with subscription fees, but are moving to a pay-as-you-go model. Force.com is a development platform that allows users to build business applications in the cloud. If you are willing to create software that only works in the force.com cloud, this is a simple start into cloud computing.

Software as a Service (SaaS) providers

Google

First on our list is Google Apps, not because they have the widest range of applications, but because they are probably the best known for their web-based offerings. These include of course their e-mail service, calendar, document editor, spreadsheet, and some others.

Salesforce.com

Second on our list is salesforce.com. They provide CRM software on-demand and started out with subscription fees, but are moving to a pay-as-you-go model. Currently they offer two SaaS services: Sales cloud and Service cloud. The first contains accounts, contacts,  leads, quotes, etc. The second contains a customer portal, knowledge base, analytics, etc.

Zoho

Last on our list is zoho.com. This company might be less known than Google and salesforce.com, but they offer much more types of applications that you can try out for free. These include an e-mail service, document editor, presentation tool, invoicing, reporting, applicant tracking, and many more. You only start paying for these services when you want to use the more advanced features.

Using a Xen virtual machine image with Eucalyptus

eucalyptus
In the previous article we created a CentOS virtual machine image that was usable with Xen. In this short article we will use this image in Eucalyptus.

Bundle, upload and register the kernel with Eucalyptus:

ec2-bundle-image --image /boot/vmlinuz-2.6.18-128.1.6.el5xen --kernel true
ec2-upload-bundle --bucket centos-kernel-bucket --manifest /tmp/vmlinuz-2.6.18-128.1.6.el5xen.manifest.xml
ec2-register centos-kernel-bucket/vmlinuz-2.6.18-128.1.6.el5xen.manifest.xml

Bundle, upload and register the ramdisk with Eucalyptus:

ec2-bundle-image --image /root/centos-ramdisk.img --ramdisk true
ec2-upload-bundle --bucket centos-ramdisk-bucket --manifest /tmp/centos-ramdisk.img.manifest.xml
ec2-register centos-ramdisk-bucket/centos-ramdisk.img.manifest.xml

Bundle, upload and register the filesystem with Eucalyptus:

ec2-bundle-image --image /root/centos-root.img
ec2-upload-bundle --bucket centos-root-bucket --manifest /tmp/centos-root.img.manifest.xml
ec2-register centos-root-bucket/centos-root.img.manifest.xml

Start an instance, replacing the emi-, eki- and eri-identifiers with your own:

ec2-run-instances emi-F4CB118E --kernel eki-38AC43DC --ramdisk eri-98FE2101

Creating a CentOS Xen virtual machine image

xen
In this article we will create a CentOS 5.2 virtual machine image that can be used by Xen. We need to bring three pieces together:

  1. The kernel (vmlinuz)
  2. The ramdisk (initrd)
  3. The filesystem

Kernel

The easiest way to create a suitable kernel, ramdisk and filesystem is to use a system with CentOS 5.2 on it with xen enabled:

yum install xen

Change the default kernel to the new kernel with xen support by editing /boot/grub/menu.lst:

default=0

and reboot

Ramdisk

Create the ramdisk by running mkinitrd:

mkinitrd --omit-scsi-modules --with=xennet --with=xenblk --preload=xenblk /root/xen-image/centos-ramdisk.img

Filesystem

The hardest job is getting a filesystem. Create a directory where we will hold the filesystem image:

mkdir /root/xen-image
cd /root/xen-image

Create an image file and make a filesystem on this file:

dd if=/dev/zero of=centos-root.img bs=1M count=1999
mkfs.ext3 centos-root.img

Create a directory where we can mount the newly created image:

mkdir rootdisk
mount -o loop centos-root.img /root/xen-image/rootdisk/
cd rootdisk

Create the /etc directory:

mkdir etc

and edit the file /etc/fstab:

/dev/sda1   /          ext3     defaults         1 1
none        /dev/pts   devpts   gid=5,mode=620   0 0
none        /dev/shm   tmpfs    defaults         0 0
none        /proc      proc     defaults         0 0
none        /sys       sysfs    defaults         0 0

Create some necessary device files:

mkdir dev
for i in console null zero; do /sbin/MAKEDEV -d /root/xen-image/rootdisk/dev -x $i; done

Copy the kernel modules:

mkdir -p lib/modules
cp -a /lib/modules/2.6.18-128.1.6.el5xen/ /root/xen-image/rootdisk/lib/modules/

Create the directory for network scripts:

mkdir -p etc/sysconfig/network-scripts

Edit the file that describes the first network interface, /etc/sysconfig/network-scripts/ifcfg-eth0:

DEVICE=eth0
BOOTPROTO=dhcp
ONBOOT=yes

Do the same for the second network interface, /etc/sysconfig/network-scripts/ifcfg-eth1:

DEVICE=eth1
BOOTPROTO=dhcp
ONBOOT=yes

And the last network script, etc/sysconfig/network:

NETWORKING=yes
HOSTNAME=centos52
GATEWAY=x.x.x.x

Create the RPM lock directory:

mkdir -p var/lock/rpm

We need to create a specific configuration file for yum, /root/xen-image/yum-xen.conf:

[main]
cachedir=/var/cache/yum
keepcache=0
debuglevel=2
logfile=/var/log/yum.log
distroverpkg=redhat-release
tolerant=1
exactarch=1
obsoletes=1
gpgcheck=0
plugins=1
metadata_expire=1h

[base]
name=CentOS-5.2 - Base
baseurl=http://mirror.centos.org/centos/5.2/os/i386/
enabled=1

[extras]
name=CentOS-5.2 - Extras
baseurl=http://mirror.centos.org/centos/5.2/extras/i386/
enabled=1

[updates]
name=CentOS-5.2 - Updates
baseurl=http://mirror.centos.org/centos/5.2/updates/i386/
enabled=1

[addons]
name=CentOS-5.2 - Addons
baseurl=http://mirror.centos.org/centos/5.2/addons/i386/
enabled=1

Use the configuration file we just edited to install the base system and add the openssh server:

yum -c /root/xen-image/yum-xen.conf --installroot=/root/xen-image/rootdisk -y groupinstall base
yum -c /root/xen-image/yum-xen.conf --installroot=/root/xen-image/rootdisk -y install openssh openssh-server

To set an initial password for root, we chroot into the rootdisk we created:

chroot /root/xen-image/rootdisk

Edit the /etc/passwd file inside the chroot and change the ‘*’ on the first line with an ‘x’:

root:x:0:0:root:/root:/bin/bash

Run pwconv to enable shadow passwords and set the root password:

pwconv
passwd root

Disable TLS:

mv /lib/tls /lib/tls.disabled

Exit from the chroot:

exit

Unmount the rootdisk:

cd /root
umount /root/xen-image/rootdisk

Putting it all together

Create a new configuration file for Xen specific to this image, /etc/xen/centos52:

kernel = "/boot/vmlinuz-2.6.18-128.1.6.el5xen"
ramdisk = "/root/xen-image/centos-ramdisk.img"
name = "centos52"
memory = "256"
disk = [ 'file:/root/xen-image/centos-root.img,sda1,w' ]
root = '/dev/sda1 ro'
vif = [ 'bridge=xenbr0', '']
vcpus=1
on_reboot = 'destroy'
on_crash = 'destroy'

And finally, starting the Xen VM image:

/usr/sbin/xm create -c centos52

Fetching meta-data and user-data in Eucalyptus

eucalyptus
Eucalyptus is a cloud computing platform that is intended to be API compatible with Amazon’s EC2, but can be installed on your own machines. In the latest stable version available at the time of writing, version 1.4, there is a catch in fetching meta-data and user-data. According to Amazon’s API description, you can get the meta-data by doing an HTTP GET on the URL:

http://169.254.169.254/latest/meta-data

However, this doesn’t work in Eucalyptus 1.4. The correct host to fetch this data from is the gateway for the running instance and the port is 8773. This little script will print the correct host and port:

echo http://$(route -n | awk '$4 ~ ".*G.*" {print $2}'):8773/

Meta-data

You can fetch a list of meta-data keys by getting /latest/meta-data. Fetching the value for a key is as easy as getting /latest/meta-data/key. The following table shows an example of the keys and associated values:

local-ipv4 10.0.2.3
reservation-id r-412F078F
local-hostname 10.0.2.3
security-groups [default]
product-codes not yet supported.
ami-launch-index 0
public-hostname 192.168.1.3
hostname 192.168.1.3
public-ipv4 192.168.1.3
ramdisk-id emi-F4CB118E
kernel-id eki-90591388
ami-id emi-7C83135D
placement/ availability-zone
public-keys/ 0=admin_key
ancestor-ami-ids none
block-device-mapping not yet supported.
ami-manifest-path http://cloud1.example.org:8773/services/Walrus/ubuntu804-bucket/ubuntu.8-04.img.manifest.xml
instance-type m1.small
instance-id i-2ECA052C

User-data

You can supply user-data as an option to ec2-run-instances:

ec2-run-instances <emi> -k <key> -d <user-data>

This data is fetched by getting /latest/user-data.

Installing Eucalyptus on CentOS

eucalyptus
Setup

Eucalyptus is software that enables you to run your own cloud. It aims to be API compatible with Amazon’s EC2, which means you can use most of the tools that are written for that system with Eucalyptus too.

This articles describes my successful attempt at installing Eucalyptus 1.4 on CentOS 5.2 with managed network mode. The installation is performed on two different types of machines:

  • The front-end, which is the machine controlling the cloud
  • The compute nodes, which run the virtual machines

Front-end

eucalyptus-head-node

The front-end has two network interfaces. One (eth0) is connected to the compute nodes and has IP address 192.168.1.254. The other (eth1) is connected to the LAN and has IP address 172.16.0.254.

We start by disabling SELinux. This is accomplished by editing the file /etc/selinux/config:

SELINUX=disabled

Now reboot for this change to take effect.

Allow the machine to forward IP packets by editing /etc/sysctl.conf:

net.ipv4.ip_forward = 1

and change the value immediately without rebooting:

sysctl -p /etc/sysctl.conf

Start with a clean iptables firewall and allow NAT:

iptables --flush
iptables --table nat --flush
iptables --delete-chain
iptables --table nat --delete-chain
iptables --table nat --append POSTROUTING --out-interface eth1 -j MASQUERADE
iptables --append FORWARD --in-interface eth0 -j ACCEPT

Make the changes permanent by running:

/etc/init.d/iptables save

Download the Sun Java Development Kit version 6 and install it:

chmod +x jdk-6u13-linux-i586-rpm.bin
./jdk-6u13-linux-i586-rpm.bin

Download Apache ANT and install it:

cd /opt
tar -zxvf apache-ant-1.7.1-bin.tar.gz

Download the EC2 tools and install them:

cd /opt
unzip /root/ec2-ami-tools-1.3-26357.zip
unzip /root/ec2-api-tools-1.3-30349.zip

Some environment variables need to be present for Eucalyptus to work. Edit the file /etc/profile and add the following:

export JAVA_HOME=/usr/java/jdk1.6.0_13
export EC2_HOME=/opt/ec2-api-tools-1.3-30349
export EC2_AMITOOL_HOME=/opt/ec2-ami-tools-1.3-26357
export PATH=$PATH:/opt/apache-ant-1.7.1/bin:$EC2_HOME/bin:$EC2_AMITOOL_HOME/bin

Install some dependencies:

yum install dhcp xen-libs bridge-utils

Download the Eucalyptus files and extract the RPM dependencies file:

tar -zxvf eucalyptus-rpm-deps-i386.tar.gz

Install the Eucalyptus RPMs:

rpm -Uvh euca-axis2c-1.4-1.i386.rpm \
         euca-httpd-1.4-1.i386.rpm \
         euca-libvirt-1.4-1.i386.rpm \
         eucalyptus-1.4-2.i386.rpm \
         eucalyptus-cloud-1.4-2.i386.rpm \
         eucalyptus-gl-1.4-2.i386.rpm \
         eucalyptus-cc-1.4-2.i386.rpm

The configuration file /opt/eucalyptus/etc/eucalyptus/eucalyptus.conf contains the following for our setup:

EUCALYPTUS="/opt/eucalyptus"
START_CLOUD="Y"
START_CC="Y"
START_NC="N"
ENABLE_WS_SECURITY="Y"
LOGLEVEL="DEBUG"
CLOUD_PORT="8773"
CLOUD_SSL_PORT="8443"
CC_PORT="8774"
SCHEDPOLICY="GREEDY"
NODES="192.168.1.1 192.168.1.2 192.168.1.3"
NC_SERVICE="axis2/services/EucalyptusNC"
NC_PORT="8775"
VNET_INTERFACE="eth0"
VNET_DHCPDAEMON="/usr/sbin/dhcpd"
VNET_MODE="MANAGED"
VNET_SUBNET="10.0.0.0"
VNET_NETMASK="255.0.0.0"
VNET_DNS="172.16.0.1"
VNET_ADDRSPERNET="64"
VNET_PUBLICIPS="172.16.0.11 172.16.0.12 172.16.0.13 172.16.0.14"

Compute nodes

eucalyptus-compute-node

The compute nodes have one network interface (eth0) which is connected to the front-end and they have IP addresses ranging from 192.168.1.1 to 192.168.1.3 (for three compute nodes).

We start by disabling SELinux. This is accomplished by editing the file /etc/selinux/config:

SELINUX=disabled

Now reboot for this change to take effect.

Start with a clean iptables firewall:

iptables --flush
iptables --table nat --flush
iptables --delete-chain
iptables --table nat --delete-chain

Make the changes permanent by running:

/etc/init.d/iptables save

Install Xen:

yum install xen

Make sure the correct kernel with xen enabled is started at boot by editing the file /boot/grub/menu.lst:

default=0

And reboot.

Download the Eucalyptus files and extract the RPM dependencies file:

tar -zxvf eucalyptus-rpm-deps-i386.tar.gz

Install the Eucalyptus RPMs:

rpm -Uvh euca-axis2c-1.4-1.i386.rpm \
euca-libvirt-1.4-1.i386.rpm \
euca-httpd-1.4-1.i386.rpm \
eucalyptus-1.4-2.i386.rpm \
eucalyptus-gl-1.4-2.i386.rpm \
eucalyptus-nc-1.4-2.i386.rpm

Make a directory for holding the running instances:

mkdir -p /usr/local/instances

The configuration file /opt/eucalyptus/etc/eucalyptus/eucalyptus.conf contains the following for our setup:

EUCALYPTUS="/opt/eucalyptus"
START_CLOUD="N"
START_CC="N"
START_NC="Y"
ENABLE_WS_SECURITY="Y"
LOGLEVEL="DEBUG"
NC_PORT="8775"
INSTANCE_PATH="/usr/local/instances"
VNET_INTERFACE="peth0"
VNET_BRIDGE="xenbr0"
VNET_MODE="MANAGED"

Start Eucalyptus by running:

/etc/init.d/eucalyptus start

Back to the front-end

Start Eucalyptus by running:

/etc/init.d/eucalyptus start

If all went according to plan, the following website should be viewable: https://172.16.0.254:8443. Login with username admin and password admin. Add a cluster with a name you like, e.g. my_cluster, and the IP address of the front-end, in our case 172.16.0.254.

Now download the x509 certificate to be able to connect to the Eucalyptus setup. Unzip the contents of this file:

mkdir /root/.euca
cd /root/.euca
unzip /root/euca2-admin-x509.zip

There is a file containing aliases for several commands you run that needs to be sourced every time you login. It is easier to do this automatically, so edit the file ~/.bash_profile and add this line to the end:

source /root/.euca/eucarc

Synchronise the keys between all the hosts:

/opt/eucalyptus/usr/sbin/euca_sync_key -c /opt/eucalyptus/etc/eucalyptus/eucalyptus.conf

Install ruby to be able to run the EC2 command line tools:

yum install ruby

You can now see if the EC2 command line tools and Eucalyptus are working:

ec2-describe-availability-zones verbose

This should give you an overview of the running system, with a listing of the compute nodes and the capacity of them in terms of number of running instances they can hold.

Extract the example VM image from Eucalyptus:

tar -zxvf euca-ttylinux.tgz

Bundle the image, upload it and register it:

ec2-bundle-image --image /root/ttylinux/vmlinuz-2.6.16.33-xen --kernel true
ec2-upload-bundle --bucket kernel-bucket --manifest /tmp/vmlinuz-2.6.16.33-xen.manifest.xml
ec2-register kernel-bucket/vmlinuz-2.6.16.33-xen.manifest.xml

ec2-bundle-image --image /root/ttylinux/ttylinux.img
ec2-upload-bundle --bucket image-bucket --manifest /tmp/ttylinux.img.manifest.xml
ec2-register image-bucket/ttylinux.img.manifest.xml

If all went well, you can list the images just uploaded:

ec2-describe-images

Add a keypair for the current user to Eucalyptus:

ec2-add-keypair admin_key > /root/admin_key.private
chmod 0600 /root/admin_key.private

Run a VM (change emi-F4CB118E with the emi-identifier returned by ec2-describe-images):

ec2-run-instances emi-F4CB118E -k admin_key

Get a list of instances:

ec2-describe-instances

To see the console output of the running instance (change i-3F170798 with the instance-identifier returned by ec2-describe-instances):

ec2-get-console-output i-3F170798