Author Archives: jansipke

Creating backups of running VMs in XenServer

By jansipke on 2010-12-06

With XenServer it is possible to create backups of VMs, even if they are running. The process is as follows:

  • Search for the uuid of the VMs to backup
  • Create a snapshot of each (running) VM
  • Save the snapshot to file
  • Remove the created snapshot

First look for the uuid of the VMs to backup. We don’t want to backup the control domain itself, so we add is-control-domain=false to the vm-list command:

xe vm-list is-control-domain=false

Now we create a snapshot of the VMs we want to backup, replacing the uuid one by one with the ones we found with the previous command. Also replace the name of the snapshot if desired:

xe vm-snapshot uuid=d61bfc1a-33b2-5406-7ea5-76e4f7113220 new-name-label=snapshotname

This command has a return value: the uuid of the created snapshot. Then we transform the snapshot into a VM to be able to save it to a file, replacing uuid with the return value of the previous command:

xe template-param-set is-a-template=false ha-always-run=false uuid=b759625c-eab5-4e0f-be5e-a05bcbad869a

In the next step we save the snapshot to a file, replacing uuid with the snapshot uuid and providing a meaningful filename:

xe vm-export vm=b759625c-eab5-4e0f-be5e-a05bcbad869a filename=filename.xva

In the final step we delete the snapshot:

xe vm-uninstall uuid=b759625c-eab5-4e0f-be5e-a05bcbad869a force=true

Python is installed by default on XenServer hosts, so the following script will work out of the box. Download the script from this location to save it to your XenServer host, replacing the .txt extension with .py.

#!/usr/bin/python

import commands, time

def get_backup_vms():
   result = []

   cmd = "xe vm-list is-control-domain=false"
   output = commands.getoutput(cmd)

   for vm in output.split("nnn"):
      lines = vm.split("n")
      uuid = lines[0].split(":")[1][1:]
      name = lines[1].split(":")[1][1:]
      result += [(uuid, name)]

   return result

def backup_vm(uuid, filename, timestamp):
   cmd = "xe vm-snapshot uuid=" + uuid + " new-name-label=" + timestamp
   snapshot_uuid = commands.getoutput(cmd)

   cmd = "xe template-param-set is-a-template=false ha-always-run=false uuid=" +
   snapshot_uuid
   commands.getoutput(cmd)

   filename = filename.replace(" ", " ")
   filename = filename.replace("(", "(")
   filename = filename.replace(")", ")")
   cmd = "xe vm-export vm=" + snapshot_uuid + " filename=" + filename
   commands.getoutput(cmd)

   cmd = "xe vm-uninstall uuid=" + snapshot_uuid + " force=true"
   commands.getoutput(cmd)

for (uuid, name) in get_backup_vms():
   timestamp = time.strftime("%Y%m%d-%H%M", time.gmtime())
   print timestamp, uuid, name
   filename = timestamp + " " + name + ".xva"
   backup_vm(uuid, filename, timestamp)

Posted in Tagged 4 Comments

Assigning more than 8 virtual CPUs in XenServer

By jansipke on 2010-11-30


With the XenServer Windows client called XenCenter it is possible to manage multiple XenServer hosts, create virtual machines, watch the console of virtual machines, etc. One of the limitations I recently ran into (this is a luxury problem, I know) is that it doesn’t allow you to assign more than 8 virtual CPUs (vCPUs) to a single VM.

However, on the command line you can increase this value to 32 for Linux VMs. First, let’s see which uuid is assigned to the VM we want to give more than 8 vCPUs:

xe vm-list

Now make sure that the VM is halted and then use this uuid in the following commands:

xe vm-param-set VCPUs-max=32 uuid=replace_with_uuid
xe vm-param-set VCPUs-at-startup=32 uuid=replace_with_uuid

If you want to increase the memory size beyond 16GB as well, e.g. to 32GB, add the following commands:

xe vm-param-set memory-static-max=34359738368 uuid=replace_with_uuid
xe vm-param-set memory-dynamic-max=34359738368 uuid=replace_with_uuid
xe vm-param-set memory-dynamic-min=34359738368 uuid=replace_with_uuid
xe vm-param-set memory-static-min=34359738368 uuid=replace_with_uuid

Restart the VM and enjoy! For a quick look if everything worked out, watch the output of this command inside the VM:

cat /proc/cpuinfo | grep processor | wc -l

Posted in Tagged 3 Comments

Temperature graphs

By jansipke on 2010-11-24

In a previous article we looked at remote measurement of temperature and humidity. Now it is time to show the results of the temperature measurements. First a recap of the setup:

Three remote temperature and humidity sensors are located in the living room, the garage and outside. The following graph shows the output of about 1 week:

As you can see, the garage is not isolated very well. The temperature in the garage is quite similar to the temperature outside. If the weather predictions are correct, we will see temperatures below 0 degrees centigrade soon. Finally a test for the parsing of negative values!

Luckily, the isolation of the living room is much better than that of the garage. During the night the temperature drops only 2 degrees from 20 to 18, even though the thermostat is programmed to keep the temperature above 15 degrees. The spikes to about 25 degrees are a result of the placement of the sensor in the living room: early in the afternoon, the sun sometimes shines directly on it.

Posted in Tagged , 1 Comment

Versioning of tables in MySQL

By jansipke on 2010-11-22


In this article we will see how we can create tables in MySQL in such a way that several versions of the same table are stored in the database and we can choose which one we want to currently use. This may be useful in situations where we have inserted data into a table and we want to update the table without the risk of damaging the old table while we we are inserting the new contents. Or we want to be able to easily switch back to an old version without having to re-insert this data.

The trick we use here is to create a view that points to the current table we want to use. To be able to atomically point the view to another table, we use a stored procedure.

We start with the SQL code to create the stored procedure createMyTable. This stored procedure has one argument: the name of the table to create, e.g. mytable_2010_11_22.

DROP PROCEDURE IF EXISTS createMyTable;

DELIMITER //
CREATE PROCEDURE createMyTable(IN table_name text)
BEGIN
   SET @sql = CONCAT("CREATE TABLE ", table_name, "(", "id int,", "name text");");

   PREPARE stmt FROM @sql;
   EXECUTE stmt;

   DEALLOCATE PREPARE stmt;
END //
DELIMITER ;

At this point we can create a new table and fill it:

CALL createMyTable("mytable_2010_11_22");
INSERT INTO mytable_2010_11_22 (id, name) VALUES (1, "Alice");
INSERT ...

The second part we need is the stored procedure that changes the view to point to the newly created table. This stored procedure also has one argument: the name of the table to point to, e.g. mytable_2010_11_22.

DROP PROCEDURE IF EXISTS createMyView;

DELIMITER //
CREATE PROCEDURE createMyView(IN name text)
BEGIN
    SET @sql = CONCAT("CREATE OR REPLACE VIEW mytable AS SELECT * FROM ", name);

    PREPARE stmt FROM @sql;
    EXECUTE stmt;

    DEALLOCATE PREPARE stmt;
END //
DELIMITER ;

Call this stored procedure to point to our newly created table:

CALL createMyView("mytable_2010_11_22");

Now, if we want to list the content of the table that the view currently points to, we can use something like this:

SELECT * FROM mytable;

Posted in Tagged , Leave a Comment

Measuring temperature and humidity with a JeeNode

By jansipke on 2010-09-30

In a previous article we looked at the Arduino. Now we will look at an (almost) Arduino compatible clone, the JeeNode.

It is a bit smaller and cheaper than the original version and operates at 3.3V instead of 5V. This makes it easy to connect 3.3V peripherals, such as the radio module RX868 by ELV.

Connecting this device to the JeeNode means that we can listen to signals sent out by the temperature and humidity sensor S555TH.

Many thanks go out to JeeLabs for providing the On-Off Keying (OOK) decoding used by the S555TH, and Pieter Meulenhoff for supplying me with the idea of using this sensor and initial versions of the software.

The S555TH transmits the following data within the S300 protocol:The first byte contains the sign of the temperature value and the address of the sensor. If the first bit is ’0′, the temperature sign is ‘+’, else it is ‘-’. The second to fourth bit are used for the address (1-8) of the sensor. This is used for identifying the sensor, but also helps in avoiding packet collisions. Each sensor will wait for (177 minus 0.5 * address) seconds between transmissions.

The second through fourth bytes contain the temperature and humidity values. For example, the humidity value is obtained by multiplying 10 with the first nibble of the fourth byte, adding the second nibble of the fourth byte, and adding 0.1 times the first nibble of the third byte.

In code this looks like:

uint8_t address = 1 + ((buf[0] >> 4) & 0x07);
int16_t temperature = (buf[0] >> 7 ? -1 : 1) * (100 * (buf[2] & 0x0F) + 10 * (buf[1] >> 4) + 1 * (buf[1] & 0x0F));
uint16_t humidity = 100 * (buf[3] >> 4) + 10 * (buf[3] & 0x0F) + 1 * (buf[2] >> 4);

Serial.print("S555TH: address=");
Serial.print(address, DEC);
Serial.print(" temperature=");
Serial.print(temperature / 10.0, 1);
Serial.print(" humidity=");
Serial.println(humidity / 10.0, 1);

Note that we multiplied the temperature and humidity values by 10 to be able to print them nicely to the serial monitor, in which we divide by 10 again.

The complete Arduino sketch can be downloaded here.

Posted in , Tagged , , , 1 Comment

Programming the Arduino ethernet shield

By jansipke on 2010-09-08

The Arduino is an open-source electronics prototyping platform based on flexible, easy-to-use hardware and software. It features an IDE that allows you to program the Arduino using a standard USB cable. Several shields exist that extend the functionality of the standard Arduino platform. In the picture above the Arduino is shown on the left and the ethernet shield is shown on the right.

The ethernet shield adds IP communication, acting as a client or as a server. The most well-known protocols are supported right out of the box, including TCP, UDP and ICMP. Several enthousiasts have added support for even more protocols, such as DHCP and NTP.

Creating programs for the Arduino with the ethernet shield is simple:

#include 

byte mac[] = { 0xDE, 0xAD, 0xBE, 0xEF, 0xFE, 0xED };
byte ip[] = { 10, 0, 0, 177 };
byte server[] = { 64, 233, 187, 99 }; // Google

Client client(server, 80);

void setup()
{
   Ethernet.begin(mac, ip);
   Serial.begin(9600);

   delay(1000);

   Serial.println("connecting...");

   if (client.connect()) {
      Serial.println("connected");
      client.println("GET /search?q=arduino HTTP/1.0");
      client.println();
   } else {
      Serial.println("connection failed");
   }
}

void loop()
{
   if (client.available()) {
      char c = client.read();
      Serial.print(c);
   }

   if (!client.connected()) {
      Serial.println();
      Serial.println("disconnecting.");
      client.stop();
      for(;;)
         ;
   }
}

Every Arduino program has at least two parts: a setup() function and a loop() function. The Arduino IDE creates an actual proper C program around these two functions before it uploads the complete program to the Arduino.

The Arduino IDE contains a serial monitor, which allows you to send strings to the Arduino program or receive strings from it. The example program above shows how the Serial class works. In the setup() function, the serial communication is started at 9600 baud. In the loop() function, data is sent over the serial line with the method Serial.print(). If the serial line is actually connected, the program on the Arduino will halt until the serial monitor is opened. If the serial line is not connected, e.g. when you are done debugging, the program will run normally.

The ethernet library contains the Ethernet class. In the setup() function, the ethernet stack is started with Ethernet.begin(mac, ip). The mac and ip parameters are both byte arrays. If only these two parameters are present and the IP address has the form a.b.c.d, the gateway is assumed to be a.b.c.1 and the subnetmask is assumed to be 255.255.255.0. There are other constructors that take one or two extra parameters for people with different network setups.

After ethernet initialization, the class Client is used to create a TCP connection to the supplied server IP address and port number. The Client.connect() method tries to connect to the server and a call to Client.println() sends data over the TCP connection to the server. The Client.available() method tells if the server has sent data over the TCP connection to our client. If it has, a call to Client.read() gives these bytes.

Posted in , , Tagged 1 Comment

Metering the meter cupboard

By jansipke on 2010-08-29

In this article we will look at my meter cupboard and the meters inside it. In future articles we will see how I used a microcontroller and some sensors to meter the meters (at the time of writing the electricity meter works). From left to right, these are the electricity, gas and water meter:

Electricity

The electricity meter is a relatively new one with a digital display and a flashing LED. This LED flashes one thousand times for each kWh consumed, i.e. once per Wh.

Gas

The gas meter doesn’t have a flashing LED or something comparable. There is hope however, because there is a reflective area on the digit ’6′ at the far right of the display, just left of the ‘m2′. It also looks like there is some magnetic material in one of the last two digits, so there are actually two ways to go with this meter.

Water

The water meter doesn’t have a flashing LED or a reflective area anywhere. It also doesn’t have a colored dial, which most commercial water meter detectors use. There is a tiny bit of hope in the hole below ‘PR’. Using a magnetic sensor, I found some fluctuation in the values, but nothing consistent so far. The orientation of the sensor seems to be important, so I will fiddle a bit more with this one in the future.

Posted in Tagged 2 Comments

Model View Controller (MVC) with JSP and JSTL

By jansipke on 2010-07-20

In this article we will create a small web application that uses the Model View Controller (MVC) pattern with Java Server Pages (JSP) and JSP Standard Template Library (JSTL). A container like Tomcat is needed to run this combination.

Thanks go out to the author of the JSP – MVC Tutorial, who thought of the sample application that we will use here, the coffee advisor. The user is first presented with a choice in coffee taste she prefers. Pressing a button moves on to a page with advise about the type of coffee to drink based on that taste.

An MVC application has three parts:

  • Model. The model is the domain-specific representation of the data upon which the application operates. In our case this is implemented in the CoffeeExpert class.
  • View. The view renders the model into a form suitable for interaction, typically a user interface element. In our case this is implemented in a JSP file called coffee.jsp.
  • Controller. The controller receives input and initiates a response by making calls on model objects. In our case this is implemented in the CoffeeSelect class.

In addition to these three file we need a web.xml file that tells the container how to map a URL (e.g. /CoffeeSelect.do) into a class to run (e.g. com.example.web.CoffeeSelect). We also need a start page for the user input, which we will call coffee.html. In total this means we should create a  .war file with the following structure:

coffee.html

<html>
    <body>
        <h2>Coffee Advisor Input</h2>
        <form method="POST" action="CoffeeSelect.do">
            <select name="taste" size=1">
                <option value="milky">Milky</option>
                <option value="froffy">Froffy</option>
                <option value="icey">Icey</option>
                <option value="strong">Spaced Out</option>
            </select>
            <br/><br/>
            <input type="Submit"/>
        </form>
    </body>
</html>

CoffeeExpert.java

package com.example.model;

import java.util.*;

public class CoffeeExpert {
    public List<String> getTypes(String taste) {
        List<String> result = new ArrayList<String>();
        if (taste.equals("milky")) {
            result.add("Latte");
            result.add("Cappuccino");
        } else if (taste.equals("froffy")) {
            result.add("Latte");
            result.add("Cappuccino");
            result.add("Frappuccino");
        } else if (taste.equals("icey")) {
            result.add("Frappuccino");
        } else if (taste.equals("strong")) {
            result.add("Espresso");
            result.add("Double espresso");
        } else {
            result.add("Vending machine");
        }
        return (result);
    }
}

CoffeeSelect.java

package com.example.web;

import java.io.IOException;
import java.util.List;

import javax.servlet.RequestDispatcher;
import javax.servlet.ServletException;
import javax.servlet.http.HttpServlet;
import javax.servlet.http.HttpServletRequest;
import javax.servlet.http.HttpServletResponse;

import com.example.model.CoffeeExpert;

public class CoffeeSelect extends HttpServlet {

    private static final long serialVersionUID = 1L;

    public void doPost(HttpServletRequest request, HttpServletResponse response) throws IOException, ServletException {
        List<String> types = new CoffeeExpert().getTypes(request.getParameter("taste"));
        request.setAttribute("types", types);
        RequestDispatcher view = request.getRequestDispatcher("coffee.jsp");
        view.forward(request, response);
    }
}

coffee.jsp

<%@ taglib uri="http://java.sun.com/jsp/jstl/core" prefix="c" %>

<html>
    <body>
        <h2>Coffee Advisor Output</h2>
        <c:forEach var="type" items="${types}">
            <c:out value="${type}"/>
            <br />
        </c:forEach>
    </body>
</html>

web.xml

<?xml version="1.0" encoding="ISO-8859-1"?>
<web-app xmlns="http://java.sun.com/xml/ns/j2ee"
    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xsi:schemaLocation="http://java.sun.com/xml/ns/j2ee http://java.sun.com/xml/ns/j2ee/web-app_2_4.xsd"
    version="2.4">

    <servlet>
        <servlet-name>Coffee</servlet-name>
        <servlet-class>com.example.web.CoffeeSelect</servlet-class>
    </servlet>

    <servlet-mapping>
        <servlet-name>Coffee</servlet-name>
        <url-pattern>/CoffeeSelect.do</url-pattern>
    </servlet-mapping>
</web-app>

Posted in Tagged , , , 2 Comments

Changing CPU and memory settings on XenServer VMs

By jansipke on 2010-06-08

You can change the CPU and memory settings of XenServer virtual machines using the API. XenServer allows you to set the priority of the virtual machine CPU (called weight), set a limit on the amount of CPU the virtual machine can use (called cap) and set the amount of memory the virtual machine gets.

CPU priority

You can set the weight parameter to a number between 1 and 65536. The default value is 256, meaning normal priority. A virtual machine with a weight of 512 will get twice as much CPU as a virtual machine with a weight of 256 on a contended XenServer host. The following code sets the weight to 128, meaning that it will get half as much CPU as a normal virtual machine.

Map<String, String> vcpuParameters = new HashMap<String, String>();
vcpuParameters.put("weight", "128");
vm.setVCPUsParams(connection, vcpuParameters);

CPU limit

You can set the cap parameter to a percentage number. The default value is 0, meaning there is no cap. The value 100 represents one virtual CPU, so half a virtual CPU can be set by using the value 50. The following code sets the cap to 35, meaning that it will not get more than 35 percent of one virtual CPU, even if the physical CPU is not used to its top.

Map<String, String> vcpuParameters = new HashMap<String, String>();
vcpuParameters.put("cap", "35");
vm.setVCPUsParams(connection, vcpuParameters);

Memory

You can set the amount of memory assigned to a virtual machine in bytes. The following code sets the amount of memory to 1 GB (1,073,741,824 bytes).

vm.setMemoryStaticMax(connection, new Long(1073741824));

Posted in , Tagged , 8 Comments

Increasing the size of an LVM volume

By jansipke on 2010-06-03

In a previous article we looked at LVM. This short article describes how to grow an existing volume by adding an extra disk to the system. Specifically, we use a XenServer guest VM with CentOS installed.

The first disk of a XenServer guest VM is called xvda, the second disk (the one we added) is called xvdb. We start with the creation of a Physical Volume (PV) on the extra disk.

pvcreate /dev/xvdb

Then we extend the existing Volume Group (VG) called VolGroup00.

vgextend VolGroup00 /dev/xvdb

Then we extend the Logical Volume (LV) by the size of the extra disk, in this case 8GB.

lvextend -L8G /dev/VolGroup00/LogVol00

Finally we resize the filesystem that uses this LV.

resize2fs /dev/VolGroup00/LogVol00

Posted in Tagged , , 1 Comment