OpenStack Nova nova.sh script explained

Update 11/24/2011: Updated article based on the latest nova.sh script.

This article describes the internals of the script nova.sh used to get the OpenStack Nova source code, install it and run it. Nova is a cloud computing fabric controller, the main part of an IaaS system.

The script can be retrieved using Git:

git clone https://github.com/vishvananda/novascript.git

Arguments

The script takes 1 mandatory argument and 2 optional arguments:

• command: “branch”, “install”, “run”, “terminate”, “clean”, “scrub”.
• source branch (branch command only): default to “lp:nova” which is the location of the source code on Launchpad.
• install directory: default to “nova”

Note: You will need to use sudo to run the script.

Initialization

The arguments are grabbed from the command line or set to their defaults:

CMD=$1
if [ "$CMD" = "branch" ]; then
    SOURCE_BRANCH=${2:-lp:nova}
    DIRNAME=${3:-nova}
else
    DIRNAME=${2:-nova}
fi

By default, sqlite will be used but you can use MySQL instead by setting the env variable USE_MYSQL and MYSQL_PASS.

USE_MYSQL=${USE_MYSQL:-0}
MYSQL_PASS=${MYSQL_PASS:-nova}

Next is the interface used as the public interface and the VLAN interface in the nova configuration file.

INTERFACE=${INTERFACE:-eth0}

Floating IP addresses are used for HA. One VM can grab a floating IP address as it is taking over.

FLOATING_RANGE=${FLOATING_RANGE:-10.6.0.0/27}

Fixed IP addresses are attached to the different interfaces.

FIXED_RANGE=${FIXED_RANGE:-10.0.0.0/24}

You can set the path where instances data are stored:

INSTANCES_PATH=${INSTANCES_PATH:-$NOVA_DIR/instances}

You can also force the script to run some unit tests (Python unit tests) when you use the command “run”. For example, it will test the api, authentication, compute, network modules and much more. You can take a look at the folder nova/tests/ to see all the tests.

TEST=${TEST:-0}

LDAP can be used for authentication. It is not used by default. The database is used by default to store authentication data.

USE_LDAP=${USE_LDAP:-0}

OpenDJ can be used instead of OpenLDAP when LDAP is used. OpenDJ is a new LDAPv3 compliant directory service, developed for the Java platform, providing a high performance, highly available and secure store for the identities managed by enterprises.

USE_OPENDJ=${USE_OPENDJ:-0}

IPv6 support can be enabled:

USE_IPV6=${USE_IPV6:-0}

Nova has support for libvirt and you can set LIBVIRT_TYPE to something if you don’t like the default qemu. You can set it to “uml” and a different libvirt XML template will be used. Libvirt is a virtualization API.

LIBVIRT_TYPE=${LIBVIRT_TYPE:-qemu}

Next is the network manager type. It defaults to VlanManager where a host-managed VLAN will be created for each project. Other types are FlatManager, FlatDHCPManager. See Network Manager Documentation for more details.

NET_MAN=${NET_MAN:-VlanManager}

In case you are using FlatDHCP on multiple hosts, you need to set the env variable FLAT_INTERFACE to a network interface with no defined IP.

FLAT_INTERFACE=ethx

The first network interface IP address is grabbed using the ifconfig command. It is explained in the script that if you have more than 1 network interfaces then you should set the environment variable HOST_IP.

if [ ! -n "$HOST_IP" ]; then
HOST_IP=`LC_ALL=C ifconfig  | grep -m 1 'inet addr:'| cut -d: -f2 | awk '{print $1}'`
fi

The connection to the database is defined the following way. It will be MySQL or sqlite based on your choice.

if [ "$USE_MYSQL" == 1 ]; then
    SQL_CONN=mysql://root:$MYSQL_PASS@localhost/nova
else
    SQL_CONN=sqlite:///$NOVA_DIR/nova.sqlite
fi

The authentication driver is set based on your choice: LDAP or not. If LDAP is not selected (by default), it will use the database to store the authentication data.

if [ "$USE_LDAP" == 1 ]; then
    AUTH=ldapdriver.LdapDriver
else
    AUTH=dbdriver.DbDriver
fi

Branch command

This command installs Bazaar (bzr) which is a distributed version control system, initializes the repository, retrieves the latest source code for Nova and places it in the Nova folder you just defined It also creates the “instances” folder and “networks” folders.

if [ "$CMD" == "branch" ]; then
    sudo apt-get install -y bzr
    if [ ! -e "$DIR/.bzr" ]; then
        bzr init-repo $DIR
    fi
    rm -rf $NOVA_DIR
    bzr branch $SOURCE_BRANCH $NOVA_DIR
    cd $NOVA_DIR
    mkdir -p $NOVA_DIR/instances
    mkdir -p $NOVA_DIR/networks
    exit
fi

LXC setup

The libvirt LXC driver manages “Linux Containers”. Containers are sets of processes with private namespaces which can (but don’t always) look like separate machines, but do not have their own OS. If you use “lxc” for the libvirt type, some cgroups controllers need to be mounted on the host OS.

has_fsmp() {
  # has_fsmp(mountpoint,file): does file have an fstab entry for mountpoint
  awk '$1 !~ /#/ && $2 == mp { e=1; } ; END { exit(!e); }' "mp=$1" "$2" ;
}

function lxc_setup() {
  local mntline cmd=""
  mntline="none /cgroups cgroup cpuacct,memory,devices,cpu,freezer,blkio 0 0"
  has_fsmp "/cgroups" /etc/fstab ||
     cmd="$cmd && mkdir -p /cgroups && echo '$mntline' >> /etc/fstab"
  has_fsmp "/cgroups" /proc/mounts ||
     cmd="$cmd && mount /cgroups"

  [ -z "$cmd" ] && return 0
  sudo sh -c ": $cmd"
}

[ "$LIBVIRT_TYPE" != "lxc" ] || lxc_setup || fail "failed to setup lxc"

Install command

The following Debian packages are installed if not already installed:

• python-software-properties: This software provides an abstraction of the used apt repositories. It allows you to easily manage your distribution and independent software vendor software sources.
• dnsmasq-base: A small caching DNS proxy and DHCP/TFTP server.
• kpartx: create device mappings for partitions.
• kvm: Full virtualization on x86 hardware.
• gawk: a pattern scanning and processing language.
• iptables: administration tools for packet filtering and NAT.
• ebtables: Ethernet bridge frame table administration.
• user-mode-linux: User-mode Linux (kernel).
• libvirt-bin: the programs for the libvirt library.
• screen: terminal multiplexor with VT100/ANSI terminal emulation.
• euca2ools: managing cloud instances for Eucalyptus.
• vlan: user mode programs to enable VLANs on your ethernet devices.
• curl: Get a file from an HTTP, HTTPS or FTP server.
• rabbitmq-server: An AMQP server written in Erlang.
• lvm2: The Linux Logical Volume Manager.
• iscsitarget: iSCSI Enterprise Target userland tools.
• open-iscsi: High performance, transport independent iSCSI implementation.
• socat: multipurpose relay for bidirectional data transfer.
• unzip: De-archiver for .zip files.
• glance: The Glance project provides an image registration and discovery service (Parallax) and an image delivery service (Teller).
• radvd: Router Advertisement Daemon.
• python-twisted: Event-based framework for internet applications.
• python-sqlalchemy: SQL toolkit and Object Relational Mapper for Python.
• python-suds: Lightweight SOAP client for Python.
• python-lockfile: file locking library for Python.
• python-mox: a mock object framework for Python.
• python-lxml: pythonic binding for the libxml2 and libxslt libraries.
• python-kombu: AMQP Messaging Framework for Python.
• python-greenlet: Lightweight in-process concurrent programming.
• python-carrot: An AMQP messaging queue framework.
• python-migrate: Database schema migration for SQLAlchemy.
• python-eventlet: Eventlet is a concurrent networking library for Python.
• python-gflags: Python implementation of the Google command line flags module.
• python-novaclient: client library for OpenStack Compute API.
• python-ipy: Python module for handling IPv4 and IPv6 addresses and networks.
• python-cheetah: text-based template engine and Python code generator.
• python-libvirt: libvirt Python bindings.
• python-libxml2: Python bindings for the GNOME XML library.
• python-routes: Routing Recognition and Generation Tools.
• python-paste: Tools for using a Web Server Gateway Interface stack.
• python-netaddr: manipulation of various common network address notations.
• python-tempita: very small text templating language.
• python-pastedeploy: Load, configure, and compose WSGI applications and servers.
• python-glance: OpenStack Image Registry and Delivery Service.

The script also adds an APT repository: “ppa:nova-core/trunk” which contains some patched versions of some of the packages above.

The modules kvm and ndb are loaded. iscsitarget and libvirt-bin are restarted and a test image is downloaded and uncompressed.

If you enable IPv6 support, radvd will be installed, IPv6 forwarding will be enabled and router advertisement messages will be ignored.

If you chose to use MySQL, the root password is set for you based on the environment variable MYSQL_PASS and the following 2 packages are installed: mysql-server and python-mysqldb.

if [ "$CMD" == "install" ]; then
    sudo apt-get install -y python-software-properties
    sudo add-apt-repository ppa:nova-core/trunk
    sudo apt-get update
    sudo apt-get install -y dnsmasq-base kpartx kvm gawk iptables ebtables
    sudo apt-get install -y user-mode-linux kvm libvirt-bin
    # Bypass  RabbitMQ "OK" dialog
    echo "rabbitmq-server rabbitmq-server/upgrade_previous note" | sudo debconf-set-selections
    sudo apt-get install -y screen euca2ools vlan curl rabbitmq-server
    sudo apt-get install -y lvm2 iscsitarget open-iscsi
    sudo apt-get install -y socat unzip glance
    echo "ISCSITARGET_ENABLE=true" | sudo tee /etc/default/iscsitarget
    sudo /etc/init.d/iscsitarget restart
    sudo modprobe kvm
    sudo /etc/init.d/libvirt-bin restart
    sudo modprobe ndb
    sudo apt-get install -y python-mox python-lxml python-kombu python-paste
    sudo apt-get install -y python-migrate python-gflags python-greenlet
    sudo apt-get install -y python-libvirt python-libxml2 python-routes
    sudo apt-get install -y python-netaddr python-pastedeploy python-eventlet
    sudo apt-get install -y python-novaclient python-glance python-cheetah
    sudo apt-get install -y python-carrot python-tempita python-sqlalchemy
    sudo apt-get install -y python-suds python-lockfile python-netaddr
    
    if [ "$USE_IPV6" == 1 ]; then
        sudo apt-get install -y radvd
        sudo bash -c "echo 1 > /proc/sys/net/ipv6/conf/all/forwarding"
        sudo bash -c "echo 0 > /proc/sys/net/ipv6/conf/all/accept_ra"
    fi

    if [ "$USE_MYSQL" == 1 ]; then
        cat <<MYSQL_PRESEED | debconf-set-selections
mysql-server-5.1 mysql-server/root_password password $MYSQL_PASS
mysql-server-5.1 mysql-server/root_password_again password $MYSQL_PASS
mysql-server-5.1 mysql-server/start_on_boot boolean true
MYSQL_PRESEED
        apt-get install -y mysql-server python-mysqldb
    fi
    exit
fi

Run command

A lot is happening in this section. First is

A new screen is started in detached mode with the session name specified in the environment variable SCREEN_NAME. The following code also checks if a screen with the same session name already exists and asks the user to kill it if it is the case. Screen is a full-screen window manager that multiplexes a physical terminal between several processes.

# check for existing screen, exit if present
  found=$(screen -ls | awk '-F\t' '$2 ~ m {print $2}' "m=[0-9]+[.]$SCREEN_NAME")
  if [ -n "$found" ]; then
    {
    echo "screen named '$SCREEN_NAME' already exists!"
    echo " kill it with: screen -r '$SCREEN_NAME' -x -X quit"
    echo " attach to it with: screen -d -r '$SCREEN_NAME'"
    exit 1;
    } 2>&1
  fi
  screen -d -m -S $SCREEN_NAME -t nova
  sleep 1
  if [ "$SCREEN_STATUS" != "0" ]; then
    screen -r "$SCREEN_NAME" -X hardstatus alwayslastline "%-Lw%{= BW}%50>%n%f* %t%{-}%+Lw%< %= %H"
  fi

Based on the environment variables set above, the script writes the nova flags to nova.conf and creates the nova folder in /etc/.

cat >$NOVA_DIR/bin/nova.conf << NOVA_CONF_EOF
--verbose
--nodaemon
--dhcpbridge_flagfile=$NOVA_DIR/bin/nova.conf
--network_manager=nova.network.manager.$NET_MAN
--my_ip=$HOST_IP
--public_interface=$INTERFACE
--vlan_interface=$INTERFACE
--sql_connection=$SQL_CONN
--auth_driver=nova.auth.$AUTH
--libvirt_type=$LIBVIRT_TYPE
--fixed_range=$FIXED_RANGE
--lock_path=$LOCK_PATH
--instances_path=$INSTANCES_PATH
--flat_network_bridge=br100
NOVA_CONF_EOF

if [ -n "$FLAT_INTERFACE" ]; then
    echo "--flat_interface=$FLAT_INTERFACE" >>$NOVA_DIR/bin/nova.conf
fi

if [ "$USE_IPV6" == 1 ]; then
    echo "--use_ipv6" >>$NOVA_DIR/bin/nova.conf
fi

Next, all dnsmasq processes are killed: DNS cache proxy + DHCP server.

killall dnsmasq

In case IPv6 support is enabled, radvd is killed:

if [ "$USE_IPV6" == 1 ]; then
   killall radvd
fi

The script recreates the database “nova”.

if [ "$USE_MYSQL" == 1 ]; then
    mysql -p$MYSQL_PASS -e 'DROP DATABASE nova;'
    mysql -p$MYSQL_PASS -e 'CREATE DATABASE nova;'
else
    rm $NOVA_DIR/nova.sqlite
fi

If you decided to use LDAP, OpenLDAP or OpenDJ needs to be configured:

if [ "$USE_LDAP" == 1 ]; then
    if [ "$USE_OPENDJ" == 1 ]; then
        echo '--ldap_user_dn=cn=Directory Manager' >> \
            /etc/nova/nova-manage.conf
        sudo $NOVA_DIR/nova/auth/opendj.sh
    else
        sudo $NOVA_DIR/nova/auth/slap.sh
    fi
fi

The script also recreates the instances and networks folders.

rm -rf $NOVA_DIR/instances
mkdir -p $NOVA_DIR/instances
rm -rf $NOVA_DIR/networks
mkdir -p $NOVA_DIR/networks

If test mode is enabled (see above), the unit tests are run:

if [ "$TEST" == 1 ]; then
    cd $NOVA_DIR
    python $NOVA_DIR/run_tests.py
    cd $DIR
fi

A new database is created

$NOVA_DIR/bin/nova-manage db sync

A new admin user is added:

$NOVA_DIR/bin/nova-manage user admin admin admin admin

A new project “admin” managed by “admin” is created:

$NOVA_DIR/bin/nova-manage project create admin admin

A small network is created with 32 IPs from the fixed range:

$NOVA_DIR/bin/nova-manage network create private $FIXED_RANGE 1 32

Create some floating IPs using the floating range.

$NOVA_DIR/bin/nova-manage floating create $FLOATING_RANGE

Download an image from ansolabs and untar it in the images dir.

if [ ! -d $DIR/images ]; then
   mkdir -p $DIR/images
   wget -c http://images.ansolabs.com/tty.tgz
   tar -C $DIR/images -zxf tty.tgz
fi

If ami-tty image in images service then convert the image in directory from the old (Bexar) format to the new format.

if ! glance details | grep ami-tty; then
    $NOVA_DIR/bin/nova-manage image convert $DIR/images
fi

The file novarc looks like this for me. I am running things on Amazon EC2 right now. 10.240.95.3 is my EC2 instance private IP. The EC2 API server is listening on port 8773 and the OpenStack API is listening on port 8774. The data store will be running on port 3333.

The different servers, controllers and stores are started and we can browse them through the screen session created above. See Nova Concepts and Introduction for more details on the Nova architecture.

screen_it api "$NOVA_DIR/bin/nova-api"
screen_it objectstore "$NOVA_DIR/bin/nova-objectstore"
screen_it compute "$NOVA_DIR/bin/nova-compute"
screen_it network "$NOVA_DIR/bin/nova-network"
screen_it scheduler "$NOVA_DIR/bin/nova-scheduler"
screen_it volume "$NOVA_DIR/bin/nova-volume"
screen_it ajax_console_proxy "$NOVA_DIR/bin/nova-ajax-console-proxy"
sleep 2
$NOVA_DIR/bin/nova-manage project zipfile admin admin $NOVA_DIR/nova.zip
unzip -o $NOVA_DIR/nova.zip -d $NOVA_DIR/
screen_it test "export PATH=$NOVA_DIR/bin:$PATH;. $NOVA_DIR/novarc"
if [ "$CMD" != "run_detached" ]; then
  screen -S nova -x
fi

NOVA_KEY_DIR=$(pushd $(dirname $BASH_SOURCE)>/dev/null; pwd; popd>/dev/null)
export EC2_ACCESS_KEY="admin:admin"
export EC2_SECRET_KEY="admin"
export EC2_URL="http://10.240.95.3:8773/services/Cloud"
export S3_URL="http://10.240.95.3:3333"
export EC2_USER_ID=42 # nova does not use user id, but bundling requires it
export EC2_PRIVATE_KEY=${NOVA_KEY_DIR}/pk.pem
export EC2_CERT=${NOVA_KEY_DIR}/cert.pem
export NOVA_CERT=${NOVA_KEY_DIR}/cacert.pem
export EUCALYPTUS_CERT=${NOVA_CERT} # euca-bundle-image seems to require this set
alias ec2-bundle-image="ec2-bundle-image --cert ${EC2_CERT} --privatekey ${EC2_PRIVATE_KEY} --user 42 --ec2cert ${NOVA_CERT}"
alias ec2-upload-bundle="ec2-upload-bundle -a ${EC2_ACCESS_KEY} -s ${EC2_SECRET_KEY} --url ${S3_URL} --ec2cert ${NOVA_CERT}"
export CLOUD_SERVERS_API_KEY="admin"
export CLOUD_SERVERS_USERNAME="admin"
export CLOUD_SERVERS_URL="http://10.240.95.3:8774/v1.0/"

Once the nova session is finished, the instances are terminated and the volumes deleted. This code is also run when the command “terminate” is used.

if [ "$CMD" == "run" ] || [ "$CMD" == "terminate" ]; then
    # shutdown instances
    . $NOVA_DIR/novarc; euca-describe-instances | grep i- | cut -f2 | xargs     euca-terminate-instances
    sleep 2
    # delete volumes
    . $NOVA_DIR/novarc; euca-describe-volumes | grep vol- | cut -f2 | xargs -n1 euca-delete-volume
    sleep 2
fi

The screen session is forced to shutdown. This part is also called when the command “clean” is executed.

if [ "$CMD" == "run" ] || [ "$CMD" == "clean" ]; then
    screen -S nova -X quit
    rm *.pid*
fi

There is one last command called “scrub” which is used to remove the bridges and VLANs configuration. It also destroys the domains.

if [ "$CMD" == "scrub" ]; then
    $NOVA_DIR/tools/clean-vlans
    if [ "$LIBVIRT_TYPE" == "uml" ]; then
        virsh -c uml:///system list | grep i- | awk '{print \$1}' | xargs -n1   virsh -c uml:///system destroy
    else
        virsh list | grep i- | awk '{print \$1}' | xargs -n1 virsh destroy
    fi
fi

When you run the branch, install and run commands, you end up with the different components (API server, object store, scheduler…) running which is really neat.

You can use the screen windows to check each component status. Do a “man screen” to learn how to navigate between the different windows if you are not familiar with the tool screen.

API server:

# /opt/novascript/trunk/bin/nova-api
(nova.root 2011.1-LOCALBRANCH:LOCALREVISION): AUDIT [N/A] Starting /opt/novascript/trunk/bin/nova-api on 0.0.0.0:8774
(nova.root 2011.1-LOCALBRANCH:LOCALREVISION): AUDIT [N/A] Starting /opt/novascript/trunk/bin/nova-api on 0.0.0.0:8773

Object store:

(nova.root 2011.1-LOCALBRANCH:LOCALREVISION): DEBUG [N/A] network_topic : network from MainProcess (pid=13763) serve /opt/novascript/trunk/nova/twistd.py:266
(nova.root 2011.1-LOCALBRANCH:LOCALREVISION): AUDIT [N/A] Starting nova-objectstore
2011-01-14 22:15:09+0000 [-] Log opened.
2011-01-14 22:15:09+0000 [-] twistd 10.0.0 (/usr/bin/python 2.6.5) starting up.
2011-01-14 22:15:09+0000 [-] reactor class: twisted.internet.selectreactor.SelectReactor.
2011-01-14 22:15:09+0000 [-] twisted.web.server.Site starting on 3333
2011-01-14 22:15:09+0000 [-] Starting factory <twisted.web.server.Site instance at 0xaaf5bec>

…

You can launch a new instance this way:

cd /tmp/
euca-add-keypair test > test.pem
euca-run-instances -k test -t m1.tiny ami-tty

You can list your instances this way:

# euca-describe-instances
RESERVATION     r-ky6gm38t      admin
INSTANCE        i-00000001      ami-tty        10.0.0.3        10.0.0.3        running test (admin, domU-12-31-39-04-58-F5)    0               m1.tiny 2011-01-14 22:32:03.466420      nova

Running ifconfig, we can now see the bridge and VLAN created

br100     Link encap:Ethernet  HWaddr 12:31:39:04:58:f5  
          inet addr:10.0.0.1  Bcast:10.0.0.31  Mask:255.255.255.224
...
vlan100   Link encap:Ethernet  HWaddr 12:31:39:04:58:f5  
          inet6 addr: fe80::1031:39ff:fe04:58f5/64 Scope:Link
...

A network route has been set up for us to use the bridge interface when the packets are sent to 10.0.0.x:

# route -n
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
10.0.0.0        0.0.0.0         255.255.255.224 U     0      0        0 br100
...

Voilà. Don’t hesitate to post a comment if you have any feedback.