Update 11/24/2011: Updated article based on the latest nova.sh script.
This article describes the internals of the script nova.sh used to get the OpenStack Nova source code, install it and run it. Nova is a cloud computing fabric controller, the main part of an IaaS system.
The script can be retrieved using Git:
git clone https://github.com/vishvananda/novascript.git
Arguments
The script takes 1 mandatory argument and 2 optional arguments:
- command: “branch”, “install”, “run”, “terminate”, “clean”, “scrub”.
- source branch (branch command only): default to “lp:nova” which is the location of the source code on Launchpad.
- install directory: default to “nova”
Note: You will need to use sudo to run the script.
Initialization
The arguments are grabbed from the command line or set to their defaults:
CMD=$1 if [ "$CMD" = "branch" ]; then SOURCE_BRANCH=${2:-lp:nova} DIRNAME=${3:-nova} else DIRNAME=${2:-nova} fi
By default, sqlite will be used but you can use MySQL instead by setting the env variable USE_MYSQL and MYSQL_PASS.
USE_MYSQL=${USE_MYSQL:-0} MYSQL_PASS=${MYSQL_PASS:-nova}
Next is the interface used as the public interface and the VLAN interface in the nova configuration file.
INTERFACE=${INTERFACE:-eth0}
Floating IP addresses are used for HA. One VM can grab a floating IP address as it is taking over.
FLOATING_RANGE=${FLOATING_RANGE:-10.6.0.0/27}
Fixed IP addresses are attached to the different interfaces.
FIXED_RANGE=${FIXED_RANGE:-10.0.0.0/24}
You can set the path where instances data are stored:
INSTANCES_PATH=${INSTANCES_PATH:-$NOVA_DIR/instances}
You can also force the script to run some unit tests (Python unit tests) when you use the command “run”. For example, it will test the api, authentication, compute, network modules and much more. You can take a look at the folder nova/tests/ to see all the tests.
TEST=${TEST:-0}
LDAP can be used for authentication. It is not used by default. The database is used by default to store authentication data.
USE_LDAP=${USE_LDAP:-0}
OpenDJ can be used instead of OpenLDAP when LDAP is used. OpenDJ is a new LDAPv3 compliant directory service, developed for the Java platform, providing a high performance, highly available and secure store for the identities managed by enterprises.
USE_OPENDJ=${USE_OPENDJ:-0}
IPv6 support can be enabled:
USE_IPV6=${USE_IPV6:-0}
Nova has support for libvirt and you can set LIBVIRT_TYPE to something if you don’t like the default qemu. You can set it to “uml” and a different libvirt XML template will be used. Libvirt is a virtualization API.
LIBVIRT_TYPE=${LIBVIRT_TYPE:-qemu}
Next is the network manager type. It defaults to VlanManager where a host-managed VLAN will be created for each project. Other types are FlatManager, FlatDHCPManager. See Network Manager Documentation for more details.
NET_MAN=${NET_MAN:-VlanManager}
In case you are using FlatDHCP on multiple hosts, you need to set the env variable FLAT_INTERFACE to a network interface with no defined IP.
FLAT_INTERFACE=ethx
The first network interface IP address is grabbed using the ifconfig command. It is explained in the script that if you have more than 1 network interfaces then you should set the environment variable HOST_IP.
if [ ! -n "$HOST_IP" ]; then HOST_IP=`LC_ALL=C ifconfig | grep -m 1 'inet addr:'| cut -d: -f2 | awk '{print $1}'` fi
The connection to the database is defined the following way. It will be MySQL or sqlite based on your choice.
if [ "$USE_MYSQL" == 1 ]; then SQL_CONN=mysql://root:$MYSQL_PASS@localhost/nova else SQL_CONN=sqlite:///$NOVA_DIR/nova.sqlite fi
The authentication driver is set based on your choice: LDAP or not. If LDAP is not selected (by default), it will use the database to store the authentication data.
if [ "$USE_LDAP" == 1 ]; then AUTH=ldapdriver.LdapDriver else AUTH=dbdriver.DbDriver fi
Branch command
This command installs Bazaar (bzr) which is a distributed version control system, initializes the repository, retrieves the latest source code for Nova and places it in the Nova folder you just defined It also creates the “instances” folder and “networks” folders.
if [ "$CMD" == "branch" ]; then sudo apt-get install -y bzr if [ ! -e "$DIR/.bzr" ]; then bzr init-repo $DIR fi rm -rf $NOVA_DIR bzr branch $SOURCE_BRANCH $NOVA_DIR cd $NOVA_DIR mkdir -p $NOVA_DIR/instances mkdir -p $NOVA_DIR/networks exit fi
LXC setup
The libvirt LXC driver manages “Linux Containers”. Containers are sets of processes with private namespaces which can (but don’t always) look like separate machines, but do not have their own OS. If you use “lxc” for the libvirt type, some cgroups controllers need to be mounted on the host OS.
has_fsmp() { # has_fsmp(mountpoint,file): does file have an fstab entry for mountpoint awk '$1 !~ /#/ && $2 == mp { e=1; } ; END { exit(!e); }' "mp=$1" "$2" ; } function lxc_setup() { local mntline cmd="" mntline="none /cgroups cgroup cpuacct,memory,devices,cpu,freezer,blkio 0 0" has_fsmp "/cgroups" /etc/fstab || cmd="$cmd && mkdir -p /cgroups && echo '$mntline' >> /etc/fstab" has_fsmp "/cgroups" /proc/mounts || cmd="$cmd && mount /cgroups" [ -z "$cmd" ] && return 0 sudo sh -c ": $cmd" } [ "$LIBVIRT_TYPE" != "lxc" ] || lxc_setup || fail "failed to setup lxc"
Install command
The following Debian packages are installed if not already installed:
- python-software-properties: This software provides an abstraction of the used apt repositories. It allows you to easily manage your distribution and independent software vendor software sources.
- dnsmasq-base: A small caching DNS proxy and DHCP/TFTP server.
- kpartx: create device mappings for partitions.
- kvm: Full virtualization on x86 hardware.
- gawk: a pattern scanning and processing language.
- iptables: administration tools for packet filtering and NAT.
- ebtables: Ethernet bridge frame table administration.
- user-mode-linux: User-mode Linux (kernel).
- libvirt-bin: the programs for the libvirt library.
- screen: terminal multiplexor with VT100/ANSI terminal emulation.
- euca2ools: managing cloud instances for Eucalyptus.
- vlan: user mode programs to enable VLANs on your ethernet devices.
- curl: Get a file from an HTTP, HTTPS or FTP server.
- rabbitmq-server: An AMQP server written in Erlang.
- lvm2: The Linux Logical Volume Manager.
- iscsitarget: iSCSI Enterprise Target userland tools.
- open-iscsi: High performance, transport independent iSCSI implementation.
- socat: multipurpose relay for bidirectional data transfer.
- unzip: De-archiver for .zip files.
- glance: The Glance project provides an image registration and discovery service (Parallax) and an image delivery service (Teller).
- radvd: Router Advertisement Daemon.
- python-twisted: Event-based framework for internet applications.
- python-sqlalchemy: SQL toolkit and Object Relational Mapper for Python.
- python-suds: Lightweight SOAP client for Python.
- python-lockfile: file locking library for Python.
- python-mox: a mock object framework for Python.
- python-lxml: pythonic binding for the libxml2 and libxslt libraries.
- python-kombu: AMQP Messaging Framework for Python.
- python-greenlet: Lightweight in-process concurrent programming.
- python-carrot: An AMQP messaging queue framework.
- python-migrate: Database schema migration for SQLAlchemy.
- python-eventlet: Eventlet is a concurrent networking library for Python.
- python-gflags: Python implementation of the Google command line flags module.
- python-novaclient: client library for OpenStack Compute API.
- python-ipy: Python module for handling IPv4 and IPv6 addresses and networks.
- python-cheetah: text-based template engine and Python code generator.
- python-libvirt: libvirt Python bindings.
- python-libxml2: Python bindings for the GNOME XML library.
- python-routes: Routing Recognition and Generation Tools.
- python-paste: Tools for using a Web Server Gateway Interface stack.
- python-netaddr: manipulation of various common network address notations.
- python-tempita: very small text templating language.
- python-pastedeploy: Load, configure, and compose WSGI applications and servers.
- python-glance: OpenStack Image Registry and Delivery Service.
The script also adds an APT repository: “ppa:nova-core/trunk” which contains some patched versions of some of the packages above.
The modules kvm and ndb are loaded. iscsitarget and libvirt-bin are restarted and a test image is downloaded and uncompressed.
If you enable IPv6 support, radvd will be installed, IPv6 forwarding will be enabled and router advertisement messages will be ignored.
If you chose to use MySQL, the root password is set for you based on the environment variable MYSQL_PASS and the following 2 packages are installed: mysql-server and python-mysqldb.
if [ "$CMD" == "install" ]; then sudo apt-get install -y python-software-properties sudo add-apt-repository ppa:nova-core/trunk sudo apt-get update sudo apt-get install -y dnsmasq-base kpartx kvm gawk iptables ebtables sudo apt-get install -y user-mode-linux kvm libvirt-bin # Bypass RabbitMQ "OK" dialog echo "rabbitmq-server rabbitmq-server/upgrade_previous note" | sudo debconf-set-selections sudo apt-get install -y screen euca2ools vlan curl rabbitmq-server sudo apt-get install -y lvm2 iscsitarget open-iscsi sudo apt-get install -y socat unzip glance echo "ISCSITARGET_ENABLE=true" | sudo tee /etc/default/iscsitarget sudo /etc/init.d/iscsitarget restart sudo modprobe kvm sudo /etc/init.d/libvirt-bin restart sudo modprobe ndb sudo apt-get install -y python-mox python-lxml python-kombu python-paste sudo apt-get install -y python-migrate python-gflags python-greenlet sudo apt-get install -y python-libvirt python-libxml2 python-routes sudo apt-get install -y python-netaddr python-pastedeploy python-eventlet sudo apt-get install -y python-novaclient python-glance python-cheetah sudo apt-get install -y python-carrot python-tempita python-sqlalchemy sudo apt-get install -y python-suds python-lockfile python-netaddr if [ "$USE_IPV6" == 1 ]; then sudo apt-get install -y radvd sudo bash -c "echo 1 > /proc/sys/net/ipv6/conf/all/forwarding" sudo bash -c "echo 0 > /proc/sys/net/ipv6/conf/all/accept_ra" fi if [ "$USE_MYSQL" == 1 ]; then cat <<MYSQL_PRESEED | debconf-set-selections mysql-server-5.1 mysql-server/root_password password $MYSQL_PASS mysql-server-5.1 mysql-server/root_password_again password $MYSQL_PASS mysql-server-5.1 mysql-server/start_on_boot boolean true MYSQL_PRESEED apt-get install -y mysql-server python-mysqldb fi exit fi
Run command
A lot is happening in this section. First is
A new screen is started in detached mode with the session name specified in the environment variable SCREEN_NAME. The following code also checks if a screen with the same session name already exists and asks the user to kill it if it is the case. Screen is a full-screen window manager that multiplexes a physical terminal between several processes.
# check for existing screen, exit if present found=$(screen -ls | awk '-F\t' '$2 ~ m {print $2}' "m=[0-9]+[.]$SCREEN_NAME") if [ -n "$found" ]; then { echo "screen named '$SCREEN_NAME' already exists!" echo " kill it with: screen -r '$SCREEN_NAME' -x -X quit" echo " attach to it with: screen -d -r '$SCREEN_NAME'" exit 1; } 2>&1 fi screen -d -m -S $SCREEN_NAME -t nova sleep 1 if [ "$SCREEN_STATUS" != "0" ]; then screen -r "$SCREEN_NAME" -X hardstatus alwayslastline "%-Lw%{= BW}%50>%n%f* %t%{-}%+Lw%< %= %H" fi
Based on the environment variables set above, the script writes the nova flags to nova.conf and creates the nova folder in /etc/.
cat >$NOVA_DIR/bin/nova.conf << NOVA_CONF_EOF --verbose --nodaemon --dhcpbridge_flagfile=$NOVA_DIR/bin/nova.conf --network_manager=nova.network.manager.$NET_MAN --my_ip=$HOST_IP --public_interface=$INTERFACE --vlan_interface=$INTERFACE --sql_connection=$SQL_CONN --auth_driver=nova.auth.$AUTH --libvirt_type=$LIBVIRT_TYPE --fixed_range=$FIXED_RANGE --lock_path=$LOCK_PATH --instances_path=$INSTANCES_PATH --flat_network_bridge=br100 NOVA_CONF_EOF if [ -n "$FLAT_INTERFACE" ]; then echo "--flat_interface=$FLAT_INTERFACE" >>$NOVA_DIR/bin/nova.conf fi if [ "$USE_IPV6" == 1 ]; then echo "--use_ipv6" >>$NOVA_DIR/bin/nova.conf fi
Next, all dnsmasq processes are killed: DNS cache proxy + DHCP server.
killall dnsmasq
In case IPv6 support is enabled, radvd is killed:
if [ "$USE_IPV6" == 1 ]; then killall radvd fi
The script recreates the database “nova”.
if [ "$USE_MYSQL" == 1 ]; then mysql -p$MYSQL_PASS -e 'DROP DATABASE nova;' mysql -p$MYSQL_PASS -e 'CREATE DATABASE nova;' else rm $NOVA_DIR/nova.sqlite fi
If you decided to use LDAP, OpenLDAP or OpenDJ needs to be configured:
if [ "$USE_LDAP" == 1 ]; then if [ "$USE_OPENDJ" == 1 ]; then echo '--ldap_user_dn=cn=Directory Manager' >> \ /etc/nova/nova-manage.conf sudo $NOVA_DIR/nova/auth/opendj.sh else sudo $NOVA_DIR/nova/auth/slap.sh fi fi
The script also recreates the instances and networks folders.
rm -rf $NOVA_DIR/instances mkdir -p $NOVA_DIR/instances rm -rf $NOVA_DIR/networks mkdir -p $NOVA_DIR/networks
If test mode is enabled (see above), the unit tests are run:
if [ "$TEST" == 1 ]; then cd $NOVA_DIR python $NOVA_DIR/run_tests.py cd $DIR fi
A new database is created
$NOVA_DIR/bin/nova-manage db sync
A new admin user is added:
$NOVA_DIR/bin/nova-manage user admin admin admin admin
A new project “admin” managed by “admin” is created:
$NOVA_DIR/bin/nova-manage project create admin admin
A small network is created with 32 IPs from the fixed range:
$NOVA_DIR/bin/nova-manage network create private $FIXED_RANGE 1 32
Create some floating IPs using the floating range.
$NOVA_DIR/bin/nova-manage floating create $FLOATING_RANGE
Download an image from ansolabs and untar it in the images dir.
if [ ! -d $DIR/images ]; then mkdir -p $DIR/images wget -c http://images.ansolabs.com/tty.tgz tar -C $DIR/images -zxf tty.tgz fi
If ami-tty image in images service then convert the image in directory from the old (Bexar) format to the new format.
if ! glance details | grep ami-tty; then $NOVA_DIR/bin/nova-manage image convert $DIR/images fi
The file novarc looks like this for me. I am running things on Amazon EC2 right now. 10.240.95.3 is my EC2 instance private IP. The EC2 API server is listening on port 8773 and the OpenStack API is listening on port 8774. The data store will be running on port 3333.
The different servers, controllers and stores are started and we can browse them through the screen session created above. See Nova Concepts and Introduction for more details on the Nova architecture.
screen_it api "$NOVA_DIR/bin/nova-api" screen_it objectstore "$NOVA_DIR/bin/nova-objectstore" screen_it compute "$NOVA_DIR/bin/nova-compute" screen_it network "$NOVA_DIR/bin/nova-network" screen_it scheduler "$NOVA_DIR/bin/nova-scheduler" screen_it volume "$NOVA_DIR/bin/nova-volume" screen_it ajax_console_proxy "$NOVA_DIR/bin/nova-ajax-console-proxy" sleep 2 $NOVA_DIR/bin/nova-manage project zipfile admin admin $NOVA_DIR/nova.zip unzip -o $NOVA_DIR/nova.zip -d $NOVA_DIR/ screen_it test "export PATH=$NOVA_DIR/bin:$PATH;. $NOVA_DIR/novarc" if [ "$CMD" != "run_detached" ]; then screen -S nova -x fi
NOVA_KEY_DIR=$(pushd $(dirname $BASH_SOURCE)>/dev/null; pwd; popd>/dev/null) export EC2_ACCESS_KEY="admin:admin" export EC2_SECRET_KEY="admin" export EC2_URL="http://10.240.95.3:8773/services/Cloud" export S3_URL="http://10.240.95.3:3333" export EC2_USER_ID=42 # nova does not use user id, but bundling requires it export EC2_PRIVATE_KEY=${NOVA_KEY_DIR}/pk.pem export EC2_CERT=${NOVA_KEY_DIR}/cert.pem export NOVA_CERT=${NOVA_KEY_DIR}/cacert.pem export EUCALYPTUS_CERT=${NOVA_CERT} # euca-bundle-image seems to require this set alias ec2-bundle-image="ec2-bundle-image --cert ${EC2_CERT} --privatekey ${EC2_PRIVATE_KEY} --user 42 --ec2cert ${NOVA_CERT}" alias ec2-upload-bundle="ec2-upload-bundle -a ${EC2_ACCESS_KEY} -s ${EC2_SECRET_KEY} --url ${S3_URL} --ec2cert ${NOVA_CERT}" export CLOUD_SERVERS_API_KEY="admin" export CLOUD_SERVERS_USERNAME="admin" export CLOUD_SERVERS_URL="http://10.240.95.3:8774/v1.0/"
Once the nova session is finished, the instances are terminated and the volumes deleted. This code is also run when the command “terminate” is used.
if [ "$CMD" == "run" ] || [ "$CMD" == "terminate" ]; then # shutdown instances . $NOVA_DIR/novarc; euca-describe-instances | grep i- | cut -f2 | xargs euca-terminate-instances sleep 2 # delete volumes . $NOVA_DIR/novarc; euca-describe-volumes | grep vol- | cut -f2 | xargs -n1 euca-delete-volume sleep 2 fi
The screen session is forced to shutdown. This part is also called when the command “clean” is executed.
if [ "$CMD" == "run" ] || [ "$CMD" == "clean" ]; then screen -S nova -X quit rm *.pid* fi
There is one last command called “scrub” which is used to remove the bridges and VLANs configuration. It also destroys the domains.
if [ "$CMD" == "scrub" ]; then $NOVA_DIR/tools/clean-vlans if [ "$LIBVIRT_TYPE" == "uml" ]; then virsh -c uml:///system list | grep i- | awk '{print \$1}' | xargs -n1 virsh -c uml:///system destroy else virsh list | grep i- | awk '{print \$1}' | xargs -n1 virsh destroy fi fi
When you run the branch, install and run commands, you end up with the different components (API server, object store, scheduler…) running which is really neat.
You can use the screen windows to check each component status. Do a “man screen” to learn how to navigate between the different windows if you are not familiar with the tool screen.
API server:
# /opt/novascript/trunk/bin/nova-api (nova.root 2011.1-LOCALBRANCH:LOCALREVISION): AUDIT [N/A] Starting /opt/novascript/trunk/bin/nova-api on 0.0.0.0:8774 (nova.root 2011.1-LOCALBRANCH:LOCALREVISION): AUDIT [N/A] Starting /opt/novascript/trunk/bin/nova-api on 0.0.0.0:8773
Object store:
(nova.root 2011.1-LOCALBRANCH:LOCALREVISION): DEBUG [N/A] network_topic : network from MainProcess (pid=13763) serve /opt/novascript/trunk/nova/twistd.py:266 (nova.root 2011.1-LOCALBRANCH:LOCALREVISION): AUDIT [N/A] Starting nova-objectstore 2011-01-14 22:15:09+0000 [-] Log opened. 2011-01-14 22:15:09+0000 [-] twistd 10.0.0 (/usr/bin/python 2.6.5) starting up. 2011-01-14 22:15:09+0000 [-] reactor class: twisted.internet.selectreactor.SelectReactor. 2011-01-14 22:15:09+0000 [-] twisted.web.server.Site starting on 3333 2011-01-14 22:15:09+0000 [-] Starting factory <twisted.web.server.Site instance at 0xaaf5bec>
…
You can launch a new instance this way:
cd /tmp/ euca-add-keypair test > test.pem euca-run-instances -k test -t m1.tiny ami-tty
You can list your instances this way:
# euca-describe-instances RESERVATION r-ky6gm38t admin INSTANCE i-00000001 ami-tty 10.0.0.3 10.0.0.3 running test (admin, domU-12-31-39-04-58-F5) 0 m1.tiny 2011-01-14 22:32:03.466420 nova
Running ifconfig, we can now see the bridge and VLAN created
br100 Link encap:Ethernet HWaddr 12:31:39:04:58:f5 inet addr:10.0.0.1 Bcast:10.0.0.31 Mask:255.255.255.224 ... vlan100 Link encap:Ethernet HWaddr 12:31:39:04:58:f5 inet6 addr: fe80::1031:39ff:fe04:58f5/64 Scope:Link ...
A network route has been set up for us to use the bridge interface when the packets are sent to 10.0.0.x:
# route -n Kernel IP routing table Destination Gateway Genmask Flags Metric Ref Use Iface 10.0.0.0 0.0.0.0 255.255.255.224 U 0 0 0 br100 ...
VoilĂ . Don’t hesitate to post a comment if you have any feedback.
Comments
Thanks so much, really appreciated!!
thanks Laurent !!
Thanks from me too, brilliant!!!
Thanks!!! Very very much!
Superb post Laurent, you save me a lot of time figuring out how the script works!
Excelent Post. There have been a few updates to nova.sh recently. Most notably, it uses a smaller image called ami-tty. It apt-gets a few more needed packages, and you can also specify the directory to use with run after the run argument: sudo ./nova.sh run some-dir will run from that dir. This is much easier than the previous sudo ./nova.sh run “” some-dir also, it might be good to mention that you will need to sudo ./nova.sh run
@Vish Ishaya: Thanks. I updated the article based on your feedback and based on the latest nova.sh script. I am now watching your project on GitHub so I will make sure the article stays up-to-date.
Comments are closed.