« back to all changes in this revision
Viewing changes to doc/source/adminguide/multi.node.install.rst
- browse files at revision 468.2.1
- compare with another revision
- compare with revision 1511
- download diff
- download tarball
- view history from revision 468.2.1
- Committer: Anne Gentle
- Date: 2010-12-17 22:53:27 UTC
- mto: This revision was merged to the branch mainline in revision 550.
- Revision ID: anne@openstack.org-20101217225327-cxaqedk6oo51ei5a
Adding new install script plus changes to multinode install doc
- files added:
- contrib/nova-CC-installer.sh
- files modified:
- doc/source/adminguide/multi.node.install.rst
added
removed
doc/source/adminguide/multi.node.install.rst
1
1
..
2
Copyright 2010 United States Government as represented by the
3
Administrator of the National Aeronautics and Space Administration.
2
Copyright 2010 OpenStack LLC
3
4
4
All Rights Reserved.
5
5
6
6
Licensed under the Apache License, Version 2.0 (the "License"); you may
16
16
under the License.
17
17
18
18
Installing Nova on Multiple Servers
19
19
20
===================================
20
21
21
22
When you move beyond evaluating the technology and into building an actual
22
23
production environment, you will need to know how to configure your datacenter
23
24
and how to deploy components across your clusters. This guide should help you
24
25
through that process.
25
26
26
27
You can install multiple nodes to increase performance and availability of the OpenStack Compute installation.
27
28
28
29
This setup is based on an Ubuntu Lucid 10.04 installation with the latest updates. Most of this works around issues that need to be resolved in the installation and configuration scripts as of October 18th 2010. It also needs to eventually be generalized, but the intent here is to get the multi-node configuration bootstrapped so folks can move forward.
29
30
30
31
31
32
Requirements for a multi-node installation
32
33
------------------------------------------
33
34
34
35
* You need a real database, compatible with SQLAlchemy (mysql, postgresql) There's not a specific reason to choose one over another, it basically depends what you know. MySQL is easier to do High Availability (HA) with, but people may already know Postgres. We should document both configurations, though.
35
36
* For a recommended HA setup, consider a MySQL master/slave replication, with as many slaves as you like, and probably a heartbeat to kick one of the slaves into being a master if it dies.
36
37
* For performance optimization, split reads and writes to the database. MySQL proxy is the easiest way to make this work if running MySQL.
37
38
38
39
Assumptions
39
^^^^^^^^^^^
40
40
------------------------------------
41
41
42
* Networking is configured between/through the physical machines on a single subnet.
42
* Installation and execution are both performed by root user.
43
44
45
46
Step 1 Use apt-get to get the latest code
43
* Installation and execution are both performed by ROOT user.
44
45
46
Step 1 - Use apt-get to get the latest code
47
47
-----------------------------------------
48
49
1. Setup Nova PPA with https://launchpad.net/~nova-core/+archive/ppa.
50
51
::
52
53
sudo apt-get install python-software-properties
54
sudo add-apt-repository ppa:nova-core/ppa
55
56
2. Run update.
57
58
::
59
60
sudo apt-get update
61
62
3. Install nova-pkgs (dependencies should be automatically installed).
63
64
::
65
66
sudo apt-get install python-greenlet
67
sudo apt-get install nova-common nova-doc python-nova nova-api nova-network nova-objectstore nova-scheduler
68
69
It is highly likely that there will be errors when the nova services come up since they are not yet configured. Don't worry, you're only at step 1!
70
71
Step 2 Setup configuration file (installed in /etc/nova)
48
49
1. Setup Nova PPA with https://launchpad.net/~nova-core/+archive/trunk. The ‘python-software-properties’ package is a pre-requisite for setting up the nova package repo:
50
51
::
52
53
apt-get -y install python-software-properties
54
add-apt-repository ppa:nova-core/trunk
55
56
2. Update apt-get:
57
58
::
59
60
apt-get update
61
62
3. Install nova-packages (dependencies should be automatically installed).
63
64
::
65
66
apt-get -y install bzr nova-common nova-doc python-mysqldb python-greenlet python-nova nova-api nova-network nova-objectstore nova-scheduler nova-compute unzip vim euca2ools rabbitmq-server dnsmasq open-iscsi kpartx kvm gawk iptables ebtables user-mode-linux kvm libvirt-bin screen iscsitarget euca2ools vlan curl python-twisted python-sqlalchemy python-mox python-greenlet python-carrot python-daemon python-eventlet python-gflags python-libvirt python-libxml2 python-routes
67
68
Step 2 – Setting up nova.conf (installed in /etc/nova)
72
69
---------------------------------------------------------
73
74
Note: CC_ADDR=<the external IP address of your cloud controller>
75
76
Nova development has consolidated all .conf files to nova.conf as of November 2010. References to specific .conf files may be ignored.
77
78
#. These need to be defined in the nova.conf configuration file::
79
80
--sql_connection=mysql://root:nova@$CC_ADDR/nova # location of nova sql db
81
--s3_host=$CC_ADDR # This is where nova is hosting the objectstore service, which
82
# will contain the VM images and buckets
83
--rabbit_host=$CC_ADDR # This is where the rabbit AMQP messaging service is hosted
84
--cc_host=$CC_ADDR # This is where the the nova-api service lives
85
--verbose # Optional but very helpful during initial setup
86
--ec2_url=http://$CC_ADDR:8773/services/Cloud
87
--network_manager=nova.network.manager.FlatManager # simple, no-vlan networking type
88
89
--fixed_range=<network/prefix> # ip network to use for VM guests, ex 192.168.2.64/26
90
--network_size=<# of addrs> # number of ip addrs to use for VM guests, ex 64
91
92
#. Create a nova group::
93
94
sudo addgroup nova
95
96
The Nova config file should have its owner set to root:nova, and mode set to 0640, since they contain your MySQL server's root password.
97
98
::
99
100
cd /etc/nova
101
chown -R root:nova .
102
103
Step 3 Setup the sql db
70
1. Nova development has consolidated all config files to nova.conf as of November 2010. There is a default set of options that are already configured in nova.conf:
71
72
::
73
74
--daemonize=1
75
--dhcpbridge_flagfile=/etc/nova/nova.conf
76
--dhcpbridge=/usr/bin/nova-dhcpbridge
77
--logdir=/var/log/nova
78
--state_path=/var/lib/nova
79
80
The following items ALSO need to be defined in /etc/nova/nova.conf. I’ve added some explanation of the variables, as comments CANNOT be in nova.conf. There seems to be an issue with nova-manage not processing the comments/whitespace correctly:
81
82
--sql_connection ### Location of Nova SQL DB
83
84
--s3_host ### This is where Nova is hosting the objectstore service, which will contain the VM images and buckets
85
86
--rabbit_host ### This is where the rabbit AMQP messaging service is hosted
87
88
--cc_host ### This is where the the nova-api service lives
89
90
--verbose ### Optional but very helpful during initial setup
91
92
--ec2_url ### The location to interface nova-api
93
94
--network_manager ### Many options here, discussed below. This is how your controller will communicate with additional Nova nodes and VMs:
95
96
nova.network.manager.FlatManager # Simple, no-vlan networking type
97
nova.network.manager. FlatDHCPManager # Flat networking with DHCP
98
nova.network.manager.VlanManager # Vlan networking with DHCP – /DEFAULT/ if no network manager is defined in nova.conf
99
100
--fixed_range=<network/prefix> ### This will be the IP network that ALL the projects for future VM guests will reside on. E.g. 192.168.0.0/12
101
102
--network_size=<# of addrs> ### This is the total number of IP Addrs to use for VM guests, of all projects. E.g. 5000
103
104
The following code can be cut and paste, and edited to your setup:
105
106
## Note: CC_ADDR=<the external IP address of your cloud controller>##
107
## Detailed explanation of the following entries are right above this ##
108
109
::
110
111
--sql_connection=mysql://root:nova@<CC_ADDR>/nova
112
--s3_host=<CC_ADDR>
113
--rabbit_host=<CC_ADDR>
114
--cc_host=<CC_ADDR>
115
--verbose
116
--ec2_url=http://<CC_ADDR>:8773/services/Cloud
117
--network_manager=nova.network.manager.VlanManager
118
--fixed_range=<network/prefix>
119
--network_size=<# of addrs>
120
121
2. Create a “nova” group, and set permissions:
122
123
::
124
125
addgroup nova
126
127
The Nova config file should have its owner set to root:nova, and mode set to 0644, since they contain your MySQL server's root password.
128
129
::
130
131
chown -R root:nova /etc/nova
132
chmod 644 /etc/nova/nova.conf
133
134
Step 3 - Setup the SQL DB (MySQL for this setup)
104
135
-----------------------
105
106
1. First you 'preseed' (using the Quick Start method :doc:`../quickstart`). Run this as root.
107
108
::
109
110
sudo apt-get install bzr git-core
111
sudo bash
112
export MYSQL_PASS=nova
113
114
115
::
116
117
cat <<MYSQL_PRESEED | debconf-set-selections
118
mysql-server-5.1 mysql-server/root_password password $MYSQL_PASS
119
mysql-server-5.1 mysql-server/root_password_again password $MYSQL_PASS
120
mysql-server-5.1 mysql-server/start_on_boot boolean true
121
MYSQL_PRESEED
122
123
2. Install mysql
124
125
::
126
127
sudo apt-get install -y mysql-server
128
129
4. Edit /etc/mysql/my.cnf and set this line: bind-address=0.0.0.0 and then sighup or restart mysql
130
131
5. create nova's db
132
133
::
134
135
mysql -uroot -pnova -e 'CREATE DATABASE nova;'
136
137
138
6. Update the db to include user 'root'@'%'
139
140
::
141
142
mysql -u root -p nova
143
GRANT ALL PRIVILEGES ON *.* TO 'root'@'%' WITH GRANT OPTION;
144
SET PASSWORD FOR 'root'@'%' = PASSWORD('nova');
145
146
7. Branch and install Nova
147
148
::
149
150
sudo -i
151
cd ~
152
export USE_MYSQL=1
153
export MYSQL_PASS=nova
154
git clone https://github.com/vishvananda/novascript.git
155
cd novascript
156
./nova.sh branch
157
./nova.sh install
158
./nova.sh run
159
160
Step 4 Setup Nova environment
161
-----------------------------
162
163
::
164
165
/usr/bin/python /usr/bin/nova-manage user admin <user_name>
166
/usr/bin/python /usr/bin/nova-manage project create <project_name> <user_name>
167
/usr/bin/python /usr/bin/nova-manage project create network
168
169
Note: The nova-manage service assumes that the first IP address is your network (like 192.168.0.0), that the 2nd IP is your gateway (192.168.0.1), and that the broadcast is the very last IP in the range you defined (192.168.0.255). If this is not the case you will need to manually edit the sql db 'networks' table.o.
170
171
On running this command, entries are made in the 'networks' and 'fixed_ips' table. However, one of the networks listed in the 'networks' table needs to be marked as bridge in order for the code to know that a bridge exists. The Network is marked as bridged automatically based on the type of network manager selected.
172
173
More networking details to create a network bridge for flat network
174
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
175
176
Nova defaults to a bridge device named 'br100'. This needs to be created and somehow integrated into YOUR network. In my case, I wanted to keep things as simple as possible and have all the vm guests on the same network as the vm hosts (the compute nodes). Thus, I set the compute node's external IP address to be on the bridge and added eth0 to that bridge. To do this, edit your network interfaces config to look like the following::
177
136
137
1. First you 'preseed' to bypass all the installation prompts
138
::
139
140
bash
141
MYSQL_PASS=nova
142
cat <<MYSQL_PRESEED | debconf-set-selections
143
mysql-server-5.1 mysql-server/root_password password $MYSQL_PASS
144
mysql-server-5.1 mysql-server/root_password_again password $MYSQL_PASS
145
mysql-server-5.1 mysql-server/start_on_boot boolean true
146
MYSQL_PRESEED
147
148
2. Install MySQL:
149
150
::
151
152
apt-get install -y mysql-server
153
154
3. Edit /etc/mysql/my.cnf to change ‘bind-address’ from localhost to any:
155
156
::
157
158
sed -i 's/127.0.0.1/0.0.0.0/g' /etc/mysql/my.cnf
159
service mysql restart
160
161
3. Network Configuration
162
163
If you use FlatManager (as opposed to VlanManager that we set) as your network manager, there are some additional networking changes you’ll have to make to ensure connectivity between your nodes and VMs. If you chose VlanManager or FlatDHCP, you may skip this section, as it’s set up for you automatically.
164
165
Nova defaults to a bridge device named 'br100'. This needs to be created and somehow integrated into YOUR network. To keep things as simple as possible, have all the VM guests on the same network as the VM hosts (the compute nodes). To do so, set the compute node's external IP address to be on the bridge and add eth0 to that bridge. To do this, edit your network interfaces config to look like the following
166
167
::
168
178
169
< begin /etc/network/interfaces >
179
170
# The loopback network interface
180
171
auto lo
181
172
iface lo inet loopback
182
173
183
174
# Networking for NOVA
184
175
auto br100
185
176
186
177
iface br100 inet dhcp
187
178
bridge_ports eth0
188
179
bridge_stp off
189
180
bridge_maxwait 0
190
181
bridge_fd 0
191
182
< end /etc/network/interfaces >
192
193
183
184
194
185
Next, restart networking to apply the changes::
195
196
sudo /etc/init.d/networking restart
197
198
Step 5: Create nova certs.
186
187
sudo /etc/init.d/networking restart
188
189
4. MySQL DB configuration:
190
191
Create NOVA database:
192
193
::
194
195
mysql -uroot -p$MYSQL_PASS -e 'CREATE DATABASE nova;'
196
197
198
Update the DB to include user 'root'@'%' with super user privileges
199
200
::
201
202
mysql -uroot -p$MYSQL_PASS -e "GRANT ALL PRIVILEGES ON *.* TO 'root'@'%' WITH GRANT OPTION;"
203
204
Set mySQL root password
205
206
::
207
208
mysql -uroot -p$MYSQL_PASS -e "SET PASSWORD FOR 'root'@'%' = PASSWORD('$MYSQL_PASS');"
209
210
211
Step 4 - Setup Nova environment
212
-----------------------------
213
214
::
215
216
/usr/bin/python /usr/bin/nova-manage user admin <user_name>
217
/usr/bin/python /usr/bin/nova-manage project create <project_name> <user_name>
218
/usr/bin/python /usr/bin/nova-manage network create <project-network> <number-of-networks-in-project> <IPs in project>
219
220
Here is an example of what this looks like with real data:
221
222
/usr/bin/python /usr/bin/nova-manage user admin dub
223
/usr/bin/python /usr/bin/nova-manage project create dubproject dub
224
/usr/bin/python /usr/bin/nova-manage network create 192.168.0.0/24 1 255
225
226
(I chose a /24 since that falls inside my /12 range I set in ‘fixed-range’ in nova.conf. Currently, there can only be one network, and I am using the max IP’s available in a /24. You can choose to use any valid amount that you would like.)
227
228
Note: The nova-manage service assumes that the first IP address is your network (like 192.168.0.0), that the 2nd IP is your gateway (192.168.0.1), and that the broadcast is the very last IP in the range you defined (192.168.0.255). If this is not the case you will need to manually edit the sql db 'networks' table.o.
229
230
On running this command, entries are made in the 'networks' and 'fixed_ips' table. However, one of the networks listed in the 'networks' table needs to be marked as bridge in order for the code to know that a bridge exists. The Network is marked as bridged automatically based on the type of network manager selected. This is ONLY necessary if you chose FlatManager as your network type. More information can be found at the end of this document discussing setting up the bridge device.
231
232
233
Step 5 - Create Nova certs
199
234
--------------------------
200
201
Generate the certs as a zip file::
202
203
mkdir creds
204
sudo /usr/bin/python /usr/bin/nova-manage project zip admin admin creds/nova.zip
205
206
you can get the rc file more easily with::
207
208
sudo /usr/bin/python /usr/bin/nova-manage project env admin admin creds/novarc
209
210
unzip them in your home directory, and add them to your environment::
211
212
unzip creds/nova.zip
213
echo ". creds/novarc" >> ~/.bashrc
214
~/.bashrc
215
216
Step 6 Restart all relevant services
217
------------------------------------
218
219
Restart Libvirt::
220
221
sudo /etc/init.d/libvirt-bin restart
222
223
Restart relevant nova services::
224
225
sudo /etc/init.d/nova-compute restart
226
sudo /etc/init.d/nova-volume restart
227
228
229
.. todo:: do we still need the content below?
230
231
Bare-metal Provisioning Notes
232
-----------------------------
233
234
To install the base operating system you can use PXE booting.
235
236
Types of Hosts
237
--------------
238
239
A single machine in your cluster can act as one or more of the following types
240
of host:
241
242
Nova Services
243
244
* Network
245
* Compute
246
* Volume
247
* API
248
* Objectstore
249
250
Other supporting services
251
252
* Message Queue
253
* Database (optional)
254
* Authentication database (optional)
255
256
Initial Setup
257
-------------
258
259
* Networking
260
* Cloudadmin User Creation
261
262
Deployment Technologies
263
-----------------------
264
265
Once you have machines with a base operating system installation, you can deploy
266
code and configuration with your favorite tools to specify which machines in
267
your cluster have which roles:
268
269
* Puppet
270
* Chef
235
236
1. Generate the certs as a zip file. These are the certs you will use to launch instances, bundle images, and all the other assorted api functions:
237
238
::
239
240
mkdir –p /root/creds
241
/usr/bin/python /usr/bin/nova-manage project zipfile $NOVA_PROJECT $NOVA_PROJECT_USER /root/creds/novacreds.zip
242
243
2. Unzip them in your home directory, and add them to your environment:
244
245
::
246
247
unzip /root/creds/novacreds.zip -d /root/creds/
248
cat /root/creds/novarc >> ~/.bashrc
249
source ~/.bashrc
250
251
Step 6 - Restart all relevant services
252
------------------------------------
253
Restart all six services in total, just to cover the entire spectrum:
254
255
::
256
257
libvirtd restart; service nova-network restart; service nova-compute restart; service nova-api restart; service nova-objectstore restart; service nova-scheduler restart
258
259
Step 7 - Closing steps, and cleaning up:
260
------------------------------------
261
262
One of the most commonly missed configuration areas is not allowing the proper access to VMs. Use the 'euca-authorize' command to enable access. Below, you will find the commands to allow 'ping' and 'ssh' to your VMs:
263
264
::
265
266
euca-authorize -P icmp -t -1:-1 default
267
euca-authorize -P tcp -p 22 default
268
269
Another common issue is you cannot ping or SSH your instances after issusing the 'euca-authorize' commands. Something to look at is the amount of 'dnsmasq' processes that are running. If you have a running instance, check to see that TWO 'dnsmasq' processes are running. If not, perform the following:
270
271
::
272
273
killall dnsmasq
274
service nova-network restart
275
276
Step 8 – Testing the installation
277
------------------------------------
278
279
You can then use `euca2ools` to test some items:
280
281
::
282
283
euca-describe-images
284
euca-describe-instances
285
286
If you have issues with the API key, you may need to re-source your creds file:
287
288
::
289
290
. /root/creds/novarc
291
292
If you don’t get any immediate errors, you’re successfully making calls to your cloud!
293
294
The next thing you are going to need is an image to test. There will soon be an update on how to capture an image and use it as a bootable AMI so you can ping, ssh, show instances spinning up, etc.
295
296
Enjoy your new private cloud, and play responsibly!
297
Loggerhead is a web-based interface for Breezy
Version: 2.0.1