1 == How To Install Ganeti Clusters and Instances ==
5 Suppose that there are two identical hosts: foo.debian.org and bar.debian.org.
7 They are running squeeze and have been integrated into Debian infrastructure.
9 They will serve as nodes in a ganeti cluster named foobar.debian.org.
11 They have a RAID1 array exposing three partitions: c0d0p1 for /, c0d0p2 for
12 swap and c0d0p3 for lvm volume groups to be used by ganeti via drbd.
14 They have two network interfaces: eth0 (public) and eth1 (private).
16 The public network is A.B.C.0/24 with gateway A.B.C.254.
18 The private network is E.F.G.0/24 with no gateway.
20 Suppose that the first instance to be hosted on foobar.debian.org is
23 The following DNS records exist:
26 foobar.debian.org. IN A A.B.C.1
27 foo.debian.org. IN A A.B.C.2
28 bar.debian.org. IN A A.B.C.3
29 qux.debian.org. IN A A.B.C.4
30 foo.debprivate-hoster.debian.org. IN A E.F.G.2
31 bar.debprivate-hoster.debian.org. IN A E.F.G.3
34 === install required packages ===
36 On each node, install the required packages:
39 apt-get install fai-client ssed extlinux
40 apt-get install -t squeeze-backports drbd8-utils ganeti2 ganeti-instance-debootstrap ganeti-htools
43 === configure kernel modules ===
45 On each node, ensure that the required kernel modules are loaded at boot:
48 ainsl /etc/modules 'drbd minor_count=255 usermode_helper=/bin/true'
49 ainsl /etc/modules 'hmac'
50 ainsl /etc/modules 'tun'
53 === configure networking ===
55 On each node, ensure that br0 (not eth0) and eth1 are configured.
57 The bridge interface, br0, is used by the guest virtual machines to reach the
60 If the guest virtual machines need to access the private network, then br1
61 should be configured rather than eth1.
63 To prevent the link address changing due to startup/shutdown of virtual
64 machines, explicitly set the value.
67 # /etc/networking/interfaces for foo.debian.org
77 up ip link set addr $(cat /sys/class/net/$IFACE/address) dev $IFACE
80 iface eth1 inet static
86 # /etc/networking/interfaces for bar.debian.org
96 up ip link set addr $(cat /sys/class/net/$IFACE/address) dev $IFACE
99 iface eth1 inet static
101 netmask 255.255.255.0
104 === configure lvm ===
106 On each node, configure lvm to ignore drbd devices and to prefer
107 {{{/dev/cciss}}} devices names over {{{/dev/block}}} device names
108 ([[http://code.google.com/p/ganeti/issues/detail?id=93|why?]]):
112 -e 's#^\(\s*filter\s\).*#\1= [ "a|.*|", "r|/dev/drbd[0-9]+|" ]#' \
113 -e 's#^\(\s*preferred_names\s\).*#\1= [ "^/dev/dm-*/", "^/dev/cciss/" ]#' \
118 === create lvm volume groups ===
120 On each node, create a volume group:
123 vgcreate vg_ganeti /dev/cciss/c0d0p3
126 === create kvmU symbolic links ===
128 On each node, create vmlinuz and initrd symlinks as expected by the kvm
129 hypervisor (not strictly required if all guests will boot from their own
133 ln -s -T /boot/vmlinuz-2.6.32-5-amd64 /boot/vmlinuz-2.6-kvmU
134 ln -s -T /boot/initrd.img-2.6.32-5-amd64 /boot/initrd.img-2.6-kvmU
137 === exchange ssh keys ===
142 mkdir -m 0700 -p /root/.ssh &&
143 ln -s /etc/ssh/ssh_host_rsa_key /root/.ssh/id_rsa
146 === configure iptables (via ferm) ===
148 ipsum loren bacon puppet: the nodes must connect to each other over the private network for drbd purposes; puppet can handle this (poorly; needs improvement)
150 === instantiate the cluster ===
152 On the master node (foo) only:
156 --master-netdev br0 \
157 --vg-name vg_ganeti \
158 --secondary-ip E.F.G.2 \
159 --enabled-hypervisors kvm \
160 --nic-parameters link=br0 \
161 --mac-prefix 00:16:37 \
164 --hypervisor-parameters kvm:initrd_path=/boot/initrd.img-2.6-kvmU,kernel_path=/boot/vmlinuz-2.6-kvmU \
170 * the master network device is set to br0, matching the public network bridge interface created above
171 * the volume group is set to vg_ganeti, matching the volume group created above
172 * the secondary IP address is set to the value of the master node's interface on the private network
173 * the nic parameters for instances is set to use br0 as default bridge
174 * the MAC prefix is registered in the dsa-kvm git repo
176 === add slave nodes ===
178 For each slave node (only bar for this example):
180 on the slave, append the master's /etc/ssh/ssh_host_rsa_key.pub to
181 /etc/ssh/userkeys/root. This is only required temporarily - once
182 everything works puppet will put it/keep it there.
184 on the master node (foo):
188 --secondary-ip E.F.G.3 \
196 * the secondary IP address is set to the value of the slave node's interface on the private network
198 === verify cluster ===
200 On the master node (foo):
206 If everything has been configured correctly, no errors should be reported.
208 === create the 'dsa' variant ===
210 ipsum loren bacon deboostrap+dsa
212 * create /etc/ganeti/instance-debootstrap/variants/dsa.conf
213 * add 'dsa' to /etc/ganeti/instance-debootstrap/variants.list
214 * obtain ??-dsa-* from tristano:/etc/ganeti/instance-debootstrap/hooks
218 == How To Install Ganeti Instances ==
220 Suppose that qux.debian.org will be an instance (a virtual machine) hosted on
221 the foobar.debian.org ganeti cluster.
223 Before adding the instance, an LDAP entry must be created so that an A record
224 for the instance (A.B.C.4) exists.
226 === create the instance ===
228 On the master node (foo):
233 --disk-template drbd \
235 --os-type debootstrap+dsa \
236 --hypervisor-parameters kvm:initrd_path=,kernel_path= \
243 * the primary and secondary nodes have been explicitly set
244 * the operating system type is 'debootstrap+dsa'
245 * the network interfarce 0 (eth1 on the system) is set to the instance's interface on the public network
246 * If qux.d.o does not yet exist in DNS/LDAP, you may need --no-ip-check --no-name-check. Be careful that the hostname and IP address are not taken already!
252 If the instances require access to the private network, then there are two modifications necessary.
254 === re-configure networking ===
256 Ensure that br1 is configured (rather than eth1).
259 # /etc/networking/interfaces for foo.debian.org
262 iface br0 inet static
267 netmask 255.255.255.0
269 up ip link set addr $(cat /sys/class/net/$IFACE/address) dev $IFACE
272 iface br1 inet static
277 netmask 255.255.255.0
278 up ip link set addr $(cat /sys/class/net/$IFACE/address) dev $IFACE
281 === create or update the instance ===
283 When creating the instance, indicate both netwokrs:
288 --disk-template drbd \
290 --os-type debootstrap+dsa \
291 --hypervisor-parameters kvm:initrd_path=,kernel_path= \
297 * If qux.d.o does not yet exist in DNS/LDAP, you may need --no-ip-check --no-name-check. Be careful that the hostname and IP address are not taken already!
299 When updating an existing instance, add the interface:
302 gnt-instance shutdown qux.debian.org
303 gnt-instance modify \
304 --net add:ip=E.F.G.4 \
306 gnt-instance startup qux.debian.org
309 Please note that the hook scripts are run only at instance instantiation. When
310 adding interfaces to an instance, the guest opearting system must be updated