1 == How To Install Ganeti Clusters and Instances ==
5 Suppose that there are two identical hosts: foo.debian.org and bar.debian.org.
7 They are running squeeze and have been integrated into Debian infrastructure.
9 They will serve as nodes in a ganeti cluster named foobar.debian.org.
11 They have a RAID1 array exposing three partitions: c0d0p1 for /, c0d0p2 for
12 swap and c0d0p3 for lvm volume groups to be used by ganeti via drbd.
14 They have two network interfaces: eth0 (public) and eth1 (private).
16 The public network is A.B.C.0/24 with gateway A.B.C.254.
18 The private network is E.F.G.0/24 with no gateway.
20 Suppose that the first instance to be hosted on foobar.debian.org is
23 The following DNS records exist:
26 foobar.debian.org. IN A A.B.C.1
27 foo.debian.org. IN A A.B.C.2
28 bar.debian.org. IN A A.B.C.3
29 qux.debian.org. IN A A.B.C.4
30 foo.debprivate-hoster.debian.org. IN A E.F.G.2
31 bar.debprivate-hoster.debian.org. IN A E.F.G.3
34 === install required packages ===
36 On each node, install the required packages:
39 apt-get install fai-client ssed extlinux
40 apt-get install {drbd8-utils,ganeti2,ganeti-htools}/squeeze-backports ganeti-instance-debootstrap qemu-kvm
43 === configure kernel modules ===
45 On each node, ensure that the required kernel modules are loaded at boot:
48 ainsl /etc/modules 'drbd minor_count=255 usermode_helper=/bin/true'
49 ainsl /etc/modules 'hmac'
50 ainsl /etc/modules 'tun'
51 ainsl /etc/modules 'ext3'
54 === configure networking ===
56 On each node, ensure that br0 (not eth0) and eth1 are configured.
58 The bridge interface, br0, is used by the guest virtual machines to reach the
61 If the guest virtual machines need to access the private network, then br1
62 should be configured rather than eth1.
64 To prevent the link address changing due to startup/shutdown of virtual
65 machines, explicitly set the value.
67 This is the interfaces file for foo.debian.org:
78 up ip link set addr $(cat /sys/class/net/$IFACE/address) dev $IFACE
81 iface eth1 inet static
86 This is the interfaces file for bar.debian.org:
97 up ip link set addr $(cat /sys/class/net/$IFACE/address) dev $IFACE
100 iface eth1 inet static
102 netmask 255.255.255.0
105 === configure lvm ===
107 On each node, configure lvm to ignore drbd devices and to prefer
108 {{{/dev/cciss}}} devices names over {{{/dev/block}}} device names
109 ([[http://code.google.com/p/ganeti/issues/detail?id=93|why?]]):
113 -e 's#^\(\s*filter\s\).*#\1= [ "a|.*|", "r|/dev/drbd[0-9]+|" ]#' \
114 -e 's#^\(\s*preferred_names\s\).*#\1= [ "^/dev/dm-*/", "^/dev/cciss/" ]#' \
119 === create lvm volume groups ===
121 On each node, create a volume group:
124 vgcreate vg_ganeti /dev/cciss/c0d0p3
127 === create kvmU symbolic links ===
129 On each node, create vmlinuz and initrd symlinks as expected by the kvm
130 hypervisor (not strictly required if all guests will boot from their own
134 ln -s -T /boot/vmlinuz-2.6.32-5-amd64 /boot/vmlinuz-2.6-kvmU
135 ln -s -T /boot/initrd.img-2.6.32-5-amd64 /boot/initrd.img-2.6-kvmU
138 === exchange ssh keys ===
143 mkdir -m 0700 -p /root/.ssh &&
144 ln -s /etc/ssh/ssh_host_rsa_key /root/.ssh/id_rsa
147 === configure iptables (via ferm) ===
149 the nodes must connect to each other over the public and private networks for a number of reasons; see the ganeti2 module in puppet
151 === instantiate the cluster ===
153 On the master node (foo) only:
157 --master-netdev br0 \
158 --vg-name vg_ganeti \
159 --secondary-ip E.F.G.2 \
160 --enabled-hypervisors kvm \
161 --nic-parameters link=br0 \
162 --mac-prefix 00:16:37 \
165 --hypervisor-parameters kvm:initrd_path=/boot/initrd.img-2.6-kvmU,kernel_path=/boot/vmlinuz-2.6-kvmU \
171 * the master network device is set to br0, matching the public network bridge interface created above
172 * the volume group is set to vg_ganeti, matching the volume group created above
173 * the secondary IP address is set to the value of the master node's interface on the private network
174 * the nic parameters for instances is set to use br0 as default bridge
175 * the MAC prefix is registered in the dsa-kvm git repo
177 === add slave nodes ===
179 For each slave node (only bar for this example):
181 on the slave, append the master's /etc/ssh/ssh_host_rsa_key.pub to
182 /etc/ssh/userkeys/root. This is only required temporarily - once
183 everything works, puppet will put it/keep it there.
185 on the master node (foo):
189 --secondary-ip E.F.G.3 \
197 * the secondary IP address is set to the value of the slave node's interface on the private network
199 === verify cluster ===
201 On the master node (foo):
207 If everything has been configured correctly, no errors should be reported.
209 === create the 'dsa' variant ===
211 Ensure that the debootstrap+dsa variant has been installed by puppet (see ganeti2 module in dsa-puppet).
215 == How To Install Ganeti Instances ==
217 Suppose that qux.debian.org will be an instance (a virtual machine) hosted on
218 the foobar.debian.org ganeti cluster.
220 Before adding the instance, an LDAP entry must be created so that an A record
221 for the instance (A.B.C.4) exists.
223 === create the instance ===
225 On the master node (foo):
230 --disk-template drbd \
232 --os-type debootstrap+dsa \
233 --hypervisor-parameters kvm:initrd_path=,kernel_path= \
240 * the primary and secondary nodes have been explicitly set
241 * the operating system type is 'debootstrap+dsa'
242 * the network interfarce 0 (eth0 on the system) is set to the instance's interface on the public network
243 * If qux.d.o does not yet exist in DNS/LDAP, you may need --no-ip-check --no-name-check. Be careful that the hostname and IP address are not taken already!
249 If the instances require access to the private network, then there are two modifications necessary.
251 === re-configure networking ===
253 On the nodes, ensure that br1 is configured (rather than eth1).
255 This is the interfaces file for foo.debian.org:
259 iface br0 inet static
264 netmask 255.255.255.0
266 up ip link set addr $(cat /sys/class/net/$IFACE/address) dev $IFACE
269 iface br1 inet static
274 netmask 255.255.255.0
275 up ip link set addr $(cat /sys/class/net/$IFACE/address) dev $IFACE
278 This is the interfaces file for bar.debian.org:
282 iface br0 inet static
287 netmask 255.255.255.0
289 up ip link set addr $(cat /sys/class/net/$IFACE/address) dev $IFACE
292 iface br1 inet static
297 netmask 255.255.255.0
298 up ip link set addr $(cat /sys/class/net/$IFACE/address) dev $IFACE
301 === create or update the instance ===
303 When creating the instance, indicate both networks:
308 --disk-template drbd \
310 --os-type debootstrap+dsa \
311 --hypervisor-parameters kvm:initrd_path=,kernel_path= \
313 --net 1:link=br1,ip=E.F.G.4 \
317 * If qux.d.o does not yet exist in DNS/LDAP, you may need --no-ip-check --no-name-check. Be careful that the hostname and IP address are not taken already!
319 When updating an existing instance, add the interface:
322 gnt-instance shutdown qux.debian.org
323 gnt-instance modify \
324 --net add:link=br1,ip=E.F.G.4 \
326 gnt-instance startup qux.debian.org
329 Please note that the hook scripts are run only at instance instantiation. When
330 adding interfaces to an instance, the guest opearting system must be updated
334 * If you are importing an instance from libvirt with LVM setup, you can adopt LVs:
337 gnt-instance add -t plain --os-type debootstrap+dsa-wheezy \
338 --disk 0:adopt=lully-boot \
339 --disk 1:adopt=lully-root \
340 --disk 2:adopt=lully-swap \
341 --disk 3:adopt=lully-log \
342 --hypervisor-parameters kvm:initrd_path=,kernel_path= \
343 --net 0:ip=82.195.75.99 -n clementi.debian.org lully.debian.org
346 And you want to convert it to use DRBD afterwards and start it on the cluster node, so we can ensure that DRBD is correctly working.
348 gnt-instance shutdown lully.debian.org
349 gnt-instance modify -t drbd -n czerny.debian.org lully.debian.org
350 gnt-instance failover lully.debian.org
351 gnt-instance startup lully.debian.org
354 * Some instances NEED ide instead of virtio
357 gnt-instance modify --hypervisor-parameters disk_type=ide fils.debian.org