1 == How To Install Ganeti Clusters and Instances ==
5 Suppose that there are two identical hosts: foo.debian.org and bar.debian.org.
7 They are running squeeze and have been integrated into Debian infrastructure.
9 They will serve as nodes in a ganeti cluster named foobar.debian.org.
11 They have a RAID1 array exposing three partitions: c0d0p1 for /, c0d0p2 for
12 swap and c0d0p3 for lvm volume groups to be used by ganeti via drbd.
14 They have two network interfaces: eth0 (public) and eth1 (private).
16 The public network is A.B.C.0/24 with gateway A.B.C.254.
18 The private network is E.F.G.0/24 with no gateway.
20 Suppose that the first instance to be hosted on foobar.debian.org is
23 The following DNS records exist:
26 foobar.debian.org. IN A A.B.C.1
27 foo.debian.org. IN A A.B.C.2
28 bar.debian.org. IN A A.B.C.3
29 qux.debian.org. IN A A.B.C.4
30 foo.debprivate-hoster.debian.org. IN A E.F.G.2
31 bar.debprivate-hoster.debian.org. IN A E.F.G.3
34 === install required packages ===
36 On each node, install the required packages:
39 # maybe: apt-get install drbd8-utils
40 # maybe: apt-get install ganeti-instance-debootstrap
41 apt-get install ganeti2 ganeti-htools qemu-kvm
44 === configure kernel modules ===
46 On each node, ensure that the required kernel modules are loaded at boot:
49 ainsl /etc/modules 'drbd minor_count=255 usermode_helper=/bin/true'
50 ainsl /etc/modules 'hmac'
51 ainsl /etc/modules 'tun'
52 ainsl /etc/modules 'ext3'
53 ainsl /etc/modules 'ext4'
56 === configure networking ===
58 On each node, ensure that br0 (not eth0) and eth1 are configured.
60 The bridge interface, br0, is used by the guest virtual machines to reach the
63 If the guest virtual machines need to access the private network, then br1
64 should be configured rather than eth1.
66 To prevent the link address changing due to startup/shutdown of virtual
67 machines, explicitly set the value.
69 This is the interfaces file for foo.debian.org:
80 up ip link set addr $(cat /sys/class/net/$IFACE/address) dev $IFACE
83 iface eth1 inet static
88 This is the interfaces file for bar.debian.org:
99 up ip link set addr $(cat /sys/class/net/$IFACE/address) dev $IFACE
102 iface eth1 inet static
104 netmask 255.255.255.0
107 === configure lvm ===
109 On each node, configure lvm to ignore drbd devices and to prefer
110 {{{/dev/cciss}}} devices names over {{{/dev/block}}} device names
111 ([[http://code.google.com/p/ganeti/issues/detail?id=93|why?]]):
115 -e 's#^\(\s*filter\s\).*#\1= [ "a|.*|", "r|/dev/drbd[0-9]+|" ]#' \
116 -e 's#^\(\s*preferred_names\s\).*#\1= [ "^/dev/dm-*/", "^/dev/cciss/" ]#' \
121 === create lvm volume groups ===
123 On each node, create a volume group:
126 vgcreate vg_ganeti /dev/cciss/c0d0p3
129 === exchange ssh keys ===
134 mkdir -m 0700 -p /root/.ssh &&
135 ln -s /etc/ssh/ssh_host_rsa_key /root/.ssh/id_rsa
138 === configure iptables (via ferm) ===
140 the nodes must connect to each other over the public and private networks for a number of reasons; see the ganeti2 module in puppet
142 === instantiate the cluster ===
144 On the master node (foo) only:
148 --master-netdev br0 \
149 --vg-name vg_ganeti \
150 --secondary-ip E.F.G.2 \
151 --enabled-hypervisors kvm \
152 --nic-parameters link=br0 \
153 --mac-prefix 00:16:37 \
156 --hypervisor-parameters kvm:initrd_path=,kernel_path= \
162 * the master network device is set to br0, matching the public network bridge interface created above
163 * the volume group is set to vg_ganeti, matching the volume group created above
164 * the secondary IP address is set to the value of the master node's interface on the private network
165 * the nic parameters for instances is set to use br0 as default bridge
166 * the MAC prefix is registered in the dsa-kvm git repo
168 === add slave nodes ===
170 For each slave node (only bar for this example):
172 on the slave, append the master's /etc/ssh/ssh_host_rsa_key.pub to
173 /etc/ssh/userkeys/root. This is only required temporarily - once
174 everything works, puppet will put it/keep it there.
176 on the master node (foo):
180 --secondary-ip E.F.G.3 \
188 * the secondary IP address is set to the value of the slave node's interface on the private network
190 === verify cluster ===
192 On the master node (foo):
198 If everything has been configured correctly, no errors should be reported.
200 === create the 'dsa' variant ===
202 Ensure that the debootstrap+dsa variant has been installed by puppet (see ganeti2 module in dsa-puppet).
206 == How To Install Ganeti Instances ==
208 Suppose that qux.debian.org will be an instance (a virtual machine) hosted on
209 the foobar.debian.org ganeti cluster.
211 Before adding the instance, an LDAP entry must be created so that an A record
212 for the instance (A.B.C.4) exists.
214 === create the instance ===
216 On the master node (foo):
221 --disk-template drbd \
223 --os-type debootstrap+dsa \
224 --hypervisor-parameters kvm:initrd_path=,kernel_path= \
231 * the primary and secondary nodes have been explicitly set
232 * the operating system type is 'debootstrap+dsa'
233 * the network interfarce 0 (eth0 on the system) is set to the instance's interface on the public network
234 * If qux.d.o does not yet exist in DNS/LDAP, you may need --no-ip-check --no-name-check. Be careful that the hostname and IP address are not taken already!
240 If the instances require access to the private network, then there are two modifications necessary.
242 === re-configure networking ===
244 On the nodes, ensure that br1 is configured (rather than eth1).
246 This is the interfaces file for foo.debian.org:
250 iface br0 inet static
255 netmask 255.255.255.0
257 up ip link set addr $(cat /sys/class/net/$IFACE/address) dev $IFACE
260 iface br1 inet static
265 netmask 255.255.255.0
266 up ip link set addr $(cat /sys/class/net/$IFACE/address) dev $IFACE
269 This is the interfaces file for bar.debian.org:
273 iface br0 inet static
278 netmask 255.255.255.0
280 up ip link set addr $(cat /sys/class/net/$IFACE/address) dev $IFACE
283 iface br1 inet static
288 netmask 255.255.255.0
289 up ip link set addr $(cat /sys/class/net/$IFACE/address) dev $IFACE
292 === create or update the instance ===
294 When creating the instance, indicate both networks:
299 --disk-template drbd \
301 --os-type debootstrap+dsa \
302 --hypervisor-parameters kvm:initrd_path=,kernel_path= \
304 --net 1:link=br1,ip=E.F.G.4 \
308 * If qux.d.o does not yet exist in DNS/LDAP, you may need --no-ip-check --no-name-check. Be careful that the hostname and IP address are not taken already!
310 When updating an existing instance, add the interface:
313 gnt-instance shutdown qux.debian.org
314 gnt-instance modify \
315 --net add:link=br1,ip=E.F.G.4 \
317 gnt-instance startup qux.debian.org
320 Please note that the hook scripts are run only at instance instantiation. When
321 adding interfaces to an instance, the guest opearting system must be updated
325 * If you are importing an instance from libvirt with LVM setup, you can adopt LVs:
328 gnt-instance add -t plain --os-type debootstrap+dsa-wheezy \
329 --disk 0:adopt=lully-boot \
330 --disk 1:adopt=lully-root \
331 --disk 2:adopt=lully-swap \
332 --disk 3:adopt=lully-log \
333 --hypervisor-parameters kvm:initrd_path=,kernel_path= \
334 --net 0:ip=82.195.75.99 -n clementi.debian.org lully.debian.org
337 And you want to convert it to use DRBD afterwards and start it on the other cluster node, so we can ensure that DRBD is correctly working.
339 gnt-instance shutdown lully.debian.org
340 gnt-instance modify -t drbd -n czerny.debian.org lully.debian.org
341 gnt-instance failover lully.debian.org
342 gnt-instance startup lully.debian.org
345 * Some instances NEED ide instead of virtio
348 gnt-instance modify --hypervisor-parameters disk_type=ide fils.debian.org
351 * To import instances with SAN volumes
354 gnt-instance add -t blockdev --os-type debootstrap+dsa \
355 --disk 0:adopt=/dev/disk/by-id/scsi-reger-boot \
356 --disk 1:adopt=/dev/disk/by-id/scsi-reger-root \
357 --hypervisor-parameters kvm:initrd_path=,kernel_path= \
358 --net 0:ip=206.12.19.124 -n rossini.debian.org reger.debian.org
361 * How to add new LUNs to Bytemark Cluster
363 ** Add new LUN to MSA and export to all blades
366 Log into MSA controller
368 Choose which vdisk to use, use "show vdisks" to list
371 # create volume vdisk msa2k-2-500gr10 size 5G donizetti
376 or (if we assume they are all the same)
377 # show host-maps 3001438001287090
379 Make a note of the next free LUN
381 Generate map commands for all blades, all ports, run locally:
383 $ for bl in 1 2 3 4 5 6 ; do for p in 1 2 3 4; do echo "map volume donizetti lun 27 host bm-bl$bl-p$p" ; done ; done
385 Paste the output into the MSA shell
387 Find the WWN by doing show host-maps and looking for the volume name.
388 Transform it using the sed run at the top of /etc/multipath.conf:
390 echo "$WWN" | sed -re 's#(.{6})(.{6})0000(.{2})(.*)#36\1000\2\3\4#'
396 gnt-cluster command "echo 1 > /sys/bus/pci/devices/0000:0e:00.0/cciss0/rescan"
399 reload multipath-tools on gnt-master (normaly bm-bl1):
400 service multipath-tools reload
401 add the WWNs to dsa-puppet/modules/multipath/files/multipath-bm.conf and define the alias and commit that file to git.
404 gnt-cluster command "puppet agent -t"
406 This will update the multipath config on all cluster nodes. WITHOUT doing this, you can't migrate VMs between nodes.