1 == How To Install Ganeti Clusters and Instances ==
5 Suppose that there are two identical hosts: foo.debian.org and bar.debian.org.
7 They are running squeeze and have been integrated into Debian infrastructure.
9 They will serve as nodes in a ganeti cluster named foobar.debian.org.
11 They have a RAID1 array exposing three partitions: c0d0p1 for /, c0d0p2 for
12 swap and c0d0p3 for lvm volume groups to be used by ganeti via drbd.
14 They have two network interfaces: eth0 (public) and eth1 (private).
16 The public network is A.B.C.0/24 with gateway A.B.C.254.
18 The private network is E.F.G.0/24 with no gateway.
20 Suppose that the first instance to be hosted on foobar.debian.org is
23 The following DNS records exist:
26 foobar.debian.org. IN A A.B.C.1
27 foo.debian.org. IN A A.B.C.2
28 bar.debian.org. IN A A.B.C.3
29 qux.debian.org. IN A A.B.C.4
30 foo.debprivate-hoster.debian.org. IN A E.F.G.2
31 bar.debprivate-hoster.debian.org. IN A E.F.G.3
34 === install required packages ===
36 On each node, install the required packages:
39 apt-get install fai-client ssed extlinux
40 apt-get install {drbd8-utils,ganeti2,ganeti-htools}/squeeze-backports ganeti-instance-debootstrap qemu-kvm
43 === configure kernel modules ===
45 On each node, ensure that the required kernel modules are loaded at boot:
48 ainsl /etc/modules 'drbd minor_count=255 usermode_helper=/bin/true'
49 ainsl /etc/modules 'hmac'
50 ainsl /etc/modules 'tun'
51 ainsl /etc/modules 'ext3'
54 === configure networking ===
56 On each node, ensure that br0 (not eth0) and eth1 are configured.
58 The bridge interface, br0, is used by the guest virtual machines to reach the
61 If the guest virtual machines need to access the private network, then br1
62 should be configured rather than eth1.
64 To prevent the link address changing due to startup/shutdown of virtual
65 machines, explicitly set the value.
67 This is the interfaces file for foo.debian.org:
78 up ip link set addr $(cat /sys/class/net/$IFACE/address) dev $IFACE
81 iface eth1 inet static
86 This is the interfaces file for bar.debian.org:
97 up ip link set addr $(cat /sys/class/net/$IFACE/address) dev $IFACE
100 iface eth1 inet static
102 netmask 255.255.255.0
105 === configure lvm ===
107 On each node, configure lvm to ignore drbd devices and to prefer
108 {{{/dev/cciss}}} devices names over {{{/dev/block}}} device names
109 ([[http://code.google.com/p/ganeti/issues/detail?id=93|why?]]):
113 -e 's#^\(\s*filter\s\).*#\1= [ "a|.*|", "r|/dev/drbd[0-9]+|" ]#' \
114 -e 's#^\(\s*preferred_names\s\).*#\1= [ "^/dev/dm-*/", "^/dev/cciss/" ]#' \
119 === create lvm volume groups ===
121 On each node, create a volume group:
124 vgcreate vg_ganeti /dev/cciss/c0d0p3
127 === exchange ssh keys ===
132 mkdir -m 0700 -p /root/.ssh &&
133 ln -s /etc/ssh/ssh_host_rsa_key /root/.ssh/id_rsa
136 === configure iptables (via ferm) ===
138 the nodes must connect to each other over the public and private networks for a number of reasons; see the ganeti2 module in puppet
140 === instantiate the cluster ===
142 On the master node (foo) only:
146 --master-netdev br0 \
147 --vg-name vg_ganeti \
148 --secondary-ip E.F.G.2 \
149 --enabled-hypervisors kvm \
150 --nic-parameters link=br0 \
151 --mac-prefix 00:16:37 \
154 --hypervisor-parameters kvm:initrd_path=,kernel_path= \
160 * the master network device is set to br0, matching the public network bridge interface created above
161 * the volume group is set to vg_ganeti, matching the volume group created above
162 * the secondary IP address is set to the value of the master node's interface on the private network
163 * the nic parameters for instances is set to use br0 as default bridge
164 * the MAC prefix is registered in the dsa-kvm git repo
166 === add slave nodes ===
168 For each slave node (only bar for this example):
170 on the slave, append the master's /etc/ssh/ssh_host_rsa_key.pub to
171 /etc/ssh/userkeys/root. This is only required temporarily - once
172 everything works, puppet will put it/keep it there.
174 on the master node (foo):
178 --secondary-ip E.F.G.3 \
186 * the secondary IP address is set to the value of the slave node's interface on the private network
188 === verify cluster ===
190 On the master node (foo):
196 If everything has been configured correctly, no errors should be reported.
198 === create the 'dsa' variant ===
200 Ensure that the debootstrap+dsa variant has been installed by puppet (see ganeti2 module in dsa-puppet).
204 == How To Install Ganeti Instances ==
206 Suppose that qux.debian.org will be an instance (a virtual machine) hosted on
207 the foobar.debian.org ganeti cluster.
209 Before adding the instance, an LDAP entry must be created so that an A record
210 for the instance (A.B.C.4) exists.
212 === create the instance ===
214 On the master node (foo):
219 --disk-template drbd \
221 --os-type debootstrap+dsa \
222 --hypervisor-parameters kvm:initrd_path=,kernel_path= \
229 * the primary and secondary nodes have been explicitly set
230 * the operating system type is 'debootstrap+dsa'
231 * the network interfarce 0 (eth0 on the system) is set to the instance's interface on the public network
232 * If qux.d.o does not yet exist in DNS/LDAP, you may need --no-ip-check --no-name-check. Be careful that the hostname and IP address are not taken already!
238 If the instances require access to the private network, then there are two modifications necessary.
240 === re-configure networking ===
242 On the nodes, ensure that br1 is configured (rather than eth1).
244 This is the interfaces file for foo.debian.org:
248 iface br0 inet static
253 netmask 255.255.255.0
255 up ip link set addr $(cat /sys/class/net/$IFACE/address) dev $IFACE
258 iface br1 inet static
263 netmask 255.255.255.0
264 up ip link set addr $(cat /sys/class/net/$IFACE/address) dev $IFACE
267 This is the interfaces file for bar.debian.org:
271 iface br0 inet static
276 netmask 255.255.255.0
278 up ip link set addr $(cat /sys/class/net/$IFACE/address) dev $IFACE
281 iface br1 inet static
286 netmask 255.255.255.0
287 up ip link set addr $(cat /sys/class/net/$IFACE/address) dev $IFACE
290 === create or update the instance ===
292 When creating the instance, indicate both networks:
297 --disk-template drbd \
299 --os-type debootstrap+dsa \
300 --hypervisor-parameters kvm:initrd_path=,kernel_path= \
302 --net 1:link=br1,ip=E.F.G.4 \
306 * If qux.d.o does not yet exist in DNS/LDAP, you may need --no-ip-check --no-name-check. Be careful that the hostname and IP address are not taken already!
308 When updating an existing instance, add the interface:
311 gnt-instance shutdown qux.debian.org
312 gnt-instance modify \
313 --net add:link=br1,ip=E.F.G.4 \
315 gnt-instance startup qux.debian.org
318 Please note that the hook scripts are run only at instance instantiation. When
319 adding interfaces to an instance, the guest opearting system must be updated
323 * If you are importing an instance from libvirt with LVM setup, you can adopt LVs:
326 gnt-instance add -t plain --os-type debootstrap+dsa-wheezy \
327 --disk 0:adopt=lully-boot \
328 --disk 1:adopt=lully-root \
329 --disk 2:adopt=lully-swap \
330 --disk 3:adopt=lully-log \
331 --hypervisor-parameters kvm:initrd_path=,kernel_path= \
332 --net 0:ip=82.195.75.99 -n clementi.debian.org lully.debian.org
335 And you want to convert it to use DRBD afterwards and start it on the other cluster node, so we can ensure that DRBD is correctly working.
337 gnt-instance shutdown lully.debian.org
338 gnt-instance modify -t drbd -n czerny.debian.org lully.debian.org
339 gnt-instance failover lully.debian.org
340 gnt-instance startup lully.debian.org
343 * Some instances NEED ide instead of virtio
346 gnt-instance modify --hypervisor-parameters disk_type=ide fils.debian.org
349 * To import instances with SAN volumes
352 gnt-instance add -t blockdev --os-type debootstrap+dsa \
353 --disk 0:adopt=/dev/disk/by-id/scsi-reger-boot \
354 --disk 1:adopt=/dev/disk/by-id/scsi-reger-root \
355 --hypervisor-parameters kvm:initrd_path=,kernel_path= \
356 --net 0:ip=206.12.19.124 -n rossini.debian.org reger.debian.org
359 * How to add new LUNs to Bytemark Cluster
362 + Add a new LUN in the MSA and export it to all blades
365 gnt-cluster command "echo 1 > /sys/bus/pci/devices/0000:0e:00.0/cciss0/rescan"
368 reload multipath-tools on gnt-master (normaly bm-bl1):
369 /etc/init.d/multipath-tools reload
370 add the WWNs to dsa-puppet/modules/multipath/files/bm-multipath.conf and define the alias and commit that file to git.
373 gnt-cluster command "puppet agent -t"
375 This will update the multipath config on all cluster nodes. WITHOUT doing this, you can't migrate VMs between nodes.