X-Git-Url: https://git.adam-barratt.org.uk/?a=blobdiff_plain;f=input%2Fhowto%2Finstall-ganeti.creole;h=e27f02bb22dc41f093165cfbf214a66310bf0b96;hb=7ba134fb155e7b0b2ca3ab6950f046462c03a9f8;hp=72705374d1a8a46848c2de0943cd7eb67e825f1b;hpb=d5fe68cff6bae89851ff8dd1d91dc1867efdb1ad;p=mirror%2Fdsa-wiki.git diff --git a/input/howto/install-ganeti.creole b/input/howto/install-ganeti.creole index 7270537..e27f02b 100644 --- a/input/howto/install-ganeti.creole +++ b/input/howto/install-ganeti.creole @@ -1,65 +1,145 @@ == How To Install Ganeti Clusters and Instances == -=== How To Install Ganeti Clusters === +=== suppositions === -Suppose that you have two identical hosts running squeeze, foo.debian.org and -bar.debian.org, that will be used as nodes in a ganeti cluster. +Suppose that there are two identical hosts: foo.debian.org and bar.debian.org. -==== install required packages ==== +They are running squeeze and have been integrated into Debian infrastructure. + +They will serve as nodes in a ganeti cluster named foobar.debian.org. + +They have a RAID1 array exposing three partitions: c0d0p1 for /, c0d0p2 for +swap and c0d0p3 for lvm volume groups to be used by ganeti via drbd. + +They have two network interfaces: eth0 (public) and eth1 (private). + +The public network is A.B.C.0/24 with gateway A.B.C.254. + +The private network is E.F.G.0/24 with no gateway. + +Suppose that the first instance to be hosted on foobar.debian.org is +qux.debian.org. + +The following DNS records exist: + +{{{ + foobar.debian.org. IN A A.B.C.1 + foo.debian.org. IN A A.B.C.2 + bar.debian.org. IN A A.B.C.3 + qux.debian.org. IN A A.B.C.4 + foo.debprivate-hoster.debian.org. IN A E.F.G.2 + bar.debprivate-hoster.debian.org. IN A E.F.G.3 +}}} + +=== install required packages === On each node, install the required packages: {{{ - apt-get install fai-client ssed extlinux - apt-get install -t squeeze-backports drbd8-utils ganeti2 ganeti-instance-debootstrap ganeti-htools + # maybe: apt-get install drbd8-utils + # maybe: apt-get install ganeti-instance-debootstrap + apt-get install ganeti2 ganeti-htools qemu-kvm }}} -==== configure kernel modules ==== +=== configure kernel modules === -On each node, ensure that required kernel modules are loaded at boot: +On each node, ensure that the required kernel modules are loaded at boot: {{{ ainsl /etc/modules 'drbd minor_count=255 usermode_helper=/bin/true' ainsl /etc/modules 'hmac' ainsl /etc/modules 'tun' + ainsl /etc/modules 'ext3' + ainsl /etc/modules 'ext4' }}} -==== configure networking ==== +=== configure networking === -Determine a hostname for the cluster (foobar.debian.org) and assign an A record. +On each node, ensure that br0 (not eth0) and eth1 are configured. -On each node, ensure that all nodes and the cluster itself are listed in -{{{/etc/hosts}}}. +The bridge interface, br0, is used by the guest virtual machines to reach the +public network. -On each node, create a bridge interface: br0. +If the guest virtual machines need to access the private network, then br1 +should be configured rather than eth1. -==== create storage ==== +To prevent the link address changing due to startup/shutdown of virtual +machines, explicitly set the value. + +This is the interfaces file for foo.debian.org: + +{{{ + auto br0 + iface br0 inet static + bridge_ports eth0 + bridge_maxwait 0 + bridge_fd 0 + address A.B.C.2 + netmask 255.255.255.0 + gateway A.B.C.254 + up ip link set addr $(cat /sys/class/net/$IFACE/address) dev $IFACE + + auto eth1 + iface eth1 inet static + address E.F.G.2 + netmask 255.255.255.0 +}}} + +This is the interfaces file for bar.debian.org: + +{{{ + auto br0 + iface br0 inet static + bridge_ports eth0 + bridge_maxwait 0 + bridge_fd 0 + address A.B.C.3 + netmask 255.255.255.0 + gateway A.B.C.254 + up ip link set addr $(cat /sys/class/net/$IFACE/address) dev $IFACE + + auto eth1 + iface eth1 inet static + address E.F.G.3 + netmask 255.255.255.0 +}}} + +=== configure lvm === On each node, configure lvm to ignore drbd devices and to prefer -{{{/dev/cciss}}} devices names to {{{/dev/block}}} device names +{{{/dev/cciss}}} devices names over {{{/dev/block}}} device names ([[http://code.google.com/p/ganeti/issues/detail?id=93|why?]]): {{{ ssed -i \ - -e 's#^\(\s*filter\s\).*#\1= [ "a|.*|", "r|/dev/drbd[0-9]+|" ]#' + -e 's#^\(\s*filter\s\).*#\1= [ "a|.*|", "r|/dev/drbd[0-9]+|" ]#' \ -e 's#^\(\s*preferred_names\s\).*#\1= [ "^/dev/dm-*/", "^/dev/cciss/" ]#' \ /etc/lvm/lvm.conf + service lvm2 restart }}} +=== create lvm volume groups === + On each node, create a volume group: {{{ vgcreate vg_ganeti /dev/cciss/c0d0p3 }}} -==== create kvmU symbolic links ==== +=== exchange ssh keys === + +on each node: {{{ - ln -s -T /boot/vmlinuz-2.6.32-5-amd64 /boot/vmlinuz-2.6-kvmU - ln -s -T /boot/initrd.img-2.6.32-5-amd64 /boot/initrd.img-2.6-kvmU + mkdir -m 0700 -p /root/.ssh && + ln -s /etc/ssh/ssh_host_rsa_key /root/.ssh/id_rsa }}} -==== instantiate the cluster ==== +=== configure iptables (via ferm) === + +the nodes must connect to each other over the public and private networks for a number of reasons; see the ganeti2 module in puppet + +=== instantiate the cluster === On the master node (foo) only: @@ -67,29 +147,47 @@ On the master node (foo) only: gnt-cluster init \ --master-netdev br0 \ --vg-name vg_ganeti \ - --secondary-ip 192.168.2.213 \ + --secondary-ip E.F.G.2 \ --enabled-hypervisors kvm \ --nic-parameters link=br0 \ --mac-prefix 00:16:37 \ --no-ssh-init \ --no-etc-hosts \ - --hypervisor-parameters kvm:initrd_path=/boot/initrd.img-2.6-kvmU \ + --hypervisor-parameters kvm:initrd_path=,kernel_path= \ foobar.debian.org }}} -==== add slave nodes === +Note the following: + +* the master network device is set to br0, matching the public network bridge interface created above +* the volume group is set to vg_ganeti, matching the volume group created above +* the secondary IP address is set to the value of the master node's interface on the private network +* the nic parameters for instances is set to use br0 as default bridge +* the MAC prefix is registered in the dsa-kvm git repo + +=== add slave nodes === -For each slave node (bar), on the master node (foo): +For each slave node (only bar for this example): + +on the slave, append the master's /etc/ssh/ssh_host_rsa_key.pub to +/etc/ssh/userkeys/root. This is only required temporarily - once +everything works, puppet will put it/keep it there. + +on the master node (foo): {{{ gnt-node add \ - --secondary-ip 192.168.2.217 \ + --secondary-ip E.F.G.3 \ --no-ssh-key-check \ --no-node-setup \ bar.debian.org }}} -==== verify cluster ==== +Note the following: + +* the secondary IP address is set to the value of the slave node's interface on the private network + +=== verify cluster === On the master node (foo): @@ -97,18 +195,23 @@ On the master node (foo): gnt-cluster verify }}} -=== How To Install Ganeti Instances === +If everything has been configured correctly, no errors should be reported. + +=== create the 'dsa' variant === + +Ensure that the debootstrap+dsa variant has been installed by puppet (see ganeti2 module in dsa-puppet). + +---- -Suppose that the first guest on will be qux.debian.org. +== How To Install Ganeti Instances == -==== define the instance ==== +Suppose that qux.debian.org will be an instance (a virtual machine) hosted on +the foobar.debian.org ganeti cluster. -# add qux.debian.org entry in LDAP to establish A record -# ask hosting partner to establish PTR record as appropriate -# add qux.debian.org entry in puppet -# add qux.debian.org entry in nagios +Before adding the instance, an LDAP entry must be created so that an A record +for the instance (A.B.C.4) exists. -==== create the instance ==== +=== create the instance === On the master node (foo): @@ -119,7 +222,184 @@ On the master node (foo): --os-size 4GiB \ --os-type debootstrap+dsa \ --hypervisor-parameters kvm:initrd_path=,kernel_path= \ - --net 0:ip=1.2.3.4 \ + --net 0:ip=A.B.C.4 \ + qux.debian.org +}}} + +Note the following: + +* the primary and secondary nodes have been explicitly set +* the operating system type is 'debootstrap+dsa' +* the network interfarce 0 (eth0 on the system) is set to the instance's interface on the public network +* If qux.d.o does not yet exist in DNS/LDAP, you may need --no-ip-check --no-name-check. Be careful that the hostname and IP address are not taken already! + +---- + +== Variations == + +If the instances require access to the private network, then there are two modifications necessary. + +=== re-configure networking === + +On the nodes, ensure that br1 is configured (rather than eth1). + +This is the interfaces file for foo.debian.org: + +{{{ + auto br0 + iface br0 inet static + bridge_ports eth0 + bridge_maxwait 0 + bridge_fd 0 + address A.B.C.2 + netmask 255.255.255.0 + gateway A.B.C.254 + up ip link set addr $(cat /sys/class/net/$IFACE/address) dev $IFACE + + auto br1 + iface br1 inet static + bridge_ports eth1 + bridge_maxwait 0 + bridge_fd 0 + address E.F.G.2 + netmask 255.255.255.0 + up ip link set addr $(cat /sys/class/net/$IFACE/address) dev $IFACE +}}} + +This is the interfaces file for bar.debian.org: + +{{{ + auto br0 + iface br0 inet static + bridge_ports eth0 + bridge_maxwait 0 + bridge_fd 0 + address A.B.C.3 + netmask 255.255.255.0 + gateway A.B.C.254 + up ip link set addr $(cat /sys/class/net/$IFACE/address) dev $IFACE + + auto br1 + iface br1 inet static + bridge_ports eth1 + bridge_maxwait 0 + bridge_fd 0 + address E.F.G.3 + netmask 255.255.255.0 + up ip link set addr $(cat /sys/class/net/$IFACE/address) dev $IFACE +}}} + +=== create or update the instance === + +When creating the instance, indicate both networks: + +{{{ + gnt-instance add \ + --node foo:bar \ + --disk-template drbd \ + --os-size 4GiB \ + --os-type debootstrap+dsa \ + --hypervisor-parameters kvm:initrd_path=,kernel_path= \ + --net 0:ip=A.B.C.4 \ + --net 1:link=br1,ip=E.F.G.4 \ + qux.debian.org +}}} + +* If qux.d.o does not yet exist in DNS/LDAP, you may need --no-ip-check --no-name-check. Be careful that the hostname and IP address are not taken already! + +When updating an existing instance, add the interface: + +{{{ + gnt-instance shutdown qux.debian.org + gnt-instance modify \ + --net add:link=br1,ip=E.F.G.4 \ qux.debian.org + gnt-instance startup qux.debian.org }}} +Please note that the hook scripts are run only at instance instantiation. When +adding interfaces to an instance, the guest opearting system must be updated +manually. + + +* If you are importing an instance from libvirt with LVM setup, you can adopt LVs: + +{{{ + gnt-instance add -t plain --os-type debootstrap+dsa-wheezy \ + --disk 0:adopt=lully-boot \ + --disk 1:adopt=lully-root \ + --disk 2:adopt=lully-swap \ + --disk 3:adopt=lully-log \ + --hypervisor-parameters kvm:initrd_path=,kernel_path= \ + --net 0:ip=82.195.75.99 -n clementi.debian.org lully.debian.org +}}} + +And you want to convert it to use DRBD afterwards and start it on the other cluster node, so we can ensure that DRBD is correctly working. +{{{ + gnt-instance shutdown lully.debian.org + gnt-instance modify -t drbd -n czerny.debian.org lully.debian.org + gnt-instance failover lully.debian.org + gnt-instance startup lully.debian.org +}}} + +* Some instances NEED ide instead of virtio + +{{{ + gnt-instance modify --hypervisor-parameters disk_type=ide fils.debian.org +}}} + +* To import instances with SAN volumes + +{{{ + gnt-instance add -t blockdev --os-type debootstrap+dsa \ + --disk 0:adopt=/dev/disk/by-id/scsi-reger-boot \ + --disk 1:adopt=/dev/disk/by-id/scsi-reger-root \ + --hypervisor-parameters kvm:initrd_path=,kernel_path= \ + --net 0:ip=206.12.19.124 -n rossini.debian.org reger.debian.org +}}} + +* How to add new LUNs to Bytemark Cluster + +** Add new LUN to MSA and export to all blades + +{{{ + Log into MSA controller + + Choose which vdisk to use, use "show vdisks" to list + +Add the volume: + # create volume vdisk msa2k-2-500gr10 size 5G donizetti + +Find a free LUN: + + # show lun-maps + +Make a note of the next free LUN + +Generate map commands for all blades, all ports, run locally: + + $ for p in 1 2 3 4; do for bl in 1 2 3 4 5 6 ; do echo "map volume donizetti lun 27 host bm-bl$bl-p$p" ; done ; done + +Paste the output into the MSA shell + +Find the WWN by doing show host-maps and looking for the volume name. +Transform it using the sed run at the top of /etc/multipath.conf: + +echo "$WWN" | sed -re 's#(.{6})(.{6})0000(.{2})(.*)#36\1000\2\3\4#' +}}} + +{{{ + + then: + gnt-cluster command "echo 1 > /sys/bus/pci/devices/0000:0e:00.0/cciss0/rescan" + + then: + reload multipath-tools on gnt-master (normaly bm-bl1): + service multipath-tools reload + add the WWNs to dsa-puppet/modules/multipath/files/multipath-bm.conf and define the alias and commit that file to git. + + then: + gnt-cluster command "puppet agent -t" + + This will update the multipath config on all cluster nodes. WITHOUT doing this, you can't migrate VMs between nodes. +}}}