FSF France machines management - Cookbook: recipe #319, Howto upgrade a Ganeti node

recipe #319, HowTo: Howto upgrade a Ganeti node

If the new disks have the same size (or a bigger one but you don't care if you lose extra space), you can copy disks using dd.

• backup node
• failover all vms to their secondary node. move failover IPs if needed. The route will be updated by ospf.
• down the node
• boot current node and new one on rescue system
• on current node, start disc sync using (in a screen or with nohup) :

dd if=/dev/sda | ssh new.node "dd of=/dev/sda" dd if=/dev/sdb | ssh new.node "dd of=/dev/sdb"

• when sync are finished, fsck /dev/sda1 on new node and any extra partitions

• mount /dev/sda1 somewhere, and update /etc/network/interfaces with new server IP (check ovh interface) 7bis) update DNS (z2-X.pokersource.info to use new IP)
• start new node, check it works
• migrate failover IP (z2-X1.pokersource.info) from old node to new one.
• check node is back into the cluster
• redist ganeti config :

• make a "gnt-cluster verify", it will display lot of :

Thu Dec 3 09:34:05 2009 * Verifying node z2-1.host.gnt (master candidate) Thu Dec 3 09:34:05 2009 - ERROR: drbd minor 0 of instance home.binbang.vm.gnt is not active Thu Dec 3 09:34:05 2009 - ERROR: drbd minor 1 of instance rheincode.pokersource.vm.gnt is not active Thu Dec 3 09:34:05 2009 - ERROR: drbd minor 2 of instance nagios.vm.gnt is not active Thu Dec 3 09:34:05 2009 - ERROR: drbd minor 3 of instance trac.dachary.vm.gnt is not active Thu Dec 3 09:34:05 2009 - ERROR: drbd minor 4 of instance trac.pokersource.vm.gnt is not active Thu Dec 3 09:34:05 2009 - ERROR: drbd minor 5 of instance trac.fsffrance.vm.gnt is not active Thu Dec 3 09:34:05 2009 - ERROR: drbd minor 6 of instance z2work.vm.gnt is not active Thu Dec 3 09:34:05 2009 - ERROR: drbd minor 7 of instance pm-monitor.pokersource.vm.gnt is not active Thu Dec 3 09:34:05 2009 - ERROR: drbd minor 9 of instance booken.binbang.vm.gnt is not active Thu Dec 3 09:34:05 2009 - ERROR: drbd minor 10 of instance tsunami.vm.gnt is not active Thu Dec 3 09:34:05 2009 - ERROR: drbd minor 11 of instance p4m.pokersource.vm.gnt is not active Thu Dec 3 09:34:05 2009 - ERROR: drbd minor 12 of instance proxy3.vm.gnt is not active Thu Dec 3 09:34:05 2009 - ERROR: drbd minor 13 of instance cspoker-bot.vm.gnt is not active Thu Dec 3 09:34:05 2009 - ERROR: drbd minor 14 of instance pioneer.binbang.vm.gnt is not active Thu Dec 3 09:34:05 2009 - ERROR: drbd minor 15 of instance drupal-z2.pokersource.vm.gnt is not active Thu Dec 3 09:34:05 2009 - ERROR: drbd minor 16 of instance proxy2.vm.gnt is not active Thu Dec 3 09:34:05 2009 - ERROR: drbd minor 18 of instance x34xn.binbang.vm.gnt is not active Thu Dec 3 09:34:05 2009 - ERROR: drbd minor 22 of instance trac.binbang.vm.gnt is not active

All these instances, if started during node swap, will need a full resync of the drbd device. To do this without shutting down the vm :
#To remount the drbd device on slave gnt-instance activate-disk <yourinstance>

Drbd should automatically sync the slave with the changes, to check this, you can use, for example :

On new node, you should see a short sync each time you activate a disk.
Retry a "gnt-cluster verify" after that, you shouldn't have any error in "verifying z2-X.host.gnt" section.

• switch back the vm on the node.

 

 

 

 

 

Audience and Context

Audience:
   All Project Members
Feature:
   Project Main Pages

(As there is at least one of the Audience/Feature/Action context information not set, this recipe will not show up in related recipes links)