Difference: PacemakerDualPrimaryConfiguration (5 vs. 6)

Revision 62013-01-04 - WilliamSeligman

Line: 1 to 1
 
META TOPICPARENT name="Computing"

Nevis particle-physics administrative cluster configuration

Line: 121 to 121
 This may help as you work your way through the configuration:

crm configure primitive MyIPResource ocf:heartbeat:IPaddr2 params ip=192.168.85.3 \
Changed:
<
<
cidr_netmask=32 op monitor interval=30s
>
>
cidr_netmask=32 op monitor interval=30s timeout=60s
  # Which is composed of * crm ::= "cluster resource manager", the command we're executing
Line: 133 to 133
  * cidr_netmask ::= netmask; 32-bits means use this exact IP address * op ::== what follows are options * monitor interval=30s ::= check every 30 seconds that this resource is working
Changed:
<
<
# ... timeout = how to long wait before you assume a resource is dead.
>
>
* timeout ::= how long to wait before you assume an "op" is dead.
 

How to find out which scripts exist, that is, which resources can be controlled by the HA cluster:

Line: 172 to 171
 # and "crm status" commands that I frequently typed in, in order to see that # everything was correct.
Changed:
<
<
# I also omit the standard resource options
>
>
# I also omit some of the standard resource options
 # (e.g., "... op monitor interval="20" timeout="40" depth="0"...) to make the
Changed:
<
<
# commands look simpler. This particular option means to check that the # resource is running every 20 seconds, and to declare that the monitor operation # will generate an error if 40 seconds elapse without a response. You can see the
>
>
# commands look simpler. You can see the
 # complete list with "crm configure show".

# DRBD is a service that synchronizes the hard drives between two machines.

Line: 216 to 213
 crm cib new disk

Changed:
<
<
# The DRBD is available to the system. The next step is to tell LVM
>
>
# The DRBD disk is available to the system. The next step is to tell LVM
  # that the volume group ADMIN exists on the disk.

# To find out that there was a resource "ocf:heartbeat:LVM" that I could use,

Line: 261 to 258
  clone FilesystemClone FilesystemGroup meta interleave="true"

Changed:
<
<
# One more thing: It's important that we not try to set up the filesystems
>
>
# It's important that we not try to set up the filesystems
  # until the DRBD admin resource is running on a node, and has been # promoted to master.

Changed:
<
<
# A score of "inf" means "infinity"; if the DRBD resource 'AdminClone' can't # be promoted, then don't start the 'FilesystemClone' resource.
>
>
# A score of "inf:" means "infinity": 'FileSystemClone' must be on a node on which # 'AdminClone' is in the Master state; if the DRBD resource 'AdminClone' can't # be promoted, then don't start the 'FilesystemClone' resource. (You can use numeric # values instead of infinity, in which case these constraints become suggestions # instead of being mandatory.)
  colocation Filesystem_With_Admin inf: FilesystemClone AdminClone:Master order Admin_Before_Filesystem inf: AdminClone:promote FilesystemClone:start
Line: 274 to 274
  cib commit disk quit
Changed:
<
<
# Some standard Linux services are under corosync's control. They depend on some or # all of the filesystems being mounted.

# Let's start with a simple one: enable the printing service (cups):

>
>
# Once all the filesystems are mounted, we start can start other resources. Let's # define a set of cloned IP addresses that will always point to at least one of the nodes, # possibly both.
  crm
Changed:
<
<
cib new printing
>
>
cib new ip

# One address for each network

 
Changed:
<
<
# lsb = "Linux Standard Base." It just means any service which is # controlled by the one of the standard scripts in /etc/init.d
>
>
primitive IP_cluster ocf:heartbeat:IPaddr2 params ip="129.236.252.11" cidr_netmask="32" nic="eth0" primitive IP_cluster_local ocf:heartbeat:IPaddr2 params ip="10.44.7.11" cidr_netmask="32" nic="eth2" primitive IP_cluster_sandbox ocf:heartbeat:IPaddr2 params ip="10.43.7.11" cidr_netmask="32" nic="eth0.3"
 
Changed:
<
<
configure primitive Cups lsb:cups
>
>
# Group them together
 
Changed:
<
<
# Cups stores its spool files in /var/spool/cups. If the cups service # were to switch to a different server, we want the new server to see # the spooled files. So create /var/nevis/cups, link it with: # mv /var/spool/cups /var/spool/cups.ori # ln -sf /var/nevis/cups /var/spool/cups # and demand that the cups service only start if /var/nevis (and the other # high-availability directories) have been mounted.
>
>
group IPGroup IP_cluster IP_cluster_local IP_cluster_sandbox
 
Changed:
<
<
configure colocation CupsWithVar inf: Cups AdminDirectoriesGroup
>
>
# The option "globally-unique=true" works with IPTABLES to make # sure that ethernet connections are not disrupted even if one of # nodes goes down; see "Clusters From Scratch" for details.
 
Changed:
<
<
# In order to prevent chaos, make sure that the high-availability directories # have been mounted before we try to start cups.
>
>
clone IPClone IPGroup meta globally-unique="true" clone-max="2" clone-node-max="2" interleave="false"
 
Changed:
<
<
configure order VarBeforeCups inf: AdminDirectoriesGroup Cups
>
>
# Make sure the filesystems are mounted before starting the IP resources. colocation IP_With_Filesystem inf: IPClone FilesystemClone order Filesystem_Before_IP inf: FilesystemClone IPClone
 
Changed:
<
<
cib commit printing
>
>
cib commit ip
  quit
Changed:
<
<
# The other services (xinetd, dhcpd) follow the same pattern as above: # Make sure the services start on the same machine as the admin directories, # and after the admin directories are successfully mounted.
>
>
# We have to export some of the filesystems via NFS before some of the virtual machines # will be able to run.
  crm
Changed:
<
<
cib new services
>
>
cib new exports
 
Changed:
<
<
configure primitive Xinetd lsb:xinetd configure primitive Dhcpd lsb:dhcpd
>
>
# This is an example NFS export resource; I won't list them all here. See # "crm configure show" for the complete list.

primitive ExportUsrNevis ocf:heartbeat:exportfs description="Site-wide applications installed in /usr/nevis" params clientspec="*.nevis.columbia.edu" directory="/usr/nevis" fsid="20" options="ro,no_root_squash,async"

# Define a group for all the exportfs resources. You can see it's a long list, # which is why I don't list them all explicitly. I had to be careful # about the exportfs definitions; despite the locking mechanims of GFS2, # we'd get into trouble if two external systems tried to write to the same # DRBD partition at once via NFS.

 
Changed:
<
<
configure colocation XinetdWithVar inf: Xinetd AdminDirectoriesGroup configure order VarBeforeXinetd inf: VarDirectory Xinetd
>
>
group ExportsGroup ExportMail ExportMailInbox ExportMailFolders ExportMailForward ExportMailProcmailrc ExportUsrNevisHermes ExportUsrNevis ExportUsrNevisOffsite ExportWWW
 
Changed:
<
<
configure colocation DhcpdWithVar inf: Dhcpd AdminDirectoriesGroup configure order VarBeforeDhcpd inf: VarDirectory Dhcpd
>
>
# Clone the group so both nodes export the partitions. Make sure the # filesystems are mounted before we export them.
 
Changed:
<
<
cib commit services
>
>
clone ExportsClone ExportsGroup colocation Exports_With_Filesystem inf: ExportsClone FilesystemClone order Filesystem_Before_Exports inf: FilesystemClone ExportsClone

cib commit exports

  quit
Changed:
<
<
# The high-availability servers export some of the admin directories to other # systems, both real and virtual; for example, the /usr/nevis directory is # exported to all the other machines on the Nevis Linux cluster.

# NFS exporting of a shared directory can be a little tricky. As with CUPS # spooling, we want to preserve the NFS export state in a way that the # backup server can pick it up. The safest way to do this is to create a # small separate LVM partition ("nfs") and mount it as "/var/lib/nfs", # the NFS directory that contains files that keep track of the NFS state.

>
>
# Symlinks: There are some scripts that I want to run under cron. These scripts are # located in the DRBD /var/nevis file system. For them to run via cron, they have to # found in /etc/cron.d somehow. A symlink is the easiest way, and there's a # symlink pacemaker resource to manage this.
  crm
Changed:
<
<
cib new nfs
>
>
cib new cron
 
Changed:
<
<
# Define the mount for the NFS state directory /var/lib/nfs
>
>
# The ambient-temperature script periodically checks the computer room's # environment monitor, and shuts down the cluster if the temperature gets # too high.

primitive CronAmbientTemperature ocf:heartbeat:symlink description="Shutdown cluster if A/C stops" params link="/etc/cron.d/ambient-temperature" target="/var/nevis/etc/cron.d/ambient-temperature" backup_suffix=".original"

# We don't want to clone this resource; I only want one system to run this script # at any one time.

colocation Temperature_With_Filesystem inf: CronAmbientTemperature FilesystemClone order Filesystem_Before_Temperature inf: FilesystemClone CronAmbientTemperature

# Every couple of months, make a backup of the virtual machine's disk images.

primitive CronBackupVirtualDiskImages ocf:heartbeat:symlink description="Periodically save copies of the virtual machines" params link="/etc/cron.d/backup-virtual-disk-images" target="/var/nevis/etc/cron.d/backup-virtual-disk-images" backup_suffix=".original" colocation BackupImages_With_Filesystem inf: CronBackupVirtualDiskImages FilesystemClone order Filesystem_Before_BackupImages inf: FilesystemClone CronBackupVirtualDiskImages

 
Changed:
<
<
configure primitive NfsStateDirectory ocf:heartbeat:Filesystem params device="/dev/admin/nfs" directory="/var/lib/nfs" fstype="ext4" configure colocation NfsStateWithVar inf: NfsStateDirectory AdminDirectoriesGroup configure order VarBeforeNfsState inf: AdminDirectoriesGroup NfsStateDirectory
>
>
cib commit cron quit
 
Changed:
<
<
# Now that the NFS state directory is mounted, we can start nfslockd. Note that # that we're starting NFS lock on both the primary and secondary HA systems; # by default a "clone" resource is started on all systems in a cluster.
>
>
# These are the most important resources on the HA cluster: the virtual # machines.
 
Changed:
<
<
# (Placing nfslockd under the control of Pacemaker turned out to be key to # successful transfer of cluster services to another node. The nfslockd and # nfs daemon information stored in /var/lib/nfs have to be consistent.)
>
>
crm cib new vm
 
Changed:
<
<
configure primitive NfsLockInstance lsb:nfslock configure clone NfsLock NfsLockInstance
>
>
# In order to start a virtual machine, the libvirtd daemon has to run. The "lsb:" means # "Linux Standard Base", which in turn means any script located in # /etc/init.d.
 
Changed:
<
<
configure order NfsStateBeforeNfsLock inf: NfsStateDirectory NfsLock
>
>
primitive Libvirtd lsb:libvirtd
 
Changed:
<
<
# Once nfslockd has been set up, we can start NFS. (We say to colocate # NFS with 'NfsStateDirectory', instead of nfslockd, because nfslockd # is going to be started on both nodes.)
>
>
# libvirtd looks for configuration files that define the virtual machines. # These files are kept in /var/nevis, like the above cron scripts, and are # "placed" via symlinks.
 
Changed:
<
<
configure primitive Nfs lsb:nfs configure colocation NfsWithNfsState inf: Nfs NfsStateDirectory configure order NfsLockBeforeNfs inf: NfsLock Nfs
>
>
primitive SymlinkEtcLibvirt ocf:heartbeat:symlink params link="/etc/libvirt" target="/var/nevis/etc/libvirt" backup_suffix=".original" primitive SymlinkQemuSnapshot ocf:heartbeat:symlink params link="/var/lib/libvirt/qemu/snapshot" target="/var/nevis/lib/libvirt/qemu/snapshot" backup_suffix=".original"
 
Changed:
<
<
cib commit nfs quit
>
>
# Again, define a group for these resources, clone the group so they # run on both nodes, and make sure they don't run unless the # filesystems are mounted.
 
Added:
>
>
group LibvirtdGroup SymlinkEtcLibvirt SymlinkQemuSnapshot Libvirtd clone LibvirtdClone LibvirtdGroup colocation Libvirtd_With_Filesystem inf: LibvirtdClone FilesystemClone
 
Changed:
<
<
# The whole point of the entire setup is to be able to run guest virtual machines # under the control of the high-availability service. Here is the set-up for one example # virtual machine. I previously created the hogwarts virtual machine and copied its # configuration to /xen/configs/hogwarts.cfg.
>
>
# A tweak: some virtual machines require the directories exportted # by the exportfs resources defined above. Don't start the VMs until # the exports are complete.
 
Changed:
<
<
# I duplicated the same procedure for franklin (mail server), ada (web server), and # so on, but I don't show that here.
>
>
order Exports_Before_Libvirtd inf: ExportsClone LibvirtdClone
 
Changed:
<
<
crm cib new hogwarts
>
>
# The typical definition of a resource that runs a VM. I won't list # them all, just the one for the mail server. Note that all the # virtual-machine resource names start with VM_, so they'll show # up next to each other in the output of "crm configure show".
 
Changed:
<
<
# Give the virtual machine a long stop interval before flagging an error. # Sometimes it takes a while for Linux to shut down.
>
>
# VM migration is a neat feature. If pacemaker has the chance to move # a virtual machine, it can transmit it to another node without stopping it # on the source node and restarting it at the destination. If a machine # crashes, migration can't happen, but it can greatly speed up the # controlled shutdown ofa node.
 
Changed:
<
<
configure primitive Hogwarts ocf:heartbeat:Xen params xmfile="/xen/configs/Hogwarts.cfg" op stop interval="0" timeout="240"
>
>
primitive VM_franklin ocf:heartbeat:VirtualDomain params config="/etc/libvirt/qemu/franklin.xml" \ migration_transport="ssh" meta allow-migrate="true"
 
Changed:
<
<
# All the virtual machine files are stored in the /xen partition, which is one # of the high-availability admin directories. The virtual machine must run on # the system with this directory.
>
>
# We don't want to clone the VMs; it will just confuse things if there # two mail servers (with the same IP address!) running at the same time.
 
Changed:
<
<
configure colocation HogwartsWithDirectories inf: Hogwarts AdminDirectoriesGroup
>
>
colocation Mail_With_Libvirtd inf: VM_franklin LibvirtdClone order Libvirtd_Before_Mail inf: LibvirtdClone VM_franklin
 
Changed:
<
<
# All of the virtual machines depend on NFS-mounting directories which # are exported by the HA server. The safest thing to do is to make sure # NFS is running on the HA server before starting the virtual machine.
>
>
cib commit vm quit

# A less-critical resource is tftp. As above, we define the basic xinetd # resource found in /etc/init.d, include a configure file with a symlink, # then clone the resource and specify it can't run until the filesystems # are mounted.

crm cib new tftp

 
Changed:
<
<
configure order NfsBeforeHogwarts inf: Nfs Hogwarts
>
>
primitive Xinetd lsb:xinetd primitive SymlinkTftp ocf:heartbeat:symlink params link="/etc/xinetd.d/tftp" target="/var/nevis/etc/xinetd.d/tftp" backup_suffix=".original"

group TftpGroup SymlinkTftp Xinetd clone TftpClone TftpGroup colocation Tftp_With_Filesystem inf: TftpClone FilesystemClone order Filesystem_Before_Tftp inf: FilesystemClone TftpClone

 
Changed:
<
<
cib commit hogwarts
>
>
cib commit tftp
  quit
Added:
>
>
# More important is dhcpd, which assigns IP addresses dynamically. # Many systems at Nevis require a DHCP server for their IP address, # include the wireless routers. This follows the same pattern as above, # except that we don't clone the dhcpd daemon, since we want only # one DHCP server at Nevis.

crm cib new dhcp

configure primitive Dhcpd lsb:dhcpd

# Associate an IP address with the DHCP server. This is a mild # convenience for the times I update the list of MAC addresses # to be assigned permanent IP addresses. primitive IP_dhcp ocf:heartbeat:IPaddr2 params ip="10.44.107.11" cidr_netmask="32" nic="eth2"

primitive SymlinkDhcpdConf ocf:heartbeat:symlink params link="/etc/dhcp/dhcpd.conf" target="/var/nevis/etc/dhcpd.conf" backup_suffix=".original" primitive SymlinkDhcpdLeases ocf:heartbeat:symlink params link="/var/lib/dhcpd" target="/var/nevis/dhcpd" backup_suffix=".original" primitive SymlinkSysconfigDhcpd ocf:heartbeat:symlink params link="/etc/sysconfig/dhcpd" target="/var/nevis/etc/sysconfig/dhcpd" backup_suffix=".original"

group DhcpGroup SymlinkDhcpdConf SymlinkSysconfigDhcpd SymlinkDhcpdLeases Dhcpd IP_dhcp colocation Dhcp_With_Filesystem inf: DhcpGroup FilesystemClone order Filesystem_Before_Dhcp inf: FilesystemClone DhcpGroup

cib commit dhcp quit

  # An important part of a high-availability configuration is STONITH = "Shoot the # other node in the head." Here's the idea: suppose one node fails for some reason. The
Line: 429 to 518
  # The official corosync distribution from <http://www.clusterlabs.org/> # does not include a script for NUT, so I had to write one. It's located at
Changed:
<
<
# /home/bin/nut.sh on both hypatia and orestes; there are appropriate links # to this script from the stonith/external directory.
>
>
# /home/bin//home/bin/fence_nut.pl on both hypatia and orestes; there are appropriate links # to this script from /usr/sbin/fence_nut.
  # The following commands implement the STONITH mechanism for our cluster:
Line: 439 to 528
  # The STONITH resource that can potentially shut down hypatia.

Changed:
<
<
configure primitive HypatiaStonith stonith:external/nut params hostname="hypatia.nevis.columbia.edu" ups="hypatia-ups" username="admin" password="acdc"
>
>
primitive StonithHypatia stonith:fence_nut params stonith-timeout="120s" pcmk_host_check="static-list" pcmk_host_list="hypatia.nevis.columbia.edu" ups="hypatia-ups" username="XXXX" password="XXXX" cycledelay="20" ondelay="20" offdelay="20" noverifyonoff="1" debug="1"
  # The node that runs the above script cannot be hypatia; it's # not wise to trust a node to STONITH itself. Note that the score # is "negative infinity," which means "never run this resource # on the named node."
Changed:
<
<
configure location HypatiaStonithLoc HypatiaStonith -inf: hypatia.nevis.columbia.edu
>
>
location StonithHypatia_Location StonithHypatia -inf: hypatia.nevis.columbia.edu
  # The STONITH resource that can potentially shut down orestes.
Changed:
<
<
configure primitive OrestesStonith stonith:external/nut params hostname="orestes.nevis.columbia.edu" ups="orestes-ups" username="admin" password="acdc"
>
>
primitive StonithOrestes stonith:fence_nut params stonith-timeout="120s" pcmk_host_check="static-list" pcmk_host_list="orestes.nevis.columbia.edu" ups="orestes-ups" username="XXXX" password="XXXX" cycledelay="20" ondelay="20" offdelay="20" noverifyonoff="1" debug="1"
  # Again, orestes cannot be the node that runs the above script.

Changed:
<
<
configure location OresetesStonithLoc OrestesStonith -inf: orestes.nevis.columbia.edu
>
>
location StonithOrestes_Location StonithOrestes -inf: orestes.nevis.columbia.edu
  cib commit stonith quit
 
This site is powered by the TWiki collaboration platform Powered by PerlCopyright © 2008-2020 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki? Send feedback