M41 Highway

Data science and software engineering blog


Leave a comment

To achieve High Availability with two commodity computers

It is an English version of my post of “用兩台家用電腦實現高可用性部署” dated 3rd September 2013.

The High Availability becomes the implicit non functional requirement since the enterprises has been paying rising attention of the ‘data’. The High Availability is the ability to resist failures in the whole system, rather than achieved by just a piece of software, or by deploying a load balance device. There is no perfect solution for High Availability because, with limited budget, you can’t solve all the problems which may arise from all sort of possibilities, including networks, hardware, software, and even the business logic. For an example of a B2B platform, which application servers have been deployed in redundancy. If one of application servers occurs a failure, the user requests can be routed to the server in good conditions and thus the services can be regarded as High Availability. Another example is the master-slave database deployment that ensure the minimum data loss during failures. The UPS and the backup power supply is another good example in which you secure the electricity supplies. Using two broadband suppliers is also a reasonable solution to prevent Single point of failure on network level. There is no perfect solution in High Availability! You’ve done all the four deployments in the platform. Guess what will happen if the payment service provider occurs a blackout for a couple of hours. Guess what will happen if there is fire happens at your neighbor and saves all the lives luckily, but the flood damaged both your main and backup power supplies. It is tragedy obviously because either failure cause a single point of failure that is not on your budget book! With large and sufficient budget, you could minimize the most of the risks. With limited budget, you could get rid of the most risky problem, and develop a emergency plan for the rest of the problems with your stakeholders.

The High Availability is a mature functionality on both hardware and networking devices nowadays. Redundant components is ready get “online” once a failure is detected.  The complexity of handling data synchronization, data integrity, and data recovery leads to a complicated mechanism in High Availability of the database system. It makes senses to copy data in real time to achieve High Availability. This idea is implemented by copying the transaction logs files from the primary machine to the slave machine. If there is failure in the primary database, the slave database will take over and execute the transaction logs to roll back to the latest image. This way is easy but there are three problems. Firstly, it need human intervention  to change the configuration file when a failure occurs. Secondly it takes time (proportional to the volume of data) to rollback, the downtime would be 52.56 minutes a year if the system claims to be 99.9999% HA. it may not be a very nice solution for large scale system. Thirdly, it is not cost-effective to buy a standby machine which is idle in most of the time. So what about MySql Cluster? It is a good idea if you can pay expensive license fee and adapt to a more complicated replication mechanism. The semi-replication mechanism causes additional synchronization, thus it requires hardwares with higher performance and hence higher cost. Another solution builds the cluster on SAN, for example of Oracle RAC. This solution offers high performance in synchronization and high scalablity, but the SAN is a Single point of failure itself. Today I want to introduce a High Availability solution which is open sourced, offers real time data synchronization, robust data protection, replication, and automation.

First of all, please familiar with yourself about DRBD from Linbit. I would like to share a successful case of deploying High Availability solution based on DRDB technology. I am not responsible and liable in the demonstration. Please test it thoroughly before applying to the production environment.

1. DRBD User Guide (8.3.x)

2. Linux HA User Guide

3. Combine GFS2 with DRBD

4. MySQL Availability with Red Hat Enterprise Linux

5. Dual Primary (Think Twice)

Background

In general, it is not possible to synchronize two computer systems (file systems) which issue disk write operation concurrently. GFS2 is a file system developed by Red Hat to solve the file system level synchronization. I am here to demonstrate how to make use of DRBD of Linbit, CMAN, and GFS2 to implement the Dual Primary system to achieve the High Availability.

Table of Content

0. Environment

1. Installation of operating system

2. Network configuration

3. Installation of softwares

4. DRBD initialization and first synchronization

5. GFS2 formating and mounting

6. Testing

0. Environment

Hardware environment

Node 1 (sony.localdomain)
  • Intel Pentium (R) Dual CPU T2330 @ 1.60 GHz
  • 2GB RAM
Node 2 (dell2.localdomain)
  • Intel Core2Duo CPU E7200 @ 2.53 GHz
  • 4GB RAM

Software environment (identical on both computer)

1. Operating system: CentOS 6.4 (Final) kernel 2.6.32-358.el6.i686
2. cman
3. gfs-utils
4. kmod-dlm
5. modcluster
6. ricci
7. luci
8. cluster-snmp
9. isci-initiator-utils
10. openais
11. oddjobs
12. rgmanager

1. Installation of operating system

GFS2 is developed by Red Hat. You need to pay for the Cluster suite (the software stated above), so I will demonstrate on CentOS 6.4 and install the softwares manually. Firstly, you have to prepare two PCs for installing CentOS 6.4. I would stress to use two physical machines and connected with cross-over Ethernet cable because the performance will be better. If you got two PCs with Windows installed, you can install VM Player to host the CentOS. It is not recommended to use a single Windows PC to create two VM Player instances to host the CentOS. It is possible theoretically but it will be less realistic.

1.1 Install CentOS 6.4 (non VM)

1.1.1 Download CentOS 6.4 ISO image ,make it into a bootable DVD disk.

1.1.2 Install the CentOS on both computers. Click “Next” button until seeing “Which type of installation would you like?”. Because it need a stand alone partition to install DRBD, but it offers only one boot partition and another one partition for the OS, so we need to choose “Create custom layout” as the image shown as followed

Image

1.1.3 We re-organize the partition layout by deleting the existing partitions and creating new partitions. We use sda1 for boot partition, sda2 for LVM, sda3 for GFS2 partition. It is advised to leave some unused space for future expansion. Please note that don’t set the size of the GFS2 partition too large as it takes long time to synchronize. 10 GB or 20 GB is a good choice for testing purpose.

Image

After finishing the partition layout, follow the instructions to complete the whole installation.

1.2.1 Install CentOS 6.4 (VM)

For Windows installation, firstly download and install VM Player 5.0.2, and also CentOS 6.4 ISO image. To create a VM Player instance, choose “Edit virtual machine setting” on the VM instance, and then click “create new virtual disk”, follow the instructions to complete the installation, and you will see the new partition sdb.

Image

Image

2 Network configuration

2.1 To minimize network latency, use cross-over Ethernet cable to connect both computer on their NIC, and configure as followed.

root@dell2# cat /etc/sysconfig/network-scripts/ifcfg-eth0
DEVICE=eth0
TYPE=Ethernet
ONBOOT=yes
BOOTPROTO=none
IPADDR=192.168.1.16
BROADCAST=192.168.1.255
NETMASK=255.255.255.0
NETWORK=192.168.1.0
IPV6INIT=no
IPV4_FAILURE_FATAL=yes
root@sony# cat /etc/sysconfig/network-scripts/ifcfg-eth0
DEVICE=eth0
TYPE=Ethernet
ONBOOT=yes
BOOTPROTO=none
IPADDR=192.168.1.15
BROADCAST=192.168.1.255
NETMASK=255.255.255.0
NETWORK=192.168.1.0
IPV6INIT=no
IPV4_FAILURE_FATAL=yes

 2.2 DRBD configuration needs to use hostname, so configure the hostname as followed.

root@your_machine# cat /etc/hosts
192.168.1.15 sony.localdomain
192.168.1.16 dell2.localdomain
Restart the network service and test it the connectivity by pinging each other.
3. Software installation
3.1 Use yum command to install the software.

root@your_machine# yum install -y cman gfs2-utils kmod-gfs kmod-dlm 
modcluster ricci luci cluster-snmp iscsi-initiator-utils openais oddjob rgmanager
3.2 Because DRDB does not have a yum repository, you need a little bit more effort. Install Erlang on both computers.
root@your_machine# wget http://elrepo.org/
elrepo-release-6-4.el6.elrepo.noarch.rpm#sthash.busZ7CGJ.dpuf
Install it on both computers.
root@your_machine# rpm -ivUh elrepo-release-6-4.el6.elrepo.noarch.rpm
Change the configuration of Erlang on both computers. change the 8th line as ‘enable=0’
root@your_machine# gedit /etc/yum.repos.d/elrepo.repo
3.3 Now you can use yum to install DRBD
root@your_machine# yum --enablerepo=elrepo install drbd83-utils kmod-drbd83
3.4 Create the cluster configuration file on both computers as followed,
root@your_machine# gedit /etc/cluster/cluster.conf
<?xml version="1.0"?>
<cluster alias="cluster-setup" config_version="1" name="cluster-setup">
<rm log_level="4"/>
<fence_daemon clean_start="1" post_fail_delay="0" post_join_delay="3"/>
<clusternodes>
  <clusternode name="sony.localdomain" nodeid="1" votes="1">
    <fence>
      <method name="2">
        <device name="LastResortNode01"/>
      </method>
    </fence>
  </clusternode>
  <clusternode name="dell2.localdomain" nodeid="2" votes="1">
    <fence>
      <method name="2">
        <device name="LastResortNode02"/>
      </method>
    </fence>
  </clusternode>
</clusternodes>
<cman expected_votes="1" two_node="1"/>
<fencedevices>
  <fencedevice agent="fence_manual" name="LastResortNode01" nodename="sony.localdomain"/>
  <fencedevice agent="fence_manual" name="LastResortNode02" nodename="dell2.localdomain"/>
</fencedevices>
<rm/>
<totem consensus="4800" join="60" token="10000" token_retransmits_before_loss_const="20"/>
</cluster>
3.5 Configure the DRBD setting as followed, if there are include “drbd.d/global_common.conf”; 和 include “drbd.d/*.res”; on the top of the file, just remark them.

root@your_machine# gedit /etc/drbd.conf
global { usage-count yes; }
common { syncer { rate 100M; } }
resource res2 {
  protocol C;
  startup {
    wfc-timeout 20;
    degr-wfc-timeout 10;
    # we will keep this commented until tested successfully:
    # become-primary-on both; 
  }
  net {
    # the encryption part can be omitted when using a dedicated link for DRBD only:
    # cram-hmac-alg sha1;
    # shared-secret anysecrethere123;
    allow-two-primaries;
  }
  on sony.localdomain {
    device /dev/drbd2;
    disk /dev/sda3;
    address 192.168.1.15:7789;
    meta-disk internal;
  }
  on dell2.localdomain {
    device /dev/drbd2;
    disk /dev/sda3;
    address 192.168.1.16:7789;
    meta-disk internal;
  }
  disk {
    fencing resource-and-stonith;
  }
  handlers {
    #outdate-peer "/sbin/handler";
  }
}
  • ‘resource’ is the reference name in DRBD configuration. I suggest to use ‘res2’ to point to ‘/dev’drbd2’ and use ‘res0’ to point to ‘/dev/drbd0’ for ease of management. The order of the device is not important.
  • ‘device’ is the default path of the DRBD device. After DRBD is installed, the default devices will be shown as /dev/drbd0, /dev/drbd1, …., /dev/drbd9 .
  • ‘disk’ is the hard disk partition for synchronization, we will format it as GFS2 a while later, and which we have already prepared in section 1.
  • ‘address’ is the IP address of the computer. The default port number will be 7789.
3.6 To automate the startup of DRBD in every reboot, change the configuration as followed.

root@your_machine# gedit /etc/init.d/drbd
Change # chkconfig: 345 70 08 to # chkconfig: 345 22 78
Note that ‘#’ is not a remark syntax, please don’t remove it.
3.7 Configure firewall on both computer
root@your_machine# iptables -I OUTPUT -o eth0 -j ACCEPT
root@your_machine# iptables -I INPUT -i eth0 -j ACCEPT
root@your_machine# service iptables save

4. DRBD initialization and first synchronization

4.1 Start the DRBD services on both computers

root@your_machine# service drbd start
4.2 Create the meta data on both computer

root@your_machine# drbdadm create-md res2

If you got error like this “exited with code 40″

Device size would be truncated, which would corrupt data and result in 'access beyond end of device' errors. You need to either * use external meta data (recommended) * shrink that filesystem first * zero out the device (destroy the filesystem) Operation refused. Command 'drbdmeta 0 v08 /dev/hdb1 internal create-md' terminated with exit code 40 drbdadm create-md ha: exited with code 40
Use dd command to fill some bits of data in the disk partition, and then re-execute drbdadm create-md res2
root@your_machine# dd if=/dev/zero of=/dev/hdb1 bs=1M count=100
4.3 Communicate with each other

root@your_machine#  drbdadm up res2
4.4 Check the status of each other
root@your_machine#  #drbd-overview
Status will look alike this:
1:res2  Connected Secondary/Secondary Inconsistent/Inconsistent C
r----
4.5 On either one (and only one) computer to issue a synchronize command. It takes 10 mins to finish a 10GB hard disk partition for my case. You are free to check the status during the period of the process.
# drbdadm -- --overwrite-data-of-peer primary res2
The status will look alike this after synchronization complete.
1:res2  Connected Primary/Secondary UpToDate/UpToDate C r----
4.6 We need to achieve “Dual Primary”, so you need to remove the remark on this line ‘become-primary-on both;’.

#gedit /etc/drbd.conf
 5. GFS2 formatting and mounting

5.1 Format the hard disk partition as GFS2 on each computer.
# mkfs.gfs2 -p lock_dlm -t cluster-setup:res2 /dev/drbd2 -j 2
5.2 Stop the Network Manager, and start the (Fencing device) CMAN.
# /etc/init.d/NetworkManager stop  
 # service cman start
5.3 Mount the formatted partition to a directory, you must start CMAN before mounting
# mkdir /mnt/ha

# mount -t gfs2 -o noatime /dev/drbd2 /mnt/ha
6 Testing
Issue create files, modify files, delete files in the /mnt/ha directory to verify whether the synchronization succeed or not. CMAN takes effect and monitor the connectivity of either sides until the shutdown of the computer. If there is a network failure, computer shutdown (accidentally or planned), mal function of hardware, DRBD will enter protection mode until the problem is rectified, and then it will synchronize both computer (the data image written in during the down time ) to resume the service.
This demonstration show you how to achieve disk level High Availability. It has satisfied the requirement of real time synchronization, zero down-time, relatively short replication time, and automation.


1 Comment

用兩台家用電腦實現高可用性部署

隨著企業對數據的重視程度不斷提高,「高可用性」(High Availability) 已經成為電腦系統的標配。「高可用性」不是一個軟件或硬件,更加不單是一個負載均衡器 (Load balance device)就能全面達致的系統特性。由於「單一故障點」(Single point of failure)有機會在網絡、硬件、軟件,甚至業務邏輯上都會發生,在有限的預算下是沒有完美的解決方案。假使你運行一個B2B的貿易平台,應用侍服器做了冗餘(redundancy),遇上某一台侍服器的應用服務故障,用戶的請求可以透過負載均衡器分發到運作正常的侍服器。數據庫也用主備的方式實時同步,即使主數據庫故障也不致數據丟失。而公司也安裝了 UPS 裝置和後備電源,即使大廈的電力系統故障,你的平台也有足夠的後備電源持續一段時間運行。你也擔心網絡商的網絡故障,所以也多找另一間網絡商做冗餘。完美吧﹖突如其來的網絡支付服務發生短暫的故障,又或是你辦公室隔鄰的單位發生火警,消防員努力向火場射水撲救之時,大量的水把你的主電源,後備電源一并報銷,你還是不得不投降。所以「高可用性」是沒有完美方案的。當然你的預算充裕,可以把最大部分的風險降低,預算少的,把最大的風險消除後,就把沒法保障的風險讓 Stakeholder 徹底了解,並實行相應的應變措施。

「高可用性」在網絡、硬件上都有很成熟的支援,只要偵測到問題,冗餘/後備的元件很短時間就可上線運作。但數據庫的「高可用性」上,本質上要解決同步、一致性、恢復等機制,非一般的數據庫部署方式能解決。數據要得到保障,最簡單就是把數據實時複制到另一個數據庫中。數據庫的複制需要把日誌由主庫複制到備庫。當原本的數據庫發生故障時,把應用的連接轉移到備分的數據庫就可繼續運行,同時把備庫的日誌回滾到主庫就可恢復。這是最簡單的做法,但存在幾個問題,第一,在發生固障,完成服務轉移之後,主、備數據庫的配置需要人為操作,才能保持「高可用」的特性。第二,日誌的回滾需時,按數據量成正比,這段時間就包括在 down time 之中,聲稱99.9999 HA 的系統一年可包容52.56分鐘的 down time。以這方法來看並不適合。第三,兩台侍服器只有一台在運作,也未必是很化算。你會問 MySql 的 Cluster 方案可行嗎﹖只要你願意付費買高昂的License和維護一個更複雜的機制,這也是一個可靠的方案,但由於每個事務 (transaction) 的同步以半同步的方式進行,亦即集群需要處理大量的同步數據,這複雜的機制非一般性能的侍服器能膀任。另一種方案依賴 SAN 儲存架構做集群,像 Oracle 的 RAC 方案,優點是同步很快,擴容能力很高,缺點是 SAN 本報是一個 Single point of failure。今天要介紹的方案,可以完全以 open source 的軟件實現,High Availability上可達致實時同步,有扎實的保護、徹換、恢復、自動化的機制。

開始前,敬請仔細閱讀 DRBD 開發商 Linbit 的技術文件。本文章旨在分享一個以 DRBD 技術為核心的成功案例,不涉及任何法律責任,建議徹底測試才應用在生產環境。

1. DRBD User Guide (8.3.x)

2. Linux HA User Guide

3. Combine GFS2 with DRBD

4. MySQL Availability with Red Hat Enterprise Linux

5. Dual Primary (Think Twice)

背景

兩台同時進行寫的電腦要執行同步在一般的文件系統 (File system) 上是做不做的。針對處理文件系統同步的GFS2應運而生。本文示範怎樣運用 Red Hat 的 GFS2 , Linbit 的 DRBD 和 CMAN  來實現 Dual Primary 系統。

大綱

0. 環境

1. 操作系統安裝

2. 網絡設定

3. 軟件安裝

4. DRBD 初始化和首次同步

5. GFS2 執行格式化和掛載

6. 測試

0. 環境

硬件環境

Node 1 (sony.localdomain)
  • Intel Pentium (R) Dual CPU T2330 @ 1.60 GHz
  • 2GB RAM
Node 2 (dell2.localdomain)
  • Intel Core2Duo CPU E7200 @ 2.53 GHz
  • 4GB RAM

軟件環境 (兩台電腦完全相同)

1. 操作系統: CentOS 6.4 (Final) kernel 2.6.32-358.el6.i686
2. cman
3. gfs-utils
4. kmod-dlm
5. modcluster
6. ricci
7. luci
8. cluster-snmp
9. isci-initiator-utils
10. openais
11. oddjobs
12. rgmanager

1. 操作系統安裝

GFS2 是 Red Hat 的原生項目,不過以上的集群軟件套裝在 Red Hat 上是收費的,所以我在 CentOS 6.4 上進行示範。首先需要兩台電腦,普通的卓面電腦就可以了(會安裝上CentOS 6.4),重點是兩台物理機,因為稍後要用 cross-over cable 連接起來,會比較真實。如果你用的是兩台Microsoft Windows 電腦,可以安裝 VM Player 來 host CentOS 也可。千萬不要用一台 Microsoft Windows 電腦來安裝兩個 VM Player instance 來host CentOS,事實上是可行的,但希望你可以感受一下從兩台電腦同步的真實情況。

1.1 安裝 CentOS 6.4 (非VM)

1.1.1 下載 CentOS 6.4 的 ISO 影像,制成 bootable 的 DVD

1.1.2 分別在兩台電腦安裝,按指示進行,直至 “Which type of installation would you like?”, 由於DRBD 需要一個獨立的 partition 來進行同步,但默認的安裝只得一個 boot partition 和一個主要的 partition 來安裝操作系統,因此我們要選 “Create custom layout”,如下圖。

Image

1.1.3 把原有的 partition 重新劃分,sda1 作 boot partition,sda2 作 LVM,sda3 作 GFS2 partition。 預留一點空間不作分區,作將來之用。留意試測時不要把 GFS2 區間設得太大,否則同步的時間會很長。

Image

完成區間劃分可繼續至安裝,直至整個 CentOS 6.4安裝完成。

1.2.1 安裝 CentOS 6.4 (VM)

如果是從 Windows 上安裝,先下載和安裝VM Player 5.0.2。然後下載 CentOS 6.4 的 ISO 影像在VM Player中新增 CentOS 6.4 的系統,完成後在VM instance 上按 Edit virtual machine setting,然後按 “create new virtual disk”,按指示完成後,sdb 就會顥示出來。

Image

Image

2 網絡設定

2.1 為減少 network latency,直接用 cross-over Ethernet cable 把各自的 NIC 連上,並設定如下,

root@dell2# cat /etc/sysconfig/network-scripts/ifcfg-eth0
DEVICE=eth0
TYPE=Ethernet
ONBOOT=yes
BOOTPROTO=none
IPADDR=192.168.1.16
BROADCAST=192.168.1.255
NETMASK=255.255.255.0
NETWORK=192.168.1.0
IPV6INIT=no
IPV4_FAILURE_FATAL=yes
root@sony# cat /etc/sysconfig/network-scripts/ifcfg-eth0
DEVICE=eth0
TYPE=Ethernet
ONBOOT=yes
BOOTPROTO=none
IPADDR=192.168.1.15
BROADCAST=192.168.1.255
NETMASK=255.255.255.0
NETWORK=192.168.1.0
IPV6INIT=no
IPV4_FAILURE_FATAL=yes
 2.2 在DRBD的配置中需用上 host name,所以要設定如下,

root@your_machine# cat /etc/hosts
192.168.1.15 sony.localdomain
192.168.1.16 dell2.localdomain
 重啟 network 服務後,現在可以用 host name 互相 ping。
3. 軟件安裝
3.1 用 yum 命令安裝軟件

root@your_machine# yum install -y cman gfs2-utils kmod-gfs kmod-dlm 
modcluster ricci luci cluster-snmp iscsi-initiator-utils openais oddjob rgmanager
3.2  因為 DRBR 沒有 yum 的 repository,所以安裝時需要一點功夫。首先在兩台電腦各自下載 Erlang。
root@your_machine# wget http://elrepo.org/
elrepo-release-6-4.el6.elrepo.noarch.rpm#sthash.busZ7CGJ.dpuf
然後各自安裝
root@your_machine# rpm -ivUh elrepo-release-6-4.el6.elrepo.noarch.rpm
各自修改 Erlang 的配置,把第八行修改為 ‘enable=0’
root@your_machine# gedit /etc/yum.repos.d/elrepo.repo
3.3 這時可用 yum 命令安裝 DRBD 了
root@your_machine# yum --enablerepo=elrepo install drbd83-utils kmod-drbd83
3.4 各自創建 Cluster 的配置文件如下,
root@your_machine# gedit /etc/cluster/cluster.conf
<?xml version="1.0"?>
<cluster alias="cluster-setup" config_version="1" name="cluster-setup">
<rm log_level="4"/>
<fence_daemon clean_start="1" post_fail_delay="0" post_join_delay="3"/>
<clusternodes>
  <clusternode name="sony.localdomain" nodeid="1" votes="1">
    <fence>
      <method name="2">
        <device name="LastResortNode01"/>
      </method>
    </fence>
  </clusternode>
  <clusternode name="dell2.localdomain" nodeid="2" votes="1">
    <fence>
      <method name="2">
        <device name="LastResortNode02"/>
      </method>
    </fence>
  </clusternode>
</clusternodes>
<cman expected_votes="1" two_node="1"/>
<fencedevices>
  <fencedevice agent="fence_manual" name="LastResortNode01" nodename="sony.localdomain"/>
  <fencedevice agent="fence_manual" name="LastResortNode02" nodename="dell2.localdomain"/>
</fencedevices>
<rm/>
<totem consensus="4800" join="60" token="10000" token_retransmits_before_loss_const="20"/>
</cluster>
3.5 設定 DRBD 配置如下,如有 include “drbd.d/global_common.conf”; 和 include “drbd.d/*.res”; 把它們注釋掉。

root@your_machine# gedit /etc/drbd.conf
global { usage-count yes; }
common { syncer { rate 100M; } }
resource res2 {
  protocol C;
  startup {
    wfc-timeout 20;
    degr-wfc-timeout 10;
    # we will keep this commented until tested successfully:
    # become-primary-on both; 
  }
  net {
    # the encryption part can be omitted when using a dedicated link for DRBD only:
    # cram-hmac-alg sha1;
    # shared-secret anysecrethere123;
    allow-two-primaries;
  }
  on sony.localdomain {
    device /dev/drbd2;
    disk /dev/sda3;
    address 192.168.1.15:7789;
    meta-disk internal;
  }
  on dell2.localdomain {
    device /dev/drbd2;
    disk /dev/sda3;
    address 192.168.1.16:7789;
    meta-disk internal;
  }
  disk {
    fencing resource-and-stonith;
  }
  handlers {
    #outdate-peer "/sbin/handler";
  }
}
  • ‘resource’ 是 DRBD 設定的參考名字。建議用 ‘res2’ 來指向 ‘/dev/drbd2’, ‘res0’ 來指向 ‘/dev/drbd0’ 以方便管理。這裡不一要順序由 /dev/drbd0 開始用。
  • ‘device’ 是默認的 DRBD 設備的路徑。當 DRBD 安裝後,默認的設備 /dev/drbd0, /dev/drbd1, …., /dev/drbd9 就會顯示出來。
  • ‘disk’ 是用作同步的硬盤區間,稍後我們會把它格式化為 GFS2。我們在 Section 1 的時候已準備好了。
‘address’ 是電腦的 I.P. 地址。默認的端口為 7789。
3.6 要讓 DRBD 每次重啟都自動運行,修改配置如下,

root@your_machine# gedit /etc/init.d/drbd
把 # chkconfig: 345 70 08 改為 # chkconfig: 345 22 78
 留意 ‘#’ 不是注滿符號,不要刪去。
3.7 各自配置防火牆。
root@your_machine# iptables -I OUTPUT -o eth0 -j ACCEPT
root@your_machine# iptables -I INPUT -i eth0 -j ACCEPT
root@your_machine# service iptables save 

4. DRBD 初始化和首之同步

4.1 各自啟用 DRBD 服務

root@your_machine# service drbd start
4.2 各自創造 meta data

root@your_machine# drbdadm create-md res2

如果遇上錯誤 ”exited with code 40″

Device size would be truncated, which would corrupt data and result in 'access beyond end of device' errors. You need to either * use external meta data (recommended) * shrink that filesystem first * zero out the device (destroy the filesystem) Operation refused. Command 'drbdmeta 0 v08 /dev/hdb1 internal create-md' terminated with exit code 40 drbdadm create-md ha: exited with code 40
 用 dd 命令在硬盤區間注入一些數據,然後重執行 drbdadm create-md res2
root@your_machine# dd if=/dev/zero of=/dev/hdb1 bs=1M count=100
4.3 讓兩台電腦互通一下清息

root@your_machine#  drbdadm up res2
4.4 各自查看一下目前狀態
root@your_machine#  #drbd-overview
 情況如下:
1:res2  Connected Secondary/Secondary Inconsistent/Inconsistent C
r---- 
4.5 在其中一部電腦發出執行同步命令。在我的10GB硬盤區間大約了10多分鐘完成。其間你可以用drbd-overview 查閱同步進度。
# drbdadm -- --overwrite-data-of-peer primary res2
完成同步後,各自的狀態都如下
1:res2  Connected Primary/Secondary UpToDate/UpToDate C r---- 
4.6 由於最終目標是Dual Primary,所以修改DRBD 配置把 ‘become-primary-on both;’ 的註釋除掉。

#gedit /etc/drbd.conf 
5. GFS2 執行格式化和掛載
5.1 各自格式化為GFS2
# mkfs.gfs2 -p lock_dlm -t cluster-setup:res2 /dev/drbd2 -j 2
5.2 開啟保護設備 (Fencing device) CMAN,在執行前記得把 Network Manager 關掉。
# /etc/init.d/NetworkManager stop  
 # service cman start
5.3 掛載已格式化的區間到一個目錄,掛載前一定要開啟 CMAN
# mkdir /mnt/ha

# mount -t gfs2 -o noatime /dev/drbd2 /mnt/ha
6 測試
在各自的 /mnt/ha 執行創建、更改、刪除文件,驗證兩台現腦能否實時同步。CMAN 會生效直至電腦關閉。它會監控各自的電腦,如果發生網絡終斷,電腦關閉,硬件固障等,DRBD 會進入保護狀態,如果問題已經修正,DRBD 重啟後會作新一次同步 (單機服務間寫入的數據得以保存),並恢復正常狀態。
這部署示範了disk level 的 「高可用性」優點,本質上滿足了實時同步,零終斷,短時間恢復(相比日誌回滾),全自動化的要求。