[CentOS7] mdadm software RAID + LVM 구성

개인적으로 운영하는 ftp 를 lvm 으로 묶어 Storage 형태로 만들어 사용 하고 있습니다.

한 3~4년 주기로 깨지다 보니 그때마다 lvm 복구 작업을 종종 합니다.

Linux Soft Raid 를 이용하여 md1 + md2 를 구성하여 lvm 을 만들어 사용할까 합니다.

Test 환경의 경우 Centos 7 Version 에 1G disk 4개를 추가 하여 sdb + sdc = md1 , sdd + sde = md2 로 구성 하였습니다.

주의: dd 명령으로 복구 테스트하였을 경우 raid 볼륨 자체가 깨지는 문제로 테스트가 되지 않습니다. 🙂 

 

 

  • Disk 정보 (sdb / sdc /sdd /sde 총 4장의 1G Disk 가 장착되어 있습니다.)
[root@centos7 ~]# lsblk
NAME   MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
sda      8:0    0   20G  0 disk
├─sda1   8:1    0    1G  0 part /boot
├─sda2   8:2    0    1G  0 part [SWAP]
└─sda3   8:3    0   18G  0 part /
sdb      8:16   0    1G  0 disk
sdc      8:32   0    1G  0 disk
sdd      8:48   0    1G  0 disk
sde      8:64   0    1G  0 disk
[root@centos7 ~]#

 

  • mdadm 설치 
[root@centos7 ~]# yum install -y mdadm

 

  • raid 장치 확인 
[root@centos7 ~]# cat /proc/mdstat
Personalities :
unused devices: <none>
[root@centos7 ~]#

 

  • fdisk 작업 (sdb / sdc /sdd /sde )
[root@centos7 ~]# fdisk /dev/sdb
Welcome to fdisk (util-linux 2.23.2).

Changes will remain in memory only, until you decide to write them.
Be careful before using the write command.

Device does not contain a recognized partition table
Building a new DOS disklabel with disk identifier 0x051d4fae.

Command (m for help): n
Partition type:
   p   primary (0 primary, 0 extended, 4 free)
   e   extended
Select (default p): p
Partition number (1-4, default 1): 1
First sector (2048-2097151, default 2048):
Using default value 2048
Last sector, +sectors or +size{K,M,G} (2048-2097151, default 2097151):
Using default value 2097151
Partition 1 of type Linux and of size 1023 MiB is set

Command (m for help): t
Selected partition 1
Hex code (type L to list all codes): fd
Changed type of partition 'Linux' to 'Linux raid autodetect'

Command (m for help): wq
The partition table has been altered!

Calling ioctl() to re-read partition table.
Syncing disks.
[root@centos7 ~]#

~중략

[root@centos7 ~]# partprobe

 

  • /dev/sdb1 , /dev/sdc1 디스크를 이용하여 raid 1 md1 를 구성 합니다.
[root@centos7 ~]# mdadm --create /dev/md1 --level=1 --raid-device=2 /dev/sdb1 /dev/sdc1
mdadm: Note: this array has metadata at the start and
    may not be suitable as a boot device.  If you plan to
    store '/boot' on this device please ensure that
    your boot-loader understands md/v1.x metadata, or use
    --metadata=0.90
Continue creating array? y
mdadm: Fail create md1 when using /sys/module/md_mod/parameters/new_array
mdadm: Defaulting to version 1.2 metadata
mdadm: array /dev/md1 started.
[root@centos7 ~]#

 

  • /dev/sdd1 , /dev/sde1 디스크를 이용하여 raid 1 md2 를 구성 합니다. 
[root@centos7 ~]# mdadm --create /dev/md2 --level=1 --raid-device=2 /dev/sdd1 /dev/sde1
mdadm: Note: this array has metadata at the start and
    may not be suitable as a boot device.  If you plan to
    store '/boot' on this device please ensure that
    your boot-loader understands md/v1.x metadata, or use
    --metadata=0.90
Continue creating array? y
mdadm: Fail create md2 when using /sys/module/md_mod/parameters/new_array
mdadm: Defaulting to version 1.2 metadata
mdadm: array /dev/md2 started.
[root@centos7 ~]#

 

  • raid 상태 확인 
[root@centos7 ~]# cat /proc/mdstat
Personalities : [raid1]
md2 : active raid1 sde1[1] sdd1[0]
      1046528 blocks super 1.2 [2/2] [UU]

md1 : active raid1 sdc1[1] sdb1[0]
      1046528 blocks super 1.2 [2/2] [UU]

unused devices: <none>
[root@centos7 ~]#

 

  • /etc/mdadm.conf 를 생성합니다.
[root@centos7 ~]# mdadm --detail --scan > /etc/mdadm.conf
[root@centos7 ~]# cat /etc/mdadm.conf
ARRAY /dev/md1 metadata=1.2 name=centos7:1 UUID=33827f12:f44165a0:19e7a7c9:18f88f40
ARRAY /dev/md2 metadata=1.2 name=centos7:2 UUID=b17e111b:e22dcf62:51b929f6:819a5c2c
[root@centos7 ~]#

 

  • lvm2 패키지를 설치 합니다.
[root@centos7 ~]# yum install lvm2 -y

 

  • lvm 생성후 xfs 파일시스템으로 포맷을 합니다.
[root@centos7 ~]# pvcreate /dev/md1 /dev/md2
  Physical volume "/dev/md1" successfully created.
  Physical volume "/dev/md2" successfully created.
[root@centos7 ~]# vgcreate vg00 /dev/md1 /dev/md2
  Volume group "vg00" successfully created
[root@centos7 ~]# lvcreate -l 100%free -n data01 vg00
  Logical volume "data01" created.
[root@centos7 ~]#

[root@centos7 ~]# mkfs.xfs /dev/vg00/data01
meta-data=/dev/vg00/data01       isize=512    agcount=4, agsize=130560 blks
         =                       sectsz=512   attr=2, projid32bit=1
         =                       crc=1        finobt=0, sparse=0
data     =                       bsize=4096   blocks=522240, imaxpct=25
         =                       sunit=0      swidth=0 blks
naming   =version 2              bsize=4096   ascii-ci=0 ftype=1
log      =internal log           bsize=4096   blocks=2560, version=2
         =                       sectsz=512   sunit=0 blks, lazy-count=1
realtime =none                   extsz=4096   blocks=0, rtextents=0
[root@centos7 ~]#

 

  • /etc/fstab 에 마운트 후 시스템 리부팅후 마운트 상태를 확인 합니다. 
[root@centos7 ~]# mkdir /data
[root@centos7 ~]# vi /etc/fstab

/dev/vg00/data01                          /data                   xfs     defaults        0 0

[root@centos7 ~]# mount -a
[root@centos7 ~]# df -h
Filesystem               Size  Used Avail Use% Mounted on
/dev/sda3                 18G  1.4G   17G   8% /
devtmpfs                 903M     0  903M   0% /dev
tmpfs                    912M     0  912M   0% /dev/shm
tmpfs                    912M  8.7M  903M   1% /run
tmpfs                    912M     0  912M   0% /sys/fs/cgroup
/dev/sda1               1014M  135M  880M  14% /boot
tmpfs                    183M     0  183M   0% /run/user/0
/dev/mapper/vg00-data01  2.0G   33M  2.0G   2% /data
[root@centos7 ~]# init 6 
[root@centos7 ~]# df -h |grep -i data
/dev/mapper/vg00-data01  2.0G   33M  2.0G   2% /data
[root@centos7 ~]#

 

  • mdadm 명령어를 이용하여 raid 정보를 확인 합니다. 
[root@centos7 ~]# mdadm --detail /dev/md1
/dev/md1:
           Version : 1.2
     Creation Time : Thu Jul 18 00:29:08 2019
        Raid Level : raid1
        Array Size : 1046528 (1022.00 MiB 1071.64 MB)
     Used Dev Size : 1046528 (1022.00 MiB 1071.64 MB)
      Raid Devices : 2
     Total Devices : 2
       Persistence : Superblock is persistent

       Update Time : Thu Jul 18 00:34:49 2019
             State : clean
    Active Devices : 2
   Working Devices : 2
    Failed Devices : 0
     Spare Devices : 0

Consistency Policy : resync

              Name : centos7:1  (local to host centos7)
              UUID : 33827f12:f44165a0:19e7a7c9:18f88f40
            Events : 17

    Number   Major   Minor   RaidDevice State
       0       8       17        0      active sync   /dev/sdb1
       1       8       33        1      active sync   /dev/sdc1
[root@centos7 ~]# mdadm --detail /dev/md2
/dev/md2:
           Version : 1.2
     Creation Time : Thu Jul 18 00:30:30 2019
        Raid Level : raid1
        Array Size : 1046528 (1022.00 MiB 1071.64 MB)
     Used Dev Size : 1046528 (1022.00 MiB 1071.64 MB)
      Raid Devices : 2
     Total Devices : 2
       Persistence : Superblock is persistent

       Update Time : Thu Jul 18 00:35:22 2019
             State : clean
    Active Devices : 2
   Working Devices : 2
    Failed Devices : 0
     Spare Devices : 0

Consistency Policy : resync

              Name : centos7:2  (local to host centos7)
              UUID : b17e111b:e22dcf62:51b929f6:819a5c2c
            Events : 17

    Number   Major   Minor   RaidDevice State
       0       8       49        0      active sync   /dev/sdd1
       1       8       65        1      active sync   /dev/sde1
[root@centos7 ~]#

 

  • mdadm -f 옵션을 이용하여 /dev/md1 에 sdc1 디스크 장애 상황을 만듭니다. 
[root@centos7 ~]# mdadm /dev/md1 -f /dev/sdc1
mdadm: set /dev/sdc1 faulty in /dev/md1
[root@centos7 ~]# cat /proc/mdstat
Personalities : [raid1]
md2 : active raid1 sde1[1] sdd1[0]
      1046528 blocks super 1.2 [2/2] [UU]

md1 : active raid1 sdc1[1](F) sdb1[0]
      1046528 blocks super 1.2 [2/1] [U_] <---- UU 에서 U_ 로 변경 되었습니다.

unused devices: <none>
[root@centos7 ~]#


마운트는 정상적으로 되어 있고 touch 로 빈파일생성도 정상적으로 됩니다.

[root@centos7 ~]# df -h
Filesystem               Size  Used Avail Use% Mounted on
/dev/sda3                 18G  1.4G   17G   8% /
devtmpfs                 903M     0  903M   0% /dev
tmpfs                    912M     0  912M   0% /dev/shm
tmpfs                    912M  8.7M  904M   1% /run
tmpfs                    912M     0  912M   0% /sys/fs/cgroup
/dev/sda1               1014M  135M  880M  14% /boot
/dev/mapper/vg00-data01  2.0G   33M  2.0G   2% /data
tmpfs                    183M     0  183M   0% /run/user/0
[root@centos7 ~]# touch /data/0
[root@centos7 ~]# touch /data/1
[root@centos7 ~]# touch /data/2

 

  • 디스크 remove 의 경우 mdadm $mdX_name –remove $device_name 으로 디스크를 제거 하시면 됩니다.
[root@centos7 ~]# mdadm /dev/md1 --remove /dev/sdc1
mdadm: hot removed /dev/sdc1 from /dev/md1
[root@centos7 ~]# cat /proc/mdstat
Personalities : [raid1]
md2 : active raid1 sde1[1] sdd1[0]
      1046528 blocks super 1.2 [2/2] [UU]

md1 : active raid1 sdb1[0]                <--- sdc1 Disk 가 제거된것을 확인할수 있습니다.
      1046528 blocks super 1.2 [2/1] [U_]

unused devices: <none>
[root@centos7 ~]#

 

  • 시스템 shutdown 후 Disk 를 추가 합니다. 
[root@centos7 ~]# init 0

[root@centos7 ~]# lsblk
NAME              MAJ:MIN RM  SIZE RO TYPE  MOUNTPOINT
sda                 8:0    0   20G  0 disk
├─sda1              8:1    0    1G  0 part  /boot
├─sda2              8:2    0    1G  0 part  [SWAP]
└─sda3              8:3    0   18G  0 part  /
sdb                 8:16   0    1G  0 disk
└─sdb1              8:17   0 1023M  0 part
  └─md1             9:1    0 1022M  0 raid1
    └─vg00-data01 253:0    0    2G  0 lvm   /data
sdc                 8:32   0    1G  0 disk
└─sdc1              8:33   0 1023M  0 part
sdd                 8:48   0    1G  0 disk
└─sdd1              8:49   0 1023M  0 part
  └─md2             9:2    0 1022M  0 raid1
    └─vg00-data01 253:0    0    2G  0 lvm   /data
sde                 8:64   0    1G  0 disk
└─sde1              8:65   0 1023M  0 part
  └─md2             9:2    0 1022M  0 raid1
    └─vg00-data01 253:0    0    2G  0 lvm   /data
sdf                 8:80   0    1G  0 disk
sr0                11:0    1 1024M  0 rom
[root@centos7 ~]# fdisk /dev/sdf
Welcome to fdisk (util-linux 2.23.2).

Changes will remain in memory only, until you decide to write them.
Be careful before using the write command.

Device does not contain a recognized partition table
Building a new DOS disklabel with disk identifier 0xb67cc56a.

Command (m for help): n
Partition type:
   p   primary (0 primary, 0 extended, 4 free)
   e   extended
Select (default p): p
Partition number (1-4, default 1):
First sector (2048-2097151, default 2048):
Using default value 2048
Last sector, +sectors or +size{K,M,G} (2048-2097151, default 2097151):
Using default value 2097151
Partition 1 of type Linux and of size 1023 MiB is set

Command (m for help): t
Selected partition 1
Hex code (type L to list all codes): fd
Changed type of partition 'Linux' to 'Linux raid autodetect'

Command (m for help): wq
The partition table has been altered!

Calling ioctl() to re-read partition table.
Syncing disks.
[root@centos7 ~]# partprobe

 

  • raid 에 disk 추가 
[root@centos7 ~]# cat /proc/mdstat
Personalities : [raid1]
md2 : active raid1 sde1[1] sdd1[0]
      1046528 blocks super 1.2 [2/2] [UU]

md1 : active raid1 sdf1[2] sdb1[0]
      1046528 blocks super 1.2 [2/1] [U_]
      [===============>.....]  recovery = 76.4% (800384/1046528) finish=0.0min speed=200096K/sec

unused devices: <none>
[root@centos7 ~]#

 

  • /dev/md1 에서 sdb1 Disk 에 장애상황을 만듭니다. 
[root@centos7 ~]# mdadm /dev/md1 -f /dev/sdb1
mdadm: set /dev/sdb1 faulty in /dev/md1
[root@centos7 ~]# cat /proc/mdstat
Personalities : [raid1]
md2 : active raid1 sde1[1] sdd1[0]
      1046528 blocks super 1.2 [2/2] [UU]

md1 : active raid1 sdf1[2] sdb1[0](F)
      1046528 blocks super 1.2 [2/1] [_U]

unused devices: <none>
[root@centos7 ~]#

 

  • mount 상태 확인 및 touch 로 빈파일을 생성해 봅니다. 
[root@centos7 ~]# df -h
Filesystem               Size  Used Avail Use% Mounted on
/dev/sda3                 18G  1.4G   17G   8% /
devtmpfs                 903M     0  903M   0% /dev
tmpfs                    912M     0  912M   0% /dev/shm
tmpfs                    912M  8.7M  904M   1% /run
tmpfs                    912M     0  912M   0% /sys/fs/cgroup
/dev/sda1               1014M  135M  880M  14% /boot
/dev/mapper/vg00-data01  2.0G   33M  2.0G   2% /data      <--- 정상적으로 마운트 되어 있습니다.
tmpfs                    183M     0  183M   0% /run/user/0
[root@centos7 ~]# touch /data/20

[root@centos7 ~]# touch /data/30
[root@centos7 ~]# touch /data/40

 

  • /dev/md1 에서 sdb1 디스크를 제거 합니다. 
[root@centos7 ~]# mdadm /dev/md1 --remove /dev/sdb1
mdadm: hot removed /dev/sdb1 from /dev/md1
[root@centos7 ~]# cat /proc/mdstat
Personalities : [raid1]
md2 : active raid1 sde1[1] sdd1[0]
      1046528 blocks super 1.2 [2/2] [UU]

md1 : active raid1 sdf1[2]
      1046528 blocks super 1.2 [2/1] [_U]

unused devices: <none>
[root@centos7 ~]#

 

  • 시스템 종료후 disk 를 추가 하여 /dev/md1 을 복구 합니다. 
[root@centos7 ~]# init 0
[root@centos7 ~]# lsblk
NAME              MAJ:MIN RM  SIZE RO TYPE  MOUNTPOINT
sda                 8:0    0   20G  0 disk
├─sda1              8:1    0    1G  0 part  /boot
├─sda2              8:2    0    1G  0 part  [SWAP]
└─sda3              8:3    0   18G  0 part  /
sdb                 8:16   0    1G  0 disk
└─sdb1              8:17   0 1023M  0 part
sdc                 8:32   0    1G  0 disk
└─sdc1              8:33   0 1023M  0 part
sdd                 8:48   0    1G  0 disk
└─sdd1              8:49   0 1023M  0 part
  └─md2             9:2    0 1022M  0 raid1
    └─vg00-data01 253:0    0    2G  0 lvm   /data
sde                 8:64   0    1G  0 disk
└─sde1              8:65   0 1023M  0 part
  └─md2             9:2    0 1022M  0 raid1
    └─vg00-data01 253:0    0    2G  0 lvm   /data
sdf                 8:80   0    1G  0 disk
└─sdf1              8:81   0 1023M  0 part
  └─md1             9:1    0 1022M  0 raid1
    └─vg00-data01 253:0    0    2G  0 lvm   /data
sdg                 8:96   0    1G  0 disk       <--- 추가된 Disk 
[root@centos7 ~]#

fdisk 부분은 생략 합니다. 


[root@centos7 ~]# cat /proc/mdstat
Personalities : [raid1]
md2 : active raid1 sdd1[0] sde1[1]
      1046528 blocks super 1.2 [2/2] [UU]

md1 : active raid1 sdg1[3] sdf1[2]
      1046528 blocks super 1.2 [2/1] [_U]
      [=======>.............]  recovery = 38.2% (401280/1046528) finish=0.0min speed=200640K/sec

unused devices: <none>
[root@centos7 ~]#

 

  • raid 복구후 테스트 
[root@centos7 ~]# cat /proc/mdstat
Personalities : [raid1]
md2 : active raid1 sdd1[0] sde1[1]
      1046528 blocks super 1.2 [2/2] [UU]

md1 : active raid1 sdg1[3] sdf1[2]
      1046528 blocks super 1.2 [2/2] [UU]

unused devices: <none>
[root@centos7 ~]# df -h
Filesystem               Size  Used Avail Use% Mounted on
/dev/sda3                 18G  1.4G   17G   8% /
devtmpfs                 903M     0  903M   0% /dev
tmpfs                    912M     0  912M   0% /dev/shm
tmpfs                    912M  8.7M  904M   1% /run
tmpfs                    912M     0  912M   0% /sys/fs/cgroup
/dev/sda1               1014M  135M  880M  14% /boot
/dev/mapper/vg00-data01  2.0G   33M  2.0G   2% /data
tmpfs                    183M     0  183M   0% /run/user/0
[root@centos7 ~]# cp /tmp/* /data/
cp: omitting directory ‘/tmp/systemd-private-f539feae748d4cae9178d03dceab0233-vgauthd.service-gTNVZW’
cp: omitting directory ‘/tmp/systemd-private-f539feae748d4cae9178d03dceab0233-vmtoolsd.service-ISwWEL’
[root@centos7 ~]# ls -al /data/
total 4
drwxr-xr-x   2 root root 142 Jul 18 00:54 .
dr-xr-xr-x. 18 root root 256 Jul 18 00:33 ..
-rw-r--r--   1 root root   0 Jul 18 00:41 0
-rw-r--r--   1 root root   0 Jul 18 00:41 1
-rw-r--r--   1 root root   0 Jul 18 00:48 10
-rw-r--r--   1 root root   0 Jul 18 00:50 11
-rw-r--r--   1 root root   0 Jul 18 00:50 12
-rw-r--r--   1 root root   0 Jul 18 00:50 13
-rw-r--r--   1 root root   0 Jul 18 00:41 2
-rw-r--r--   1 root root   0 Jul 18 00:49 20
-rw-r--r--   1 root root   0 Jul 18 00:49 30
-rw-r--r--   1 root root   0 Jul 18 00:49 40
-rwx------   1 root root 836 Jul 18 00:54 ks-script-r7n_b9
-rw-------   1 root root   0 Jul 18 00:54 yum.log
[root@centos7 ~]#

 

댓글 남기기