개인적으로 운영하는 ftp 를 lvm 으로 묶어 Storage 형태로 만들어 사용 하고 있습니다.
한 3~4년 주기로 깨지다 보니 그때마다 lvm 복구 작업을 종종 합니다.
Linux Soft Raid 를 이용하여 md1 + md2 를 구성하여 lvm 을 만들어 사용할까 합니다.
Test 환경의 경우 Centos 7 Version 에 1G disk 4개를 추가 하여 sdb + sdc = md1 , sdd + sde = md2 로 구성 하였습니다.
주의: dd 명령으로 복구 테스트하였을 경우 raid 볼륨 자체가 깨지는 문제로 테스트가 되지 않습니다. 🙂
- Disk 정보 (sdb / sdc /sdd /sde 총 4장의 1G Disk 가 장착되어 있습니다.)
[root@centos7 ~]# lsblk NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT sda 8:0 0 20G 0 disk ├─sda1 8:1 0 1G 0 part /boot ├─sda2 8:2 0 1G 0 part [SWAP] └─sda3 8:3 0 18G 0 part / sdb 8:16 0 1G 0 disk sdc 8:32 0 1G 0 disk sdd 8:48 0 1G 0 disk sde 8:64 0 1G 0 disk [root@centos7 ~]#
- mdadm 설치
[root@centos7 ~]# yum install -y mdadm
- raid 장치 확인
[root@centos7 ~]# cat /proc/mdstat Personalities : unused devices: <none> [root@centos7 ~]#
- fdisk 작업 (sdb / sdc /sdd /sde )
[root@centos7 ~]# fdisk /dev/sdb
Welcome to fdisk (util-linux 2.23.2).
Changes will remain in memory only, until you decide to write them.
Be careful before using the write command.
Device does not contain a recognized partition table
Building a new DOS disklabel with disk identifier 0x051d4fae.
Command (m for help): n
Partition type:
p primary (0 primary, 0 extended, 4 free)
e extended
Select (default p): p
Partition number (1-4, default 1): 1
First sector (2048-2097151, default 2048):
Using default value 2048
Last sector, +sectors or +size{K,M,G} (2048-2097151, default 2097151):
Using default value 2097151
Partition 1 of type Linux and of size 1023 MiB is set
Command (m for help): t
Selected partition 1
Hex code (type L to list all codes): fd
Changed type of partition 'Linux' to 'Linux raid autodetect'
Command (m for help): wq
The partition table has been altered!
Calling ioctl() to re-read partition table.
Syncing disks.
[root@centos7 ~]#
~중략
[root@centos7 ~]# partprobe
- /dev/sdb1 , /dev/sdc1 디스크를 이용하여 raid 1 md1 를 구성 합니다.
[root@centos7 ~]# mdadm --create /dev/md1 --level=1 --raid-device=2 /dev/sdb1 /dev/sdc1
mdadm: Note: this array has metadata at the start and
may not be suitable as a boot device. If you plan to
store '/boot' on this device please ensure that
your boot-loader understands md/v1.x metadata, or use
--metadata=0.90
Continue creating array? y
mdadm: Fail create md1 when using /sys/module/md_mod/parameters/new_array
mdadm: Defaulting to version 1.2 metadata
mdadm: array /dev/md1 started.
[root@centos7 ~]#
- /dev/sdd1 , /dev/sde1 디스크를 이용하여 raid 1 md2 를 구성 합니다.
[root@centos7 ~]# mdadm --create /dev/md2 --level=1 --raid-device=2 /dev/sdd1 /dev/sde1
mdadm: Note: this array has metadata at the start and
may not be suitable as a boot device. If you plan to
store '/boot' on this device please ensure that
your boot-loader understands md/v1.x metadata, or use
--metadata=0.90
Continue creating array? y
mdadm: Fail create md2 when using /sys/module/md_mod/parameters/new_array
mdadm: Defaulting to version 1.2 metadata
mdadm: array /dev/md2 started.
[root@centos7 ~]#
- raid 상태 확인
[root@centos7 ~]# cat /proc/mdstat
Personalities : [raid1]
md2 : active raid1 sde1[1] sdd1[0]
1046528 blocks super 1.2 [2/2] [UU]
md1 : active raid1 sdc1[1] sdb1[0]
1046528 blocks super 1.2 [2/2] [UU]
unused devices: <none>
[root@centos7 ~]#
- /etc/mdadm.conf 를 생성합니다.
[root@centos7 ~]# mdadm --detail --scan > /etc/mdadm.conf [root@centos7 ~]# cat /etc/mdadm.conf ARRAY /dev/md1 metadata=1.2 name=centos7:1 UUID=33827f12:f44165a0:19e7a7c9:18f88f40 ARRAY /dev/md2 metadata=1.2 name=centos7:2 UUID=b17e111b:e22dcf62:51b929f6:819a5c2c [root@centos7 ~]#
- lvm2 패키지를 설치 합니다.
[root@centos7 ~]# yum install lvm2 -y
- lvm 생성후 xfs 파일시스템으로 포맷을 합니다.
[root@centos7 ~]# pvcreate /dev/md1 /dev/md2
Physical volume "/dev/md1" successfully created.
Physical volume "/dev/md2" successfully created.
[root@centos7 ~]# vgcreate vg00 /dev/md1 /dev/md2
Volume group "vg00" successfully created
[root@centos7 ~]# lvcreate -l 100%free -n data01 vg00
Logical volume "data01" created.
[root@centos7 ~]#
[root@centos7 ~]# mkfs.xfs /dev/vg00/data01
meta-data=/dev/vg00/data01 isize=512 agcount=4, agsize=130560 blks
= sectsz=512 attr=2, projid32bit=1
= crc=1 finobt=0, sparse=0
data = bsize=4096 blocks=522240, imaxpct=25
= sunit=0 swidth=0 blks
naming =version 2 bsize=4096 ascii-ci=0 ftype=1
log =internal log bsize=4096 blocks=2560, version=2
= sectsz=512 sunit=0 blks, lazy-count=1
realtime =none extsz=4096 blocks=0, rtextents=0
[root@centos7 ~]#
- /etc/fstab 에 마운트 후 시스템 리부팅후 마운트 상태를 확인 합니다.
[root@centos7 ~]# mkdir /data [root@centos7 ~]# vi /etc/fstab /dev/vg00/data01 /data xfs defaults 0 0 [root@centos7 ~]# mount -a [root@centos7 ~]# df -h Filesystem Size Used Avail Use% Mounted on /dev/sda3 18G 1.4G 17G 8% / devtmpfs 903M 0 903M 0% /dev tmpfs 912M 0 912M 0% /dev/shm tmpfs 912M 8.7M 903M 1% /run tmpfs 912M 0 912M 0% /sys/fs/cgroup /dev/sda1 1014M 135M 880M 14% /boot tmpfs 183M 0 183M 0% /run/user/0 /dev/mapper/vg00-data01 2.0G 33M 2.0G 2% /data [root@centos7 ~]# init 6 [root@centos7 ~]# df -h |grep -i data /dev/mapper/vg00-data01 2.0G 33M 2.0G 2% /data [root@centos7 ~]#
- mdadm 명령어를 이용하여 raid 정보를 확인 합니다.
[root@centos7 ~]# mdadm --detail /dev/md1
/dev/md1:
Version : 1.2
Creation Time : Thu Jul 18 00:29:08 2019
Raid Level : raid1
Array Size : 1046528 (1022.00 MiB 1071.64 MB)
Used Dev Size : 1046528 (1022.00 MiB 1071.64 MB)
Raid Devices : 2
Total Devices : 2
Persistence : Superblock is persistent
Update Time : Thu Jul 18 00:34:49 2019
State : clean
Active Devices : 2
Working Devices : 2
Failed Devices : 0
Spare Devices : 0
Consistency Policy : resync
Name : centos7:1 (local to host centos7)
UUID : 33827f12:f44165a0:19e7a7c9:18f88f40
Events : 17
Number Major Minor RaidDevice State
0 8 17 0 active sync /dev/sdb1
1 8 33 1 active sync /dev/sdc1
[root@centos7 ~]# mdadm --detail /dev/md2
/dev/md2:
Version : 1.2
Creation Time : Thu Jul 18 00:30:30 2019
Raid Level : raid1
Array Size : 1046528 (1022.00 MiB 1071.64 MB)
Used Dev Size : 1046528 (1022.00 MiB 1071.64 MB)
Raid Devices : 2
Total Devices : 2
Persistence : Superblock is persistent
Update Time : Thu Jul 18 00:35:22 2019
State : clean
Active Devices : 2
Working Devices : 2
Failed Devices : 0
Spare Devices : 0
Consistency Policy : resync
Name : centos7:2 (local to host centos7)
UUID : b17e111b:e22dcf62:51b929f6:819a5c2c
Events : 17
Number Major Minor RaidDevice State
0 8 49 0 active sync /dev/sdd1
1 8 65 1 active sync /dev/sde1
[root@centos7 ~]#
- mdadm -f 옵션을 이용하여 /dev/md1 에 sdc1 디스크 장애 상황을 만듭니다.
[root@centos7 ~]# mdadm /dev/md1 -f /dev/sdc1
mdadm: set /dev/sdc1 faulty in /dev/md1
[root@centos7 ~]# cat /proc/mdstat
Personalities : [raid1]
md2 : active raid1 sde1[1] sdd1[0]
1046528 blocks super 1.2 [2/2] [UU]
md1 : active raid1 sdc1[1](F) sdb1[0]
1046528 blocks super 1.2 [2/1] [U_] <---- UU 에서 U_ 로 변경 되었습니다.
unused devices: <none>
[root@centos7 ~]#
마운트는 정상적으로 되어 있고 touch 로 빈파일생성도 정상적으로 됩니다.
[root@centos7 ~]# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/sda3 18G 1.4G 17G 8% /
devtmpfs 903M 0 903M 0% /dev
tmpfs 912M 0 912M 0% /dev/shm
tmpfs 912M 8.7M 904M 1% /run
tmpfs 912M 0 912M 0% /sys/fs/cgroup
/dev/sda1 1014M 135M 880M 14% /boot
/dev/mapper/vg00-data01 2.0G 33M 2.0G 2% /data
tmpfs 183M 0 183M 0% /run/user/0
[root@centos7 ~]# touch /data/0
[root@centos7 ~]# touch /data/1
[root@centos7 ~]# touch /data/2
- 디스크 remove 의 경우 mdadm $mdX_name –remove $device_name 으로 디스크를 제거 하시면 됩니다.
[root@centos7 ~]# mdadm /dev/md1 --remove /dev/sdc1
mdadm: hot removed /dev/sdc1 from /dev/md1
[root@centos7 ~]# cat /proc/mdstat
Personalities : [raid1]
md2 : active raid1 sde1[1] sdd1[0]
1046528 blocks super 1.2 [2/2] [UU]
md1 : active raid1 sdb1[0] <--- sdc1 Disk 가 제거된것을 확인할수 있습니다.
1046528 blocks super 1.2 [2/1] [U_]
unused devices: <none>
[root@centos7 ~]#
- 시스템 shutdown 후 Disk 를 추가 합니다.
[root@centos7 ~]# init 0
[root@centos7 ~]# lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
sda 8:0 0 20G 0 disk
├─sda1 8:1 0 1G 0 part /boot
├─sda2 8:2 0 1G 0 part [SWAP]
└─sda3 8:3 0 18G 0 part /
sdb 8:16 0 1G 0 disk
└─sdb1 8:17 0 1023M 0 part
└─md1 9:1 0 1022M 0 raid1
└─vg00-data01 253:0 0 2G 0 lvm /data
sdc 8:32 0 1G 0 disk
└─sdc1 8:33 0 1023M 0 part
sdd 8:48 0 1G 0 disk
└─sdd1 8:49 0 1023M 0 part
└─md2 9:2 0 1022M 0 raid1
└─vg00-data01 253:0 0 2G 0 lvm /data
sde 8:64 0 1G 0 disk
└─sde1 8:65 0 1023M 0 part
└─md2 9:2 0 1022M 0 raid1
└─vg00-data01 253:0 0 2G 0 lvm /data
sdf 8:80 0 1G 0 disk
sr0 11:0 1 1024M 0 rom
[root@centos7 ~]# fdisk /dev/sdf
Welcome to fdisk (util-linux 2.23.2).
Changes will remain in memory only, until you decide to write them.
Be careful before using the write command.
Device does not contain a recognized partition table
Building a new DOS disklabel with disk identifier 0xb67cc56a.
Command (m for help): n
Partition type:
p primary (0 primary, 0 extended, 4 free)
e extended
Select (default p): p
Partition number (1-4, default 1):
First sector (2048-2097151, default 2048):
Using default value 2048
Last sector, +sectors or +size{K,M,G} (2048-2097151, default 2097151):
Using default value 2097151
Partition 1 of type Linux and of size 1023 MiB is set
Command (m for help): t
Selected partition 1
Hex code (type L to list all codes): fd
Changed type of partition 'Linux' to 'Linux raid autodetect'
Command (m for help): wq
The partition table has been altered!
Calling ioctl() to re-read partition table.
Syncing disks.
[root@centos7 ~]# partprobe
- raid 에 disk 추가
[root@centos7 ~]# cat /proc/mdstat
Personalities : [raid1]
md2 : active raid1 sde1[1] sdd1[0]
1046528 blocks super 1.2 [2/2] [UU]
md1 : active raid1 sdf1[2] sdb1[0]
1046528 blocks super 1.2 [2/1] [U_]
[===============>.....] recovery = 76.4% (800384/1046528) finish=0.0min speed=200096K/sec
unused devices: <none>
[root@centos7 ~]#
- /dev/md1 에서 sdb1 Disk 에 장애상황을 만듭니다.
[root@centos7 ~]# mdadm /dev/md1 -f /dev/sdb1
mdadm: set /dev/sdb1 faulty in /dev/md1
[root@centos7 ~]# cat /proc/mdstat
Personalities : [raid1]
md2 : active raid1 sde1[1] sdd1[0]
1046528 blocks super 1.2 [2/2] [UU]
md1 : active raid1 sdf1[2] sdb1[0](F)
1046528 blocks super 1.2 [2/1] [_U]
unused devices: <none>
[root@centos7 ~]#
- mount 상태 확인 및 touch 로 빈파일을 생성해 봅니다.
[root@centos7 ~]# df -h Filesystem Size Used Avail Use% Mounted on /dev/sda3 18G 1.4G 17G 8% / devtmpfs 903M 0 903M 0% /dev tmpfs 912M 0 912M 0% /dev/shm tmpfs 912M 8.7M 904M 1% /run tmpfs 912M 0 912M 0% /sys/fs/cgroup /dev/sda1 1014M 135M 880M 14% /boot /dev/mapper/vg00-data01 2.0G 33M 2.0G 2% /data <--- 정상적으로 마운트 되어 있습니다. tmpfs 183M 0 183M 0% /run/user/0 [root@centos7 ~]# touch /data/20 [root@centos7 ~]# touch /data/30 [root@centos7 ~]# touch /data/40
- /dev/md1 에서 sdb1 디스크를 제거 합니다.
[root@centos7 ~]# mdadm /dev/md1 --remove /dev/sdb1
mdadm: hot removed /dev/sdb1 from /dev/md1
[root@centos7 ~]# cat /proc/mdstat
Personalities : [raid1]
md2 : active raid1 sde1[1] sdd1[0]
1046528 blocks super 1.2 [2/2] [UU]
md1 : active raid1 sdf1[2]
1046528 blocks super 1.2 [2/1] [_U]
unused devices: <none>
[root@centos7 ~]#
- 시스템 종료후 disk 를 추가 하여 /dev/md1 을 복구 합니다.
[root@centos7 ~]# init 0
[root@centos7 ~]# lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
sda 8:0 0 20G 0 disk
├─sda1 8:1 0 1G 0 part /boot
├─sda2 8:2 0 1G 0 part [SWAP]
└─sda3 8:3 0 18G 0 part /
sdb 8:16 0 1G 0 disk
└─sdb1 8:17 0 1023M 0 part
sdc 8:32 0 1G 0 disk
└─sdc1 8:33 0 1023M 0 part
sdd 8:48 0 1G 0 disk
└─sdd1 8:49 0 1023M 0 part
└─md2 9:2 0 1022M 0 raid1
└─vg00-data01 253:0 0 2G 0 lvm /data
sde 8:64 0 1G 0 disk
└─sde1 8:65 0 1023M 0 part
└─md2 9:2 0 1022M 0 raid1
└─vg00-data01 253:0 0 2G 0 lvm /data
sdf 8:80 0 1G 0 disk
└─sdf1 8:81 0 1023M 0 part
└─md1 9:1 0 1022M 0 raid1
└─vg00-data01 253:0 0 2G 0 lvm /data
sdg 8:96 0 1G 0 disk <--- 추가된 Disk
[root@centos7 ~]#
fdisk 부분은 생략 합니다.
[root@centos7 ~]# cat /proc/mdstat
Personalities : [raid1]
md2 : active raid1 sdd1[0] sde1[1]
1046528 blocks super 1.2 [2/2] [UU]
md1 : active raid1 sdg1[3] sdf1[2]
1046528 blocks super 1.2 [2/1] [_U]
[=======>.............] recovery = 38.2% (401280/1046528) finish=0.0min speed=200640K/sec
unused devices: <none>
[root@centos7 ~]#
- raid 복구후 테스트
[root@centos7 ~]# cat /proc/mdstat
Personalities : [raid1]
md2 : active raid1 sdd1[0] sde1[1]
1046528 blocks super 1.2 [2/2] [UU]
md1 : active raid1 sdg1[3] sdf1[2]
1046528 blocks super 1.2 [2/2] [UU]
unused devices: <none>
[root@centos7 ~]# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/sda3 18G 1.4G 17G 8% /
devtmpfs 903M 0 903M 0% /dev
tmpfs 912M 0 912M 0% /dev/shm
tmpfs 912M 8.7M 904M 1% /run
tmpfs 912M 0 912M 0% /sys/fs/cgroup
/dev/sda1 1014M 135M 880M 14% /boot
/dev/mapper/vg00-data01 2.0G 33M 2.0G 2% /data
tmpfs 183M 0 183M 0% /run/user/0
[root@centos7 ~]# cp /tmp/* /data/
cp: omitting directory ‘/tmp/systemd-private-f539feae748d4cae9178d03dceab0233-vgauthd.service-gTNVZW’
cp: omitting directory ‘/tmp/systemd-private-f539feae748d4cae9178d03dceab0233-vmtoolsd.service-ISwWEL’
[root@centos7 ~]# ls -al /data/
total 4
drwxr-xr-x 2 root root 142 Jul 18 00:54 .
dr-xr-xr-x. 18 root root 256 Jul 18 00:33 ..
-rw-r--r-- 1 root root 0 Jul 18 00:41 0
-rw-r--r-- 1 root root 0 Jul 18 00:41 1
-rw-r--r-- 1 root root 0 Jul 18 00:48 10
-rw-r--r-- 1 root root 0 Jul 18 00:50 11
-rw-r--r-- 1 root root 0 Jul 18 00:50 12
-rw-r--r-- 1 root root 0 Jul 18 00:50 13
-rw-r--r-- 1 root root 0 Jul 18 00:41 2
-rw-r--r-- 1 root root 0 Jul 18 00:49 20
-rw-r--r-- 1 root root 0 Jul 18 00:49 30
-rw-r--r-- 1 root root 0 Jul 18 00:49 40
-rwx------ 1 root root 836 Jul 18 00:54 ks-script-r7n_b9
-rw------- 1 root root 0 Jul 18 00:54 yum.log
[root@centos7 ~]#
댓글을 달려면 로그인해야 합니다.