Home Linux File Server with Software RAID and iSCSI (9/10)

Continuation from Home Linux File Server

Challenges 9: Reinstall Core Operating system, configure it up, and mount and use /md.

The scenario is simple: We didn’t mirror our boot partition, so now we have to reinstall the operating system and make sure we can preserve everything we need to keep business consistency.

BACKUP SAYS THE LION!!!

Before addressing the “everything is not working” scenario, we need to backup 2 items:

/etc/mdadm.conf: This is where our /dev/md configurations are stored.

[root@eye-scrunchie ~]# ll /etc/mdadm.conf
-rw-r--r-- 1 root root 92 Jan 23 15:35 /etc/mdadm.conf
[root@eye-scrunchie ~]# cat /etc/mdadm.conf
ARRAY /dev/md/0  metadata=1.2 UUID=e47e9e3a:8b2d2d70:430fa6dc:babf2503 name=eye-scrunchie:0

/etc/tgt/targets.conf: This is where our iSCSI configuration is.

[root@eye-scrunchie ~]# ll /etc/tgt/targets.conf
-rw------- 1 root root 7077 Jan 23 22:57 /etc/tgt/targets.conf
[root@eye-scrunchie ~]# cat /etc/tgt/targets.conf
# This is a sample config file for tgt-admin.
# By default, tgt-admin looks for its config file in /etc/tgt/targets.conf
#
# The "#" symbol disables the processing of a line.


# This one includes other config files:

#include /etc/tgt/temp/*.conf


# Set the driver. If not specified, defaults to "iscsi".
#
# This can be iscsi or iser. To override a specific target set the
# "driver" setting in the target's config.
default-driver iscsi
<target iqn.2018-01.eye-scrunchie:target1>
        backing-store /dev/md0
</target>

#
...to the end...

And a minor items, such as network information in case the motherboard failed out and we’re replacing that as well:

[root@eye-scrunchie ~]# ip addr
1: lo:  mtu 65536 qdisc noqueue state UNKNOWN
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: eth0:  mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 08:00:27:bd:50:65 brd ff:ff:ff:ff:ff:ff
    inet 192.168.1.38/24 brd 192.168.1.255 scope global eth0
    inet6 fe80::a00:27ff:febd:5065/64 scope link
       valid_lft forever preferred_lft forever

Wipe, Rinse, Repeat

In an actual rebuild scenario I’d power-off the whole system and disconnect the drives that were part of the RAID. Then I’d plug in the new drive, insert in my OS installation media, and install the base OS. In this case, it’ll be Centos 6.

After the installation, and reboot, I do the customary update and pre-configuration:

# yum update
# service iptables save
# service iptables stop
# chkconfig iptables off
# cat /etc/selinux/config | sed s/=enforcing/=disabled/ > /etc/selinux/config.new && rm /etc/selinux/config && mv /etc/selinux/config.new /etc/selinux/config

And i’ll need to install the iSCSI packages as well:

# yum install scsi-target-utils
# service tgtd start
# chkconfig tgtd on

Configurations

the mdadm configuration file:

# touch /etc/mdadm.conf
# vi /etc/mdadm.conf
ARRAY /dev/md/0  metadata=1.2 UUID=e47e9e3a:8b2d2d70:430fa6dc:babf2503 name=eye-scrunchie:0

and the targets.conf file:

# vi /etc/tgt/targets.conf
...adding the following below "default-driver iscsi"
<target iqn.2018-01.eye-scrunchie:target1>
        backing-store /dev/md0
</target>

Get the drives hooked up

At this point we should have all the software necessary to get the drives up and running, and the configuration necessary to make them all run. I shut down the VM and “hook up the drives” in the order that they should be on the SATA controller. This includes the 3 “good” drives and the 1 “bad” 64mb drive. I also didn’t hook them up with the Hot-Swap flag that I used in a previous write-up.

Once they’re all connected, I turn on the VM and wait for boot-up.

# mdadm --misc --detail /dev/md0
/dev/md0:
        Version : 1.2
  Creation Time : Mon Jan 22 22:35:54 2018
     Raid Level : raid5
     Array Size : 520192 (508.00 MiB 532.68 MB)
  Used Dev Size : 260096 (254.00 MiB 266.34 MB)
   Raid Devices : 3
  Total Devices : 2
    Persistence : Superblock is persistent

    Update Time : Thu Jan 25 11:38:47 2018
          State : clean, degraded
 Active Devices : 2
Working Devices : 2
 Failed Devices : 0
  Spare Devices : 0

         Layout : left-symmetric
     Chunk Size : 512K

           Name : eye-scrunchie:0  (local to host eye-scrunchie)
           UUID : e47e9e3a:8b2d2d70:430fa6dc:babf2503
         Events : 72

    Number   Major   Minor   RaidDevice State
       0       8       17        0      active sync   /dev/sdb1
       4       8       65        1      active sync   /dev/sde1
       4       0        0        4      removed

It seems to have found 2/3 of the drives. I do have access to my iSCSI volume, as my OS has auto-connected back to it. I also see files on it as well. Lets see what the system sees, and maybe we can fix this.

# lsblk
NAME                               MAJ:MIN RM  SIZE RO TYPE  MOUNTPOINT
sr0                                 11:0    1 1024M  0 rom
sda                                  8:0    0    3G  0 disk
├─sda1                               8:1    0  500M  0 part  /boot
└─sda2                               8:2    0  2.5G  0 part
  ├─vg_eyescrunchie-lv_root (dm-0) 253:0    0  2.2G  0 lvm   /
  └─vg_eyescrunchie-lv_swap (dm-1) 253:1    0  304M  0 lvm   [SWAP]
sdb                                  8:16   0  256M  0 disk
└─sdb1                               8:17   0  255M  0 part
  └─md0                              9:0    0  508M  0 raid5
    └─md0p1                        259:0    0  505M  0 md
sdc                                  8:32   0  256M  0 disk
└─sdc1                               8:33   0  255M  0 part
sdd                                  8:48   0   64M  0 disk
sde                                  8:64   0  256M  0 disk
└─sde1                               8:65   0  255M  0 part
  └─md0                              9:0    0  508M  0 raid5
    └─md0p1                        259:0    0  505M  0 md

It appears as though /dev/sdc1 is not a part of this, when it should have been. I’m going to add it into the array and hopefully it picks it all up.

# mdadm --add /dev/md0 /dev/sdc1
mdadm: added /dev/sdc1
# mdadm --misc --detail /dev/md0
/dev/md0:
        Version : 1.2
  Creation Time : Mon Jan 22 22:35:54 2018
     Raid Level : raid5
     Array Size : 520192 (508.00 MiB 532.68 MB)
  Used Dev Size : 260096 (254.00 MiB 266.34 MB)
   Raid Devices : 3
  Total Devices : 3
    Persistence : Superblock is persistent

    Update Time : Thu Jan 25 11:44:42 2018
          State : clean, degraded, recovering
 Active Devices : 2
Working Devices : 3
 Failed Devices : 0
  Spare Devices : 1

         Layout : left-symmetric
     Chunk Size : 512K

 Rebuild Status : 51% complete

           Name : eye-scrunchie:0  (local to host eye-scrunchie)
           UUID : e47e9e3a:8b2d2d70:430fa6dc:babf2503
         Events : 82

    Number   Major   Minor   RaidDevice State
       0       8       17        0      active sync   /dev/sdb1
       4       8       65        1      active sync   /dev/sde1
       3       8       33        2      spare rebuilding   /dev/sdc1

Perfect! On complete rebuild, we are HEALTHY!

# mdadm --misc --detail /dev/md0
/dev/md0:
        Version : 1.2
  Creation Time : Mon Jan 22 22:35:54 2018
     Raid Level : raid5
     Array Size : 520192 (508.00 MiB 532.68 MB)
  Used Dev Size : 260096 (254.00 MiB 266.34 MB)
   Raid Devices : 3
  Total Devices : 3
    Persistence : Superblock is persistent

    Update Time : Thu Jan 25 11:44:45 2018
          State : clean
 Active Devices : 3
Working Devices : 3
 Failed Devices : 0
  Spare Devices : 0

         Layout : left-symmetric
     Chunk Size : 512K

           Name : eye-scrunchie:0  (local to host eye-scrunchie)
           UUID : e47e9e3a:8b2d2d70:430fa6dc:babf2503
         Events : 91

    Number   Major   Minor   RaidDevice State
       0       8       17        0      active sync   /dev/sdb1
       4       8       65        1      active sync   /dev/sde1
       3       8       33        2      active sync   /dev/sdc1

And just to make sure we’re still good, i’ll reboot the system and check the RAID again and i’m still good to go!