Continuation from Home Linux File Server
Challenges 6, 7, and 8: Break the RAID, again, Verify that we can still create/make files, and Rebuild the RAID with the Spare.
This is the crux of all of this. We need to be able to know that when a physical drive/volume fails that iSCSI won’t drop anything, that the RAID5 will still operate seamlessly in degraded mode, and that we can have the spare from our “Previously RMA’d drive” scenario take the place of the dead drive.
My current array, as it awesomely is:
# mdadm --misc --detail /dev/md0
/dev/md0:
Version : 1.2
Creation Time : Mon Jan 22 22:35:54 2018
Raid Level : raid5
Array Size : 520192 (508.00 MiB 532.68 MB)
Used Dev Size : 260096 (254.00 MiB 266.34 MB)
Raid Devices : 3
Total Devices : 4
Persistence : Superblock is persistent
Update Time : Wed Jan 24 17:07:13 2018
State : clean
Active Devices : 3
Working Devices : 4
Failed Devices : 0
Spare Devices : 1
Layout : left-symmetric
Chunk Size : 512K
Name : eye-scrunchie:0 (local to host eye-scrunchie)
UUID : e47e9e3a:8b2d2d70:430fa6dc:babf2503
Events : 42
Number Major Minor RaidDevice State
0 8 17 0 active sync /dev/sdb1
4 8 65 1 active sync /dev/sde1
3 8 49 2 active sync /dev/sdd1
5 8 33 - spare /dev/sdc1
So, in order to create a faulty scenario that was separate than the faulty scenario in the initial steps I decided to convert the drives in Virtuabox to Hot-Swap drives. I had to shut down my VM, change the drives to Hot-Swap via a checkbox, then fire back up the VM.
To break the array I created a new 64MB drive and Hot-Swapped the middle drive with it. This both invalidated the array and made it impossible to fully rebuild. The controller masterfully dropped the drive from the array, grabbed the spare, and began an immediate rebuild.
/dev/md0:
Version : 1.2
Creation Time : Mon Jan 22 22:35:54 2018
Raid Level : raid5
Array Size : 520192 (508.00 MiB 532.68 MB)
Used Dev Size : 260096 (254.00 MiB 266.34 MB)
Raid Devices : 3
Total Devices : 3
Persistence : Superblock is persistent
Update Time : Wed Jan 24 17:15:38 2018
State : clean, degraded, recovering
Active Devices : 2
Working Devices : 3
Failed Devices : 0
Spare Devices : 1
Layout : left-symmetric
Chunk Size : 512K
Rebuild Status : 82% complete
Name : eye-scrunchie:0 (local to host eye-scrunchie)
UUID : e47e9e3a:8b2d2d70:430fa6dc:babf2503
Events : 58
Number Major Minor RaidDevice State
0 8 17 0 active sync /dev/sdb1
4 8 65 1 active sync /dev/sde1
5 8 33 2 spare rebuilding /dev/sdc1
From the Break -> Rebuild, I’m sure I could’ve written a file or two but the drives are so small and the failure/recovery was very quick. I do not have any specific events on my iSCSI Initiator that my volume dropped/timed out/failed write, or anythign bad.
I’m pretty happy with this scenario.