Monday 30 July 2018

Issues booting to CentOS7 after resizing its root/swap LV

I recently faced the following strange issue...

Some background first

I installed  CentOS7 on a VM, its assigned vhd size was 10G, nothing fancy so far ...
Installation was successful and I was able to boot into the O/S without issues.

Later, at some point, I decided to increase vhd size from 10G to 13GB. To accomplish that I followed the steps below...

- Turned off the VM
- Resized the vhd to the new size
- Turned on the VM
- Booted from LiveCD [CentOS7 dvd]

  •     lsblk    

    [root@localhost ~]# lsblk
NAME            MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
sda               8:0    0   13G  0 disk
├─sda1            8:1    0  500M  0 part /boot
└─sda2            8:2    0 12.5G  0 part
  ├─centos-root 253:0    0 11.5G  0 lvm  /
  └─centos-swap 253:1    0    1G  0 lvm  [SWAP]  
     

  •        fdisk /dev/sda

    Here, I deleted the second partition (/dev/sda2), which is the LVM partition where centos/root and centos/swap LVs reside on. Then, I recreated it, by making sure that start sector was the same as before deleting the partition. Obviously, the end sector will be different as we increased vhd size. Finally, set partition type to 8e (LVM), saved settings and exited. By the way this is an 'extended' partition type.

  •        pvresize /dev/sda2        


  •        lvresize -l +100%FREE centos/root

     
       [root@localhost ~]# lvdisplay centos/root
  --- Logical volume ---
  LV Path                /dev/centos/root
  LV Name                root
  VG Name                centos
  LV UUID                1rlD1f-BSw4-R3rl-z22I-GtDN-rykj-cxdlXL
  LV Write Access        read/write
  LV Creation host, time localhost.localdomain, 2018-07-27 23:06:39 +0100
  LV Status              available
  # open                 1
  LV Size                <11 .52="" font="" gib="">
  Current LE             2948
  Segments               1
  Allocation             inherit
  Read ahead sectors     auto
  - currently set to     8192
  Block device           253:0

     
So far, so good...centos/root LV , which is the backing device for root file system (/), has its size increased.


  •      mkdir /mnt/root   #create a temp directory to mount centos/root LV
  •      mount /dev/mapper/centos-root /mnt/root   #mount centos/root to /mnt/root
  •      xfs_growfs  /mnt/root   #increase root fs filesystem to match centos/root LV size. Did I mention that root fs is XFS ?
  •      umount /mnt/root   #unmount root fs 
  •      reboot   #reboot the system, and boot from hdd this time [not LiveCD]


The issue

After rebooting from LiveCD to the normal hard drive (now 13G), the following happened ...

     - GRUB menu appeared and loaded default kernel (3.10.0-862.9.1.el7.x86_64).
     - kernel loads and initial boot sequence starts.
     - kernel hands over to initramfs (initrd).
     - initrd fails to detect/active centos/root and centos/swap LVs, so after a while I'm being dropped to dracut shell...
      In dracut shell, I execute the following commands:
           - lvm lvchange -ay centos/root   #success
           - lvm lvchange -ay centos/swap  #success
           - ln -s /dev/mapper/centos-root /dev/root   #required for boot process to continue to the real root fs (/).
           - exit  #exit dracut shell

At this point boot process continues and finally I can login to the normal CentOS7 installation. But the question remains, why I have to do all these stuff? wasn't this supposed to happen automatically ? The answer is, yes, it should,  ....but something went wrong (obviously)... and now it needs my manual intervention to succeed.

The solution

After 5 days of continuous search, finally a solution was found. The problem was not in initramfs, but on the fact that /dev/sda2 partition(PV),  after I resized it, somehow got 2  partition table signatures.
One was set as a "dos" partition with an offset "0x1fe" and second was set as "LVM_member" with an offset "0x8e".
This confused blkid in initramfs, during the initial boot stage, thinking that /dev/sda2 is *not* an LVM_member but rather a simple "dos" partition, hence refused to activate centos/root and centos/swap LVs which were required to boot the machine to the O/S.
What's interesting is that neither fdisk or parted were showing this second signature, however there was a trace of it in the dracut report file (/run/initramfs/rdsosreport.txt), specifically the one below..

+ blkid
/dev/sr0: UUID="2016-10-28-12-18-36-00" LABEL="CentOS 7 x86_64" TYPE="iso9660" PTTYPE="dos" 
/dev/sda1: UUID="a620a180-3a8c-4b5f-ad30-804f131a7261" TYPE="xfs" 
/dev/sda2: PTTYPE="dos" 
+ blkid -o udev
ID_FS_UUID=2016-10-28-12-18-36-00
ID_FS_UUID_ENC=2016-10-28-12-18-36-00
ID_FS_LABEL=CentOS_7_x86_64
ID_FS_LABEL_ENC=CentOS\x207\x20x86_64
ID_FS_TYPE=iso9660
ID_PART_TABLE_TYPE=dos
To fix this problem, I booted the system from a livecd and used "wipefs" utility to erase the problematic signature "dos" and leave only the correct one "LVM_member".


The exact command was: 'wipefs -o 0x1fe -ff /dev/sda2' 

Then a reboot and voila system booted without issues ...



No comments:

Post a Comment