|Linux 04 March 2014 -|
Lately, I ran into the situation that one of my services which is hosted in a virtual machine stopped working. A quick check revealed that the hard disk ran out of disk space and it was about time to increase the available storage. Don't laugh but the system is running on CentOS 5.x with a mere 13 GB virtual disk - well, since years already. And one golden rule learned from experience: Never touch a running system - kept me away from any modifications. Well, there's always a time that change has more benefits than not touching the system...
That is (now: was) the situation before the following steps.
And I can tell you that even a Linux OS doesn't like the lack of space on the system drive at all.
Although, I think it is not necessary to stress the fact that you should (read: have to) backup your data and drives always. Just to be on the safe side, and to revert back to the original state easily and quickly.
First, shut down your virtual machine. Even though it is possible to run almost all of the following in a running environment I doubt that for the casual (home) administrator reading this article it might not be necessary to risk your data. Okay next, it would be necessary to expand the size of virtual hard disk, and then afterwards extend the logical volume with the newly created partition.
Converting the disk format from VMDK to VDI
Well, the system has kind of a past and it was originally created on VMware Server 1.0, then upgraded to VMware Server 2.0, and some years back I switched over to SUN, eh Oracle, VirtualBox. So, the original virtual disk was still in VMware's format - vmdk. VirtualBox is not able to expand that type of disk drive (yet):
In order to complete this initial task it was necessary to convert the format from VMDK to VDI. This is done by cloning the disk into the new format:
Depending on the actual disk size and your system performance this might take some time. Remain patient and let the system do the job. Meanwhile, you might check your mails or post your intentions on various social media networks. Just kidding! ;-)
Note: Additionally to your backup, it might be interesting to keep an archive of the original VMDK file. Just in case...
Resizing the virtual drive
Now, that we have our drive in VDI format we are able to expand the disk size. Let's try our previous statement again, but this time with the newly created VDI disk:
After successful expansion it is about time to check our disk modifications in the settings of the virtual machine. Launch the Oracle VM VirtualBox Manager, select your virtual machine and either press the Settings button in the toolbar or choose from the menu Machine -- Settings (Ctrl+S). Next, select the Storage entry from the side pane and then the virtual hard disk that we just modified. You should see something similar to the following screenshot.
In case that you had to convert an VMDK drive to VDI, please select the newly cloned drive from the dropdown list in the Attributes area. You should have the .VDI disk attached to your virtual machine.
Pro tip: In case that you want to keep the original disk type in VMDK format, do a complete round-trip and clone the extended drive back to VMDK format, like so:
Using VMware Player
The above mentioned conversion is not necessary in case that you have an installation of VMware Player or even VMware Workstation at hand. In my case, I didn't. Anyway, VMware Player gives you the ability to expand the disk capacity through the UI directly. Open the Virtual Machine Settings, then select the Hard Disk from the list of devices and below the Disk information you'll have a list of utilities in the dropdown list. Choose "Expand..." in order to change the disk's capacity.
That's all for the 'physical' expansion of our hard drive. Now, start your virtual machine as we are going to take of the software part.
The system in our virtual machine is still out of space and we are going to change this now. Please follow the next steps closely. Accidentally, I trapped myself and had to revert a couple of steps because I didn't pay enough attention to the details. I'll come back to that at the end of this article. Okay, we enlarged the physical drive, and now we have to take care of that unallocated space. Let's check the partition table first like so:
This will give us an overview of the existing partitions. The new disk size has been accepted already and it's now time to create a new partition.
Creating a new partition
This article is not about how to use fdisk on the command line. There are other tutorials with more detailed information about the particular steps, and you might have a look at the manual page of fdisk directly. To create a new partition we launch fdisk on our hard drive
Then we create a new full-size partition at the end of the disk drive with the following sequence of keystrokes:
Of course, this might vary depending on your existing partition table. Please, adapt your inputs accordingly.
After completing all steps your hard drive might have a similar output like this one:
Alternative: Extend an existing partition
During my research in preparation of this 'surgery' I also came across a couple of resources online where it is stated that it is also possible to enlarge an existing partition to the full extend of the new capacity of the disk drive. The general recommendation is to use a more sophisticated partitioning tool like the freely available GParted. While using a Windows operating system, I clearly would recommend this approach, too. Mainly because it's your system drive that runs out of space and Windows is hardly capable to span over multiple drives (at least in casual environments like home and office desktops). I already did this in the past to enlarge a Windows Server 2008 R2 system, and it went as smooth as expected.
You can download the GParted LiveCD ISO here.
Once you understood the necessary steps to extend an existing logical volume it's fairly easy to reproduce. The general concept is well documented in Chapter 4 of the LVM Administrator's Guide for CentOS, and is based on the following procedure. First, you create a physical volume (PV) based on your new partition which is then extended into an existing volume group (VG), and then finally extended into a logical volume (LV) within a volume group. After resizing the logical volume you are ready to go and the new disk space is available in your system.
Creating the physical volume
This will prepare the partition for use in a volume group.
Note: Depending on whether you are going to extend your system with an additional partition or a whole drive, you might even skip the partitioning and create a physical volume using the whole hard disk.
Extending the volume group
Prior to any extension is not too bad to get an overview of the existing volume groups. This can be done like so:
With the information of the available name of the volume group, we are now able to extend it with our physical partition:
Done. Next step is to assign the available space to the logical volume that we would like to expand.
Extending the logical volume, finally
Same procedure here, it's advised to have an overview of the existing logical volume first:
In my case, I'm going to extend the first logical volume with the new physical volume.
Almost done. After expanding the volume group and the logical volume we only have to populate the information about the new total size of our logical volume.
Okay, my system has now more than triple the original disk size. Let's see how many years it is going to run before I might have to add another partition to the logical volume.
Once you understood the necessary steps on how to expand a logical volume it's actually very simple to reproduce. You start with the physical preparation of the hard disk, either by adding a new drive or by expanding an existing virtual drive. Then you create a partition of type 8e - Linux LVM - for that new unallocated drive space. And to complete the procedure you have to walk through the LVM handling by creating the physical volume, extending the volume group and finally extending and resizing the logical volume in that volume group.
Even in case of an error it's relatively simple to track down the root cause and take care of it.
As mentioned earlier I got stuck in the process because of two issues.
First, my Linux operating system, here CentOS 5.3, didn't get the new partition table automatically. So when I run the following command I got the response that the specified device isn't available.
A quick look on the internet revealed that this could happen and the modified partition table can be updated manually like so:
Or to keep things simple, just reboot the system. But that would be too easy and a waste of time...
And second, I mistyped the name of the logical volume - LogVol01 instead of LogVol00 - and added the new partition to the swap area. Well, following are some details on how to detect that kind of issue and how to remove a physical drive from a logical volume and volume group.
I came to an abrupt stop on the very last step, resizing the file system with the newly allocated volume:
That's kind of bummer. Especially, so close to the finish line. Well, it turned out that volume group had two logical volumes LogVol00 and LogVol01. And the second one is actually used for the swap area. Which easily explains why there is absolutely no valid filesystem superblock to be found - no matter how hard I would try it. First, I thought that it might have been a problem with the filesystem type, as I am using ext2, ext3, ext4, and xfs interchangeable. But a quick check confirmed that I'm using the right command. In case of xfs you might have to work with xfs_grow command instead of resize2fs.
Then I checked the overview output of my logical volumes again and I was kind of lucky that I discovered that I increased the wrong volume:
The original output stated a LV Size of 1.91 GB. Due to the nature of LVM I couldn't remove the physical hard drive because it is reported to be in use. Therefore, it is first necessary to decrease the volume size to its original value like so:
Eventually, you might have to move all allocated space off the physical volume to other free physical volumes in the same volume group like so:
Then it's possible to remove (or better said to reduce) the drive from the volume group like so:
And then to start over again by adding the fresh physical volume back to the volume group again and then assigning it to the proper logical volume, like so:
Now, everything went smooth and my last system check of the logical volumes looks like so:
As usual, follow administrator's rule number 1: Don't Panic!
Have a look at the history of your inputs, here your commands on the console, and check them entirely with the outputs of the various display commands - pvdisplay to show information about physical volumes, vgdisplay for details about volume groups, and finally lvdisplay to get an overview of your logical volumes.
Good luck with your mission and leave your comments below.