Automating LVM using Python Script and integrating it with Hadoop

To create an automated script for LVM, let us first understand what is LVM, and how it is useful to us.

What is LVM?

LVM is a tool for logical volume management which includes allocating disks, striping, mirroring, and resizing logical volumes. With LVM, a hard drive or set of hard drives is allocated to one or more physical volumes. LVM physical volumes can be placed on other block devices which might span two or more disks. In Linux, Logical Volume Manager (LVM) is a device mapper framework that provides logical volume management for the Linux kernel. Most modern Linux distributions are LVM-aware to the point of being able to have their root file systems on a logical volume.

STEP1: First we need to attach an external hard drive to our VM.

By selecting the VM and then clicking on the Settings option from the bar we can go to the settings and then select the Storage option.

STEP2: Then select the Controller: SATA option and then click on the create icon on its right. Then the following window appears. Now select Create New and then create a Virtual Hard disk of any size.

After creating a new Virtual hard disk you will be able to see an extra file named Right_Arth2.vdi.

This is our queue that our hard disk is now created.

STEP3: Now we need to create its partition by the following commands.

fdisk -l
fdisk /dev/sdc

Here we know that our hard drive name is /dev/sdc. Therefore we created its partition.

Now it's time to run our py script.

STEP4: Run the py script

python3 lvm.py

Now you’ll be able to see the menu.

Enter the choice as per your requirements.

Let me show you some of the options:

Option2: To view the physical volume

Option4: To view the Logical Volume

Similarly, we can create and extend the pre-created LVM simply by giving the name and the size.

Here, I have attached my git URL for your reference.

Integrating LVM with Hadoop:

We can also integrate it with the Hadoop Cluster

Here we can mount the LVM to the data node of the Hadoop cluster providing only the storage of LVM instead of the whole device.

As shown in the ss we can observe that initially, the data node is taking the whole storage of the os.

Before LVM

Now when we create an LVM using the py script mentioned above and then mount it on the data. By this, we have integrated the LVM with the Hadoop cluster and instead of giving the whole memory to the cluster, we can decide the amount of storage that we want to contribute to the cluster.

After LVM

Conclusion:

We can create a dynamic partition by using the LVM concept of RHEL8 by which we can change the size of the partition on the fly. We can also automate this process by creating a python script.

We can also integrate it with Hadoop and decide how much storage we want to contribute to the cluster. The storage size is the size of the partition and can change on the fly.

Hope you find this blog informative.

Thanks for Reading!!