GetCpanelSupport Blog

VIRTFS consuming lot of Disk Space- Solution

Published on : July 5, 2019 by admin

The /home/virtfs is a file system for the jailshell shell in cPanel servers. Let me explain what is jail shell first, A jailed shell  is used for restricting the access for the user. In a normal shell most binaries and libraries are available to user. It can cause security issues on your server. The purpose of jailed shell is to provide limited and restrictive environment to the user which is more secure to the server. So, cPanel & WHM uses VirtFS to provide a jailed shell environment for users who connect to a server via SSH. The jailed shell acts as a container for the user, and does not allow the user to access other users' home directories on the server. *  Unlike a normal shell environment, a jailed shell environment increases  security for a server's other users. *  Users in a jailed shell environment can run otherwise-unavailable commands (for example, crontab and passwd).  

Scenario

The user was unable to access all the domains in his VPS server and he was facing issues while accessing cPanel/WHM with his dedicated IP.

How we came to know that the issue was isolated with VIRTFS

We have entered the server via terminal having root access and typed below command.

df -h

But we were not getting any results for the command in terminal. Also, We have checked the /etc/fstab and /etc/mtab which was already empty.

We confirmed it by checking the disk space taken by the /home/virtfs folder. We used du -sch * | sort - h command in the terminal (This command shows the disk usage of each folder in a human readable format) and can see that the virtfs folder eas consuming 90% of allocated disk space. This prompt us to think that the issue was isolated with VIRTFS.

What should do if the virtfs directory is full?

Please note that `Do not` use the rm command to remove any mounted file or directory within the /home/virtfs/ directory. If you run the rm command on any mounted file or directory within the /home/virtfs/ directory, you will also delete all of the files in the directory to which it is mounted. This action will render your server nonfunctional. All the files have the same inode number.  So If you are attempting to delete the files under virtfs directory directly then the actual file also will be removed from the server which may cause issues.The user can (only) access the data under these file-systems. The disk usage for the directory will be shown as high but it will not use any disk space on the server because it is a virtual mount point.If you delete any file in this directory it will delete the actual file which is linked to.

When a user logs in to a jailed shell environment via SSH or SFTP for the first time, the system creates the /home/virtfs/cpaneluser. This directory contains configuration files, utilities, and BIND mounts.

In order to resolve the issue with the Virtfs.

First we need to check the jailed shell feature is enabeld in the server or not?

For this, please login to your WHM, click on Server Configuration -> Tweak Settings -> Use cPanel® jailshell by default

1. Use cPanel® jailshell by default >>ON >> Means it's enabled.

2. Use cPanel® jailshell by default >> OFF >> Means it's disabled.

Most cases the option is in OFF postion but still the virtual directory is present in the location "/home/virtfs/user".

It's happens due to previously some users uses jailed shell environment. We need to check the which are domains are currently using the jailed shell.

Access WHM >> Manage shell access >> Check the shell for all the users. Most users are using Normal shell. We will get the actual jailed users. We need to compare it with the users information in the server.

Analyzing the issue from the Linux Terminal.

1.  Run the following command to get the list of jail shell users.

-----------------------------------------
grep username /etc/passwd
-----------------------------------------

If you see something like this: username:x:733:733::/home/username:/usr/local/cpanel/bin/jailshell

Then it’s still enabled.  Otherwise, it’ll look something like this:

  • username:x:505:502::/home/username:/bin/bash

  • OR

  • username:x:2137:2131::/home/username:/usr/local/cpanel/bin/noshell

2. If we find some extra users that they are using the jailed shell. We need to changed the shell access by running the following command.

 
----------------------------------------------
usermod -s /bin/bash username
---------------------------------------------

3. Now you must check if the user has JailShell defined in its crontab.

----------------------------------
crontab -l -u username
----------------------------------

4. If you notice SHELL="/usr/local/cpanel/bin/jailshell" in the output, you'll need to edit user's crontab and delete that line. To access user's crontab while logged in as root, use:

----------------------------------
crontab -e -u username
---------------------------------

5. Now you are certain that the user has new default shell and that his cron jobs won't be executed in JailShell.

 

Before going any further, you should check if the user has been logged in the whole time and it's still logged in in JailShell.

Please use the follwong command to get the details about the current users.

-------
w
-------

6. Check if there’s any jailshell process running and if so, kill the process. If none, you may run the following bash one liner in order to fix this issue:

Remove all user's jailed shell environment

---------------------------------------------------------------------------------------------------
for i in `cat /proc/mounts|awk '/virtfs/ {print $2}'`; do umount $i;done
---------------------------------------------------------------------------------------------------

Or else

To unmount the VirtFS BIND mounts, run the following command, where username is the desired account username:

---------------------------------------------------------
umount /home/virtfs/username/usr/bin
---------------------------------------------------------

This script removes the /home/virtfs/username/ directory and its contents, where username is an affected account's username.

To force the removal of all VirtFS mount points, run the following command:

--------------------------------------------------------------------
/scripts/clear_orphaned_virtfs_mounts --clearall
--------------------------------------------------------------------

To check your system for VirtFS mount points, run the following command, where username is the desired account username:

---------------------------------------------
grep -i username /proc/mounts
---------------------------------------------

After running the command check the disk usage of the virtfs by running the command.

------------------------------
du -sch /home/virtfs/
------------------------------

 

The disk usage should be normal now.

Once the disk usage is normal, Please try to restore the /ect/mtab /etc/fstab if those files are empty.(This is Applicable only if these files are empty)

So , please note that we should not run rm command to fix the disk usage issue with the virtfs. If you run the rm command on any mounted file or directory within the /home/virtfs/ directory, you will also delete all of the files in the directory to which it is mounted. This action will render your server nonfunctional. All the files have the same inode number.  So If you are attempting to delete the files under virtfs directory directly then the actual file also will be removed from the server which may cause issues.The user can (only) access the data under these file-systems. The disk usage for the directory will be shown as high but it will not use any disk space on the server because it is a virtual mount point.If you delete any file in this directory it will delete the actual file which is linked to.