Monitor NFS Mount Stale Handles and Auto-Remount

Hey team, Darian Vance here. Let’s talk about a silent productivity killer: stale NFS mounts. I used to get alerts, SSH into a box, run `df -h`, see the dreaded “Stale file handle” error, and then manually run a remount. It felt like I was wasting at least an hour or two a week on this tedious, reactive task. That’s when I decided to automate the whole process. This guide will walk you through the exact Python script and setup I use in my production environments to detect and fix this issue before it becomes a real problem.

The goal here is simple: save you time and increase the reliability of your systems. Let’s dive in.

Prerequisites

• Access to a Linux server with NFS mounts.
• Python 3 installed on the server.
• Sudo privileges to run the `mount` command.
• A list of the NFS mount points you want to monitor.

The Guide: Step-by-Step

Step 1: Setting Up Your Project

First things first, let’s get our workspace organized. On the server, you’ll want to create a dedicated directory for our monitoring script. I usually call it something like `nfs_monitor`.

I’ll skip the standard virtualenv setup since you likely have your own workflow for that. The key is to work in an isolated Python environment. Once that’s active, you’ll need one third-party library. You can install it via pip: `python-dotenv`. This library makes it easy to manage our configuration. Inside your project directory, create two files: `nfs_remount_monitor.py` for our script and `config.env` for our settings.

Step 2: Configuring Your Mount Points

The `config.env` file is where we’ll list the mount points to check. This approach keeps our configuration separate from our code, which is always a good practice. Open up `config.env` and add your mounts, one per line, like this:

# List of NFS mount points to monitor
MOUNTS_TO_CHECK=/mnt/data,/mnt/backups,/mnt/shared_volume

Just a simple, comma-separated list. It’s clean and easy to update without touching the script.

Step 3: The Python Script Logic

Before we look at the code, let’s break down what our script will do. It’s a straightforward loop:

1. Load Configuration: Read the `MOUNTS_TO_CHECK` variable from our `config.env` file.
2. Check Each Mount: For each path in the list, we’ll run a command to see if it’s healthy. A simple `ls -ld` or `stat` is perfect. If the mount is stale, this command will hang or fail with a “Stale file handle” error. We’ll use a timeout to catch this.
3. Attempt Remount: If a check fails, the script will log the issue and then attempt to execute `sudo mount -o remount /path/to/mount`.
4. Log Everything: We need a clear record of what the script is doing. We’ll log which mounts are healthy, which are stale, and the outcome of any remount attempts.

Pro Tip: In some rare, stubborn cases, a standard remount might not be enough. If you find a mount is consistently failing to remount, you might consider a “lazy unmount” (`umount -l`) followed by a full `mount -a`. However, I’d use this as a last resort. A lazy unmount can sometimes hide underlying issues, so proceed with caution and thorough testing.

Step 4: The Full Python Script

Alright, let’s put the logic into code. Here is the content for your `nfs_remount_monitor.py` file. I’ve added comments to explain each part.

import os
import subprocess
import logging
from dotenv import load_dotenv

# --- Configuration ---
# Set up basic logging to a file and to the console.
logging.basicConfig(
    level=logging.INFO,
    format='%(asctime)s - %(levelname)s - %(message)s',
    handlers=[
        logging.FileHandler("nfs_monitor.log"),
        logging.StreamHandler()
    ]
)

# Load environment variables from config.env
load_dotenv('config.env')
MOUNTS_TO_CHECK = os.getenv('MOUNTS_TO_CHECK', '').split(',')

# Timeout for the check command in seconds
CHECK_TIMEOUT = 5

def check_mount_health(mount_path):
    """
    Checks if an NFS mount is responsive by running a stat command with a timeout.
    Returns True if healthy, False if stale or unresponsive.
    """
    if not os.path.ismount(mount_path):
        logging.warning(f"Path is not a mount point: {mount_path}")
        return False # Or True, depending on if you want to ignore non-mounts

    logging.info(f"Checking mount: {mount_path}")
    try:
        # The 'stat' command is a lightweight way to check the filesystem.
        # We run it with a timeout to detect a hung/stale mount.
        subprocess.run(
            ['stat', mount_path],
            check=True,
            timeout=CHECK_TIMEOUT,
            capture_output=True,
            text=True
        )
        logging.info(f"SUCCESS: Mount {mount_path} is healthy.")
        return True
    except subprocess.TimeoutExpired:
        logging.error(f"FAILURE: Timeout expired for {mount_path}. Likely a stale handle.")
        return False
    except subprocess.CalledProcessError as e:
        # Check stderr for the specific "Stale file handle" message
        if 'Stale file handle' in e.stderr:
            logging.error(f"FAILURE: Detected 'Stale file handle' on {mount_path}.")
        else:
            logging.error(f"FAILURE: Command failed on {mount_path} with error: {e.stderr}")
        return False

def attempt_remount(mount_path):
    """
    Attempts to remount the specified NFS path using 'mount -o remount'.
    Requires sudo privileges configured for this command.
    """
    logging.warning(f"Attempting to remount {mount_path}...")
    try:
        # We need sudo to run the mount command. Ensure sudoers is configured.
        result = subprocess.run(
            ['sudo', 'mount', '-o', 'remount', mount_path],
            check=True,
            capture_output=True,
            text=True
        )
        logging.info(f"REMOUNT SUCCESS: {mount_path} remounted successfully.")
        logging.info(f"Remount output: {result.stdout}")
        return True
    except subprocess.CalledProcessError as e:
        logging.critical(f"REMOUNT FAILED: Could not remount {mount_path}. Error: {e.stderr}")
        return False

def main():
    """ Main function to orchestrate the monitoring and remounting process. """
    logging.info("--- Starting NFS Mount Health Check ---")
    if not MOUNTS_TO_CHECK or MOUNTS_TO_CHECK == ['']:
        logging.warning("No mount points configured in config.env. Nothing to do.")
        return

    for mount in MOUNTS_TO_CHECK:
        mount = mount.strip()
        if not mount:
            continue

        if not check_mount_health(mount):
            attempt_remount(mount)
    
    logging.info("--- NFS Mount Health Check Finished ---")

if __name__ == "__main__":
    main()

Step 5: Automating with Cron

A script is only useful if it runs automatically. For this, I use a simple cron job. You’ll need to edit your user’s crontab to add a new entry. The goal is to run our Python script on a regular schedule. I find every 15 minutes is a good balance between responsiveness and system load.

Here’s what the cron entry would look like. Remember to adjust the path to wherever you placed your project.

*/15 * * * * cd /path/to/your/nfs_monitor && python3 nfs_remount_monitor.py

This command changes to the script’s directory (so it can find `config.env` and the log file) and then executes it using `python3`.

Common Pitfalls (Where I Usually Mess Up)

• Sudo Permissions: This is the big one. The script calls `sudo mount`, which will fail if it prompts for a password. You need to configure the `sudoers` file to allow the user running the script to execute `mount -o remount` without a password. Be very specific here for security—only allow that exact command. A typical `sudoers` entry might look something like `youruser ALL=(ALL) NOPASSWD: /path/to/mount`. Work with your security team to get this right.
• Incorrect Paths in `config.env`: A simple typo in a mount path means the script won’t check it. Double-check every path you add to the configuration file.
• Forgetting the Virtual Environment in Cron: If you use a virtual environment, your cron command needs to activate it or call the python executable from within the venv’s directory. A common approach is to use the full path to the python executable from your venv in the cron job.

Conclusion

And that’s it. You now have a robust, automated system for handling one of the most annoying infrastructure hiccups. This script has saved me countless hours and prevented application failures caused by unresponsive storage. It’s a classic DevOps win: we identified a repetitive, manual task and replaced it with a reliable, automated solution.

Feel free to adapt the script to your needs—add alerting, more detailed logging, or different recovery actions. The foundation is solid. Let me know if you find ways to improve it!

Darian Vance

Lead Cloud Architect & DevOps Strategist

With over 12 years in system architecture and automation, Darian specializes in simplifying complex cloud infrastructures. An advocate for open-source solutions, he founded TechResolve to provide engineers with actionable, battle-tested troubleshooting guides and robust software alternatives.

🤖 Frequently Asked Questions