Fixing Linux (SMB) Issues with Azure Files: A Troubleshooting Guide

Table of Contents

Working with SMB Azure file shares on Linux clients can occasionally present unique challenges. This comprehensive guide details common issues encountered in these environments, offering insights into their underlying causes and providing effective resolutions. Understanding these problems and their fixes is crucial for maintaining seamless data access and optimal performance for your Linux workloads integrated with Azure Files. This document serves as a valuable resource for system administrators and IT professionals looking to troubleshoot and resolve common SMB connectivity and operational issues.

Applicability of This Guide¶

This troubleshooting guide is specifically designed to address issues pertaining to SMB (Server Message Block) file shares within Azure. While Azure Files also supports NFS (Network File System) for Linux clients, the solutions detailed here are tailored for SMB protocols. It is important to identify the type of file share you are utilizing to ensure the relevance of the troubleshooting steps.

The information provided is applicable to various Azure file share configurations, ensuring broad utility for a range of deployment scenarios. Below is a detailed breakdown of the file share types for which these solutions are relevant.

File Share Type	SMB	NFS
Standard file shares (GPv2), LRS/ZRS	✔
Standard file shares (GPv2), GRS/GZRS	✔
Premium file shares (FileStorage), LRS/ZRS	✔

Running Essential Diagnostics¶

Before diving into specific issues, leveraging diagnostic tools can significantly streamline the troubleshooting process. These tools help confirm that your client systems meet the necessary prerequisites and aid in gathering vital debug information for issues that might be difficult to replicate manually. Proactive diagnostics can often prevent more extensive problems from occurring.

Using AzFileDiagnostics¶

The AzFileDiagnostics tool is an invaluable resource designed to automate symptom detection and verify prerequisites on your Linux client. It ensures that your environment is properly configured for optimal performance when interacting with Azure File Shares. Utilizing this tool can save significant time by quickly identifying common misconfigurations or missing components.

This utility not only checks for common issues but also helps in setting up your client for the best possible experience. Regularly running AzFileDiagnostics is a recommended practice to maintain a healthy connection to your Azure SMB shares. It is particularly useful when initially setting up a connection or after making significant changes to your system.

Employing the Always-On Diagnostics (AOD) Tool¶

For more persistent or elusive issues, the Always-On Diagnostics (AOD) tool offers a robust solution for collecting in-depth logs on both SMB and NFSv4 Linux clients. This daemon runs continuously in the background as a system service, providing a constant monitoring capability. It can be configured to detect anomalies across a variety of sources within your system.

AOD captures critical data from dmesg logs, debug information, error metrics, and latency metrics, offering a holistic view of system behavior. Furthermore, it can gather data from network-related tools like tcpdump, nfsstat, and mountstsat, alongside system performance indicators such as CPU and memory usage. This comprehensive data collection is essential for diagnosing complex interdependencies that might be affecting your file share operations.

Currently, the Always-On Diagnostics tool is compatible with systems running SUSE Linux Enterprise Server 15 (SLES 15) and Red Hat Enterprise Linux 8 (RHEL 8). Ensure your operating system is one of these supported versions before attempting installation. Following the specific installation instructions for your chosen OS is crucial for proper deployment and functionality.

Addressing ‘cp -p’ Command Failures¶

When attempting to copy files using the cp -p command on Linux/Unix platforms, you might encounter issues where the command fails if the source and destination files are owned by different users. This specific problem often manifests when trying to preserve file attributes like timestamps. Understanding the interaction of the copy command with file ownership is key to resolving this.

Cause¶

The root cause of this issue lies in how the cp -p command interacts with the underlying COPYFILE operation, particularly when a force flag (-f) is implicitly or explicitly involved. On Unix-like systems, if COPYFILE results in cp -p -f, the command will attempt to copy but will fail to preserve the timestamp of a file that you do not own. This limitation is by design, as modifying timestamps of files owned by other users typically requires elevated permissions.

Workaround¶

To successfully copy files while preserving their attributes, especially timestamps, you should execute the cp -p command as the storage account user. This ensures that the operation has the necessary permissions to modify file attributes, even when ownership differs. Follow these steps to perform the copy operation:

1. Define Storage Account Name:
   

   str_acc_name=[storage account name]
   

   
   This command sets a shell variable for easy use of your storage account name. Replace [storage account name] with the actual name of your Azure storage account.

2. Create a User for the Storage Account:
   

   sudo useradd $str_acc_name
   

   
   This command creates a new system user with the name of your storage account. This user will be used to perform the file copy operation.

3. Set a Password for the New User:
   

   sudo passwd $str_acc_name
   

   
   Assign a strong password to the newly created user when prompted. This step secures the new user account.

4. Switch to the Storage Account User:
   

   su $str_acc_name
   

   
   Switch your current shell session to the context of the storage account user. This ensures that subsequent commands are executed with the permissions of this user.

5. Copy the File:
   

   cp -p filename.txt /share
   

   
   Now, execute the cp -p command to copy your file. Replace filename.txt with the name of the file you wish to copy and /share with the target directory within your mounted Azure file share. This approach bypasses the ownership conflict by operating under the appropriate user context.

Resolving ‘ls: cannot access ‘‘: Input/output error’¶

Users sometimes encounter a specific issue when attempting to list files within an Azure file share using the ls command on a Linux client. The command may hang indefinitely before eventually failing and displaying the error message: ls: cannot access ‘<path>‘: Input/output error. This behavior indicates a deeper problem within the system’s interaction with the mounted file share.

Solution¶

This input/output error is often a symptom of a known bug within older Linux kernel versions that affects SMB operations. The most effective and permanent solution is to upgrade your Linux kernel to a version that includes the necessary fixes for this problem. These kernel versions have specific patches that address the underlying instability in SMB handling.

We recommend upgrading your Linux kernel to one of the following versions or any newer release:

• 4.4.87+: Any version equal to or greater than 4.4.87.
• 4.9.48+: Any version equal to or greater than 4.9.48.
• 4.12.11+: Any version equal to or greater than 4.12.11.
• All versions that are greater than or equal to 4.13: These versions inherently include the necessary fixes.

By updating your kernel, you ensure that the operating system can correctly handle file listing and other I/O operations on SMB Azure file shares without encountering this specific hang and error. Always consult your distribution’s documentation for the recommended and safest way to perform kernel upgrades.

Managing Symbolic Link Issues¶

Symbolic links, or symlinks, are a powerful feature in Linux for creating shortcuts to files or directories. However, when mounting Azure file shares on Linux using SMB, you might discover that symbolic links do not function as expected by default. Attempts to create or interact with symlinks can result in errors, indicating a lack of native support.

Consider the following scenario where an attempt to create a symbolic link fails:

sudo ln -s linked -n t

This command will typically return an error message similar to this:

ln: failed to create symbolic link 't': Operation not supported

Cause¶

The primary reason for this limitation is that mounting Azure file shares on Linux via SMB does not, by default, enable support for symbolic links. Furthermore, the standard Linux SMB client does not inherently support creating Windows-style symbolic links when communicating over the SMB 2 or 3 protocols. This incompatibility prevents the direct creation and interpretation of symlinks that might be commonly used in Windows environments.

However, the Linux client does support an alternative style of symbolic links known as Minshall+French symlinks. This specific format is compatible with both create and follow operations, offering a viable solution for users who require symbolic link functionality. This format is also widely used by macOS, making it a robust and cross-platform friendly option for symlink management.

Solution¶

To enable symbolic link functionality and resolve this issue, you need to explicitly include the mfsymlinks mount option when mounting your Azure file share. This option instructs the Linux SMB client to utilize the Minshall+French symlink format, allowing for successful creation and traversal of symbolic links. We highly recommend using mfsymlinks due to its broader compatibility and effectiveness.

To enable symlinks, append ,mfsymlinks to the end of your existing SMB mount command. The modified command will then resemble the following structure:

sudo mount -t cifs //<storage-account-name>.file.core.windows.net/<share-name> <mount-point> -o vers=<smb-version>,username=<storage-account-name>,password=<storage-account-key>,dir_mode=0777,file_mode=0777,serverino,mfsymlinks

Replace <storage-account-name>, <share-name>, <mount-point>, <smb-version>, <storage-account-key> with your specific details. Once mounted with this option, you can proceed to create and manage symbolic links within your Azure file share as you normally would on a local Linux filesystem. This allows for greater flexibility and consistency in your file management strategies.

Troubleshooting Inaccessible Folders or Files¶

A common and frustrating issue for Linux users mounting Azure file shares is the inability to access certain folders or files, despite the share being successfully mounted. Commands such as du and ls, or even third-party applications, may fail to access specific items, reporting a “No such file or directory” error. This can be particularly perplexing when the files or folders appear to exist.

Cause 1: Character Encoding Discrepancies¶

One frequent cause of inaccessible files or folders stems from character encoding discrepancies. This often occurs when files or directories are uploaded from a system that encodes certain characters at the end of a name differently than Linux expects. A notable example involves files uploaded from Macintosh computers. These systems might replace standard characters like 0x20 (space) or 0x2E (dot) with alternative characters such as “0xF028” or “0xF029” at the end of filenames. Linux, when trying to interpret these names, fails to match them, leading to the “No such file or directory” error.

Solution 1¶

To resolve character encoding issues, you can utilize the mapchars option when mounting the Azure file share on your Linux client. This option instructs the CIFS (Common Internet File System) client to remap unsupported characters in filenames to acceptable alternatives, thereby allowing Linux to correctly identify and access the files.

Instead of a standard mount command like this:

sudo mount -t cifs $smbPath $mntPath -o vers=3.0,username=$storageAccountName,password=$storageAccountKey,serverino

You should include the mapchars option:

sudo mount -t cifs $smbPath $mntPath -o vers=3.0,username=$storageAccountName,password=$storageAccountKey,serverino,mapchars

By adding ,mapchars, you enable the necessary character remapping, which often resolves access issues for files that were problematic due to character encoding. This ensures broader compatibility, especially in mixed-OS environments.

Cause 2: Zombie Files¶

Another potential cause for “No such file or directory” errors, even for seemingly existing items, is the presence of “zombie files.” This scenario occurs when a file is deleted from an SMB share while an active handle to that file remains open on a client. The SMB server, in such cases, continues to maintain a ghost or “zombie” entry for the file until the very last handle referring to it is closed.

Any subsequent attempts to perform operations, such as listing or accessing, on these zombie files by other Linux clients may result in the “No such file or directory” error. The file is logically gone, but its metadata might still be partially held by the server due to lingering open handles, causing confusion for new access attempts. This can lead to unexpected behavior and data access problems.

Solution 2¶

If you suspect zombie files are causing your access issues, and the inaccessible file or folder is critical, the most reliable solution is to restore the deleted file from your latest backup. This ensures that a clean, accessible version of the file is available, eliminating any residual issues from the zombie state. Regular backups are therefore crucial for mitigating data loss and access problems in such scenarios.

Addressing DNS Issues with Azure Storage Account Live Migration¶

One complex issue that can arise when using Azure file shares with Linux clients pertains to DNS resolution, especially after a storage account undergoes a live migration or failover. Users might suddenly experience file I/Os on their mounted filesystem failing with “Host is down” or “Permission denied” errors. Concurrently, the Linux dmesg logs on the client may show recurring errors such as:

Status code returned 0xc000006d STATUS_LOGON_FAILURE
cifs_setup_session: 2 callbacks suppressed
CIFS VFS: \\contoso.file.core.windows.net Send error in SessSetup = -13

A key indicator of this problem is when the server’s Fully Qualified Domain Name (FQDN), which typically resolves to a specific IP address, suddenly begins resolving to a different IP address than the one the client is currently connected to. This discrepancy signals a change in the underlying network mapping that the Linux client is failing to adapt to.

Cause¶

Azure storage accounts are periodically live-migrated between different storage clusters for various operational reasons, primarily for capacity load balancing and performance optimization. During such an account migration, Azure Files traffic is redirected from the source cluster to the destination cluster by updating the DNS mappings. This update points the FQDN to the IP address of the new destination cluster, effectively blocking all traffic to the old source cluster for that account.

While the expectation is that the SMB client on Linux will detect these DNS updates and redirect subsequent traffic to the new destination cluster, a bug in the Linux SMB kernel client prevents this redirection from taking effect. Consequently, the data traffic continues to be sent to the original source cluster, which, post-migration, is no longer serving that particular storage account. This persistent misdirection results in the “Host is down” or “Permission denied” errors, as the client is attempting to communicate with a non-responsive or incorrect endpoint.

Workaround¶

While a permanent fix involves updating your operating system, several workarounds can mitigate the DNS resolution issue temporarily. Rebooting the client OS will typically resolve the issue, but this is a disruptive and temporary solution, as the problem might reoccur if the underlying kernel bug persists and another migration happens.

Unmounting and then remounting the share might also appear to solve the problem initially. This is because a new mount action bypasses the stale SMB kernel cache and performs a fresh DNS resolution in user space. However, during any subsequent network disconnection or recovery, the kernel’s own DNS cache might be consulted again. If this cache still holds the old, incorrect DNS entry, the issue can resurface. This behavior is not exclusive to storage account migrations but can happen during any scenario where the server’s IP changes.

To provide a more robust temporary solution, you can explicitly clear the kernel’s DNS resolver cache. This forces the system to perform a fresh DNS lookup for the storage account’s FQDN. Follow these steps to clear the cache:

1. Display the status of the kernel dns_resolver module:
   Run the following command to check the current state of the kernel’s DNS cache:
   

   grep '.dns_resolver' /proc/keys
   

   
   You should observe output similar to this example, indicating active entries:
   

   132b6bbf I------     1 perm 1f030000     0     0 keyring   .dns_resolver: 1
   

2. Clear the kernel DNS resolver cache:
   Execute the following command to flush the cache. This command identifies the key ID of the dns_resolver and uses keyctl to clear it.
   

   sudo keyctl clear $((16#$(grep '.dns_resolver' /proc/keys | cut -f1 -d\\ ) ))
   

3. Display the status of the kernel dns_resolver module again:
   Re-run the status check to confirm that the cache has been cleared:
   

   grep '.dns_resolver' /proc/keys
   

   
   The output should now indicate an empty cache, similar to this:
   

   132b6bbf I------     1 perm 1f030000     0     0 keyring   .dns_resolver: empty
   

4. Unmount and remount the share:
   After clearing the cache, unmount and then remount your Azure file share. This action will trigger a new DNS resolution that should now correctly point to the updated IP address of the storage account, mitigating the issue.

It is important to note that on some older Linux distributions, these mitigation steps might not be fully effective. In such cases, rebooting the client OS may be the only temporary solution. For a truly permanent resolution, consider implementing a private endpoint for your storage account and connecting to the file share using a private link. This approach bypasses public DNS resolution issues entirely by providing a direct, private connection.

Solution¶

For a permanent and robust resolution to DNS issues related to Azure storage account live migration, it is imperative to upgrade your client OS to a Linux distribution version that includes specific kernel fixes for this problem. Several critical fixes for the Linux SMB kernel client have been submitted and integrated into the mainline Linux kernel. These updates specifically address the bug that prevents proper DNS resolution updates.

The following Linux distributions and their respective kernel versions have incorporated these crucial fixes:

• Ubuntu: Versions 20.04, 22.04, 24.04, and AKS 22.04. The fixes are typically rolled out in kernel version 5.15.0-1068 and later.
• Red Hat Enterprise Linux (RHEL): Versions 8.6 and later.
• SUSE Linux Enterprise Server (SLES): Versions 15SP2, 15SP3, 15SP4, and 15SP5.
• Azure Linux: Versions 2.0 (with fixes rolled out in kernel version 5.15.159.1 and later) and 3.0.

Some distributions may have backported these fixes to earlier kernel versions or different release branches. To verify if your specific distro version includes the necessary patches, you can check for the presence of the following key kernel commits:

• cifs: On cifs_reconnect, resolve the hostname again: This commit ensures that the hostname is re-resolved during a CIFS reconnection attempt.
• cifs: use the expiry output of dns_query to schedule next resolution: This patch improves how CIFS schedules future DNS resolutions based on the DNS query’s expiry information.
• cifs: set a minimum of 120s for next dns resolution: This commit enforces a minimum delay for DNS re-resolution to prevent excessive queries.
• cifs: To match file servers, make sure the server hostname matches: This fix enhances server hostname matching to ensure correct identification during connections.
• cifs: fix memory leak of smb3_fs_context_dup::server_hostname: This addresses a memory leak issue related to the server hostname in the SMB3 file system context.
• dns: Apply a default TTL to records obtained from getaddrinfo(): This patch ensures a default Time-To-Live (TTL) is applied to DNS records obtained via getaddrinfo().
• keys: Fix overwrite of key expiration on instantiation: This addresses an issue where key expiration might be incorrectly overwritten during instantiation, which can impact DNS caching mechanisms.

By ensuring your Linux client runs an updated kernel with these specific fixes, you establish a more resilient and stable connection to Azure SMB file shares, preventing the recurrence of DNS-related access issues during storage account migrations.

Resolving SMB Mount Issues when FIPS is Enabled¶

When the Federal Information Processing Standard (FIPS) mode is enabled on a Linux virtual machine, users might encounter significant difficulties mounting SMB file shares. This is due to FIPS’s strict cryptographic requirements, which can conflict with certain authentication methods. The Linux dmesg logs on the client will typically display errors indicating cryptographic failures, such as:

kernel: CIFS: VFS: Could not allocate crypto hmac(md5)
kernel: CIFS: VFS: Error -2 during NTLMSSP authentication
kernel: CIFS: VFS: \\contoso.file.core.windows.net Send error in SessSetup = -2
kernel: CIFS: VFS: cifs_mount failed w/return code = -2

These error messages clearly point to a cryptographic algorithm mismatch, specifically involving MD5 during NTLMSSP authentication. FIPS is a set of rigorous standards established by the U.S. government to ensure the security and integrity of computer systems. When a system operates in FIPS mode, it adheres to specific cryptographic requirements, often restricting the use of algorithms deemed less secure.

Cause¶

The root cause of SMB mount failures when FIPS is enabled lies in the authentication mechanism used by the SMB file share client. By default, many SMB clients use NTLMSSP (NT LAN Manager Security Support Provider) authentication, which relies on the MD5 hashing algorithm for certain cryptographic operations. However, in FIPS mode, the MD5 algorithm is explicitly restricted because it is not considered FIPS-compliant. MD5, while widely used for generating 128-bit hash values, has been deemed insecure for cryptographic purposes due to known collision vulnerabilities.

Therefore, when a Linux VM is operating in FIPS mode, any attempt to perform NTLMSSP authentication that utilizes MD5 will fail, leading to the inability to mount the SMB file share. This is a security measure to enforce compliance with the FIPS standards, preventing the use of cryptographic primitives that do not meet the required security posture.

How to check if FIPS mode is enabled

To confirm whether FIPS mode is currently active on your Linux client, execute the following command in your terminal:

sudo cat /proc/sys/crypto/fips_enabled

If the command returns a value of 1, it indicates that FIPS mode is enabled on your system. A value of 0 means FIPS mode is disabled.

Solution¶

To successfully mount an SMB file share on a Linux VM where FIPS is enabled, you have two primary options. The recommended approach is to enable a FIPS-compliant authentication method, such as Kerberos. Alternatively, if FIPS mode was enabled unintentionally or is not a strict requirement, you can disable it.

Option 1: Enable Kerberos authentication for SMB file share¶

The most secure and FIPS-compliant solution is to enable Kerberos/Azure AD authentication for your SMB file share. Kerberos is a network authentication protocol that provides strong authentication for client/server applications by using secret-key cryptography. Unlike NTLMSSP, Kerberos does not rely on MD5 for its core cryptographic functions and is fully compatible with FIPS requirements.

To implement Kerberos authentication for Linux clients accessing Azure Files, you will need to configure Azure Active Directory (Azure AD) authentication over SMB. This involves setting up the necessary AD components and ensuring your Linux clients are properly configured to use Kerberos. For detailed instructions on how to set this up, refer to the official Azure documentation on “Enable Active Directory authentication over SMB for Linux clients accessing Azure Files.” This will provide a robust and FIPS-compliant access method.

Option 2: Disable FIPS to mount the Samba share¶

If FIPS mode was enabled by mistake, or if your operational requirements do not mandate FIPS compliance, you can disable it to allow NTLMSSP authentication to proceed. Disabling FIPS involves modifying system configuration files and rebooting the VM. Proceed with caution, as disabling FIPS will remove the FIPS compliance protections from your system.

Follow these steps to disable FIPS mode:

1. Change the crypto.fips_enabled sysctl value to 0:
   Edit the /etc/sysctl.conf file and locate the line that sets crypto.fips_enabled. Change its value from 1 to 0. If the line doesn’t exist, you can add crypto.fips_enabled = 0.
   

   sudo sed -i '/^crypto.fips_enabled/c\crypto.fips_enabled = 0' /etc/sysctl.conf
   sudo sysctl -p
   

2. Modify GRUB_CMDLINE_LINUX_DEFAULT in /etc/default/grub:
   Open the /etc/default/grub file and locate the GRUB_CMDLINE_LINUX_DEFAULT parameter. Remove fips=1 from this line if it is present. For example, if it was GRUB_CMDLINE_LINUX_DEFAULT="quiet splash fips=1", change it to GRUB_CMDLINE_LINUX_DEFAULT="quiet splash".

3. Rebuild the grub2 config file:
   After modifying the GRUB configuration, you must regenerate the GRUB configuration file:
   

   sudo grub2-mkconfig -o /boot/grub2/grub.cfg
   

4. Rebuild the initramfs image:
   The initial RAM filesystem (initramfs) also needs to be rebuilt to reflect the FIPS change:
   

   sudo dracut -fv
   

5. Reboot the VM:
   For the changes to take full effect, you must reboot your virtual machine:
   

   sudo reboot
   

   
   After the reboot, FIPS mode should be disabled, allowing NTLMSSP authentication to function correctly and enabling you to mount your SMB share.

For more detailed information and distribution-specific considerations regarding FIPS mode and CIFS mounts, consult the following resources from Linux distributors:

• Red Hat: “Why would enabling FIPS mode in the kernel break CIFS mounts”
• SUSE: “CIFS mount fails with error “mount error(2): No such file or directory”“

Conclusion¶

Effectively managing and troubleshooting SMB Azure file shares on Linux clients requires a solid understanding of potential issues and their resolutions. From diagnostic tools to addressing specific kernel bugs, character encoding problems, DNS migration challenges, and FIPS compliance conflicts, this guide has provided comprehensive insights and actionable steps. By implementing these solutions, you can ensure robust connectivity and optimal performance for your Linux systems interacting with Azure Files.

We encourage you to share your experiences, ask questions, or provide feedback in the comments section below. Your contributions help foster a community of learning and improvement for all users.