Hi everyone! Today I was trying to create a Datastore for my Proxmox Backup Server installation on my WD MyCloud NAS that I mounted over SSHFS. While trying to create the datastore, I got the following error in the GUI:

2024-12-07T11:58:54+01:00: Chunkstore create: 97%
2024-12-07T11:59:00+01:00: Chunkstore create: 98%
2024-12-07T11:59:10+01:00: Chunkstore create: 99%
2024-12-07T11:59:13+01:00: TASK ERROR: unable to create chunk store 'nas1-pbs-grave' subdir "/mnt/nas1/pbs-grave/.chunks/fde6" - unable to create directory "/mnt/nas1/pbs-grave/.chunks/fde6" - EEXIST: File exists

Huh, that’s weird – Proxmox failed to create the “fde6” directory, claiming it already exists. But now check out this:

pbs.pve1:~# ls -l /mnt/nas1/pbs-grave/.chunks/fde6
ls: cannot access '/mnt/nas1/pbs-grave/.chunks/fde6': No such file or directory

Interesting – Proxmox cannot create the directory because “it already exists”, but when we try to ls it, it’s not there. So, what happens when we try to create it manually?

pbs.pve1:/mnt/nas1/pbs-grave/.chunks# mkdir fde6
mkdir: cannot create directory ‘fde6’: File exists

What the hell is this? Let’s try to SSH into the NAS and create the directory directly from there:

root@MyCloudEX2Ultra .chunks # mkdir fde6
mkdir: can't create directory 'fde6': Too many links

Aha! At least we now got a useful error message!

Finding the problem

Let’s find out why this happens and check out the filesystem:

root@MyCloudEX2Ultra ~ # df -T /shares/pbs-grave/
Filesystem           Type       1K-blocks      Used Available Use% Mounted on
/dev/md1             ext4       5810058912 2660225848 3091253588  46% /mnt/HD/HD_a2
root@MyCloudEX2Ultra ~ # df -iT /shares/pbs-grave/
Filesystem           Type          Inodes      Used Available Use% Mounted on
/dev/md1             ext4       183013376   2789798 180223578   2% /mnt/HD/HD_a2

Looks good. Neither disk space nor inode capacity is the problem. So let’s see what’s inside our folder:

Yikes. That’s a lot of directories. In fact, it’s pretty close to 65,000:

root@MyCloudEX2Ultra ~ # ls -1 /shares/pbs-grave/.chunks/ | wc -l
64998

Now it almost makes sense: ext4 only allows more than 65,000 hard links per directory when both the dir_nlink and dir_index option are enabled – unfortunately, on my filesystem I’m missing the dir_index option:

root@MyCloudEX2Ultra ~ # tune2fs -l /dev/md1
tune2fs 1.46.2 (28-Feb-2021)
Filesystem volume name:   <none>
Last mounted on:          /mnt/HD/HD_a2
Filesystem UUID:          43625a52-5ea5-469f-8776-62761a100216
Filesystem magic number:  0xEF53
Filesystem revision #:    1 (dynamic)
Filesystem features:      has_journal ext_attr filetype needs_recovery meta_bg e                                                                                                                                                                                                                                             xtent 64bit flex_bg sparse_super large_file huge_file uninit_bg dir_nlink extra_                                                                                                                                                                                                                                             isize
Filesystem flags:         unsigned_directory_hash
...

But then why does it fail to create the 64999th directoy, instead of the 65001th? Well, because . and .. are also directory entries 🙂

How to fix this?

Yeah so that’s pretty sad, I don’t want to reformat my NAS. We got two options to fix this:

Solution 1: tune2fs

We could use tune2fs to enable dir_index and then recreate the .chunks directory to turn it into a directory with a hash-based table (please note that I have not tested this):

root@MyCloudEX2Ultra ~ # tune2fs -O dir_index,dir_nlink /mnt/HD/HD_a2

However, this requires unmounting the filesystem, and the MyCloud OS probably wouldn’t like that. I also have no idea how that would behave in conjunction with MyCloud’s RAID and automatic rebuilding, so I better don’t touch the filesystem myself.

Solution 2: Split up the directory

FFS this is Linux, let’s just write a poor bash script to fix this shit. After all, we’re only like 500 directories short (Proxmox needs 65,536 directories and our limit is 64,998). We’ll just create 32,768 directories in one folder, then 32,768 in another folder, and then we link them together inside one directory using symbolic links – which works fine because the limit only applies to hard links.

The final folder structure shall looks like this:

.chunks/
  0000/ -> ../.chunks1/0000/
  0001/ -> ../.chunks1/0001/
  ...
  7ffe/ -> ../.chunks1/7ffe/
  7fff/ -> ../.chunks1/7fff/
  8000/ -> ../.chunks2/8000/
  8001/ --> ../chunks2/8001/
  ...
  fffe/ -> ../.chunks2/fffe/
  ffff/ -> ../.chunks2/ffff/

.chunks1/
  0000/
  0001/
  ...
  7ffe/
  7fff/

.chunks2/
  8000/
  8001/
  ...
  fffe/
  ffff/

Simply create the datastore in Proxmox Backup Server’s GUI, wait until it fails, and then run the following script (be sure to adjust the directory before).

#!/bin/bash

# MAINTAINED GIST of this script: https://gist.github.com/mfnalex/41f6eab4baf032206228035d784ab624

# Define base directories
base_dir="/mnt/nas1/pbs-grave" # Change this to your datastore location

chunks1_dir="$base_dir/.chunks1"
chunks2_dir="$base_dir/.chunks2"
chunks_symlink_dir="$base_dir/.chunks"

# Delete the existing directories Proxmox Backup Server has created
for dir in "$chunks_symlink_dir/*"; do
  rmdir "$dir"
done

# Create the directories if they don't exist
mkdir -p "$chunks1_dir"
mkdir -p "$chunks2_dir"
mkdir -p "$chunks_symlink_dir"

# Loop over the hex values for the first half
for i in $(seq 0 32767); do
  hex=$(printf "%04x" $i)
  # Create directory in chunks1
  mkdir -p "$chunks1_dir/$hex"
  # Create a symbolic link in chunks
  ln -s "$chunks1_dir/$hex" "$chunks_symlink_dir/$hex"
done

# Loop over the hex values for the second half
for i in $(seq 32768 65535); do
  hex=$(printf "%04x" $i)
  # Create directory in chunks2
  mkdir -p "$chunks2_dir/$hex"
  # Create a symbolic link in chunks
  ln -s "$chunks2_dir/$hex" "$chunks_symlink_dir/$hex"
done

echo "Directories and symbolic links have been created successfully."

The above mentioned script is also available here on GitHub.

Final step: Adding the datastore manually

Now all that’s left is adding the datastore manually to our Proxmox Backup Server config. Just add a new entry to your /etc/proxmox-backup/datastore.cfg:

datastore: nas1-pbs-grave
        path /mnt/nas1/pbs-grave

That’s it: Our Datastore is working fine 🙂