JEFF Media Developer Blog

Java & Spigot development

Menu
  • Blog
  • Main Website
  • Privacy Policy
Menu

Solution: Creating a Datastore on Proxmox Backup Server fails at “Chunkstore create” (“File exists” or “Too many links”)

Posted on December 7, 2024December 7, 2024 by mfnalex

Hi everyone! Today I was trying to create a Datastore for my Proxmox Backup Server installation on my WD MyCloud NAS that I mounted over SSHFS. While trying to create the datastore, I got the following error in the GUI:

2024-12-07T11:58:54+01:00: Chunkstore create: 97%
2024-12-07T11:59:00+01:00: Chunkstore create: 98%
2024-12-07T11:59:10+01:00: Chunkstore create: 99%
2024-12-07T11:59:13+01:00: TASK ERROR: unable to create chunk store 'nas1-pbs-grave' subdir "/mnt/nas1/pbs-grave/.chunks/fde6" - unable to create directory "/mnt/nas1/pbs-grave/.chunks/fde6" - EEXIST: File exists

Huh, that’s weird – Proxmox failed to create the “fde6” directory, claiming it already exists. But now check out this:

pbs.pve1:~# ls -l /mnt/nas1/pbs-grave/.chunks/fde6
ls: cannot access '/mnt/nas1/pbs-grave/.chunks/fde6': No such file or directory

Interesting – Proxmox cannot create the directory because “it already exists”, but when we try to ls it, it’s not there. So, what happens when we try to create it manually?

pbs.pve1:/mnt/nas1/pbs-grave/.chunks# mkdir fde6
mkdir: cannot create directory ‘fde6’: File exists

What the hell is this? Let’s try to SSH into the NAS and create the directory directly from there:

root@MyCloudEX2Ultra .chunks # mkdir fde6
mkdir: can't create directory 'fde6': Too many links

Aha! At least we now got a useful error message!

Finding the problem

Let’s find out why this happens and check out the filesystem:

root@MyCloudEX2Ultra ~ # df -T /shares/pbs-grave/
Filesystem           Type       1K-blocks      Used Available Use% Mounted on
/dev/md1             ext4       5810058912 2660225848 3091253588  46% /mnt/HD/HD_a2
root@MyCloudEX2Ultra ~ # df -iT /shares/pbs-grave/
Filesystem           Type          Inodes      Used Available Use% Mounted on
/dev/md1             ext4       183013376   2789798 180223578   2% /mnt/HD/HD_a2

Looks good. Neither disk space nor inode capacity is the problem. So let’s see what’s inside our folder:

Yikes. That’s a lot of directories. In fact, it’s pretty close to 65,000:

root@MyCloudEX2Ultra ~ # ls -1 /shares/pbs-grave/.chunks/ | wc -l
64998

Now it almost makes sense: ext4 only allows more than 65,000 hard links per directory when both the dir_nlink and dir_index option are enabled – unfortunately, on my filesystem I’m missing the dir_index option:

root@MyCloudEX2Ultra ~ # tune2fs -l /dev/md1
tune2fs 1.46.2 (28-Feb-2021)
Filesystem volume name:   <none>
Last mounted on:          /mnt/HD/HD_a2
Filesystem UUID:          43625a52-5ea5-469f-8776-62761a100216
Filesystem magic number:  0xEF53
Filesystem revision #:    1 (dynamic)
Filesystem features:      has_journal ext_attr filetype needs_recovery meta_bg e                                                                                                                                                                                                                                             xtent 64bit flex_bg sparse_super large_file huge_file uninit_bg dir_nlink extra_                                                                                                                                                                                                                                             isize
Filesystem flags:         unsigned_directory_hash
...

But then why does it fail to create the 64999th directoy, instead of the 65001th? Well, because . and .. are also directory entries 🙂

How to fix this?

Yeah so that’s pretty sad, I don’t want to reformat my NAS. We got two options to fix this:

Solution 1: tune2fs

We could use tune2fs to enable dir_index and then recreate the .chunks directory to turn it into a directory with a hash-based table (please note that I have not tested this):

root@MyCloudEX2Ultra ~ # tune2fs -O dir_index,dir_nlink /mnt/HD/HD_a2

However, this requires unmounting the filesystem, and the MyCloud OS probably wouldn’t like that. I also have no idea how that would behave in conjunction with MyCloud’s RAID and automatic rebuilding, so I better don’t touch the filesystem myself.

Solution 2: Split up the directory

FFS this is Linux, let’s just write a poor bash script to fix this shit. After all, we’re only like 500 directories short (Proxmox needs 65,536 directories and our limit is 64,998). We’ll just create 32,768 directories in one folder, then 32,768 in another folder, and then we link them together inside one directory using symbolic links – which works fine because the limit only applies to hard links.

The final folder structure shall looks like this:

.chunks/
  0000/ -> ../.chunks1/0000/
  0001/ -> ../.chunks1/0001/
  ...
  7ffe/ -> ../.chunks1/7ffe/
  7fff/ -> ../.chunks1/7fff/
  8000/ -> ../.chunks2/8000/
  8001/ --> ../chunks2/8001/
  ...
  fffe/ -> ../.chunks2/fffe/
  ffff/ -> ../.chunks2/ffff/

.chunks1/
  0000/
  0001/
  ...
  7ffe/
  7fff/

.chunks2/
  8000/
  8001/
  ...
  fffe/
  ffff/

Simply create the datastore in Proxmox Backup Server’s GUI, wait until it fails, and then run the following script (be sure to adjust the directory before).

#!/bin/bash

# MAINTAINED GIST of this script: https://gist.github.com/mfnalex/41f6eab4baf032206228035d784ab624

# Define base directories
base_dir="/mnt/nas1/pbs-grave" # Change this to your datastore location

chunks1_dir="$base_dir/.chunks1"
chunks2_dir="$base_dir/.chunks2"
chunks_symlink_dir="$base_dir/.chunks"

# Delete the existing directories Proxmox Backup Server has created
for dir in "$chunks_symlink_dir/*"; do
  rmdir "$dir"
done

# Create the directories if they don't exist
mkdir -p "$chunks1_dir"
mkdir -p "$chunks2_dir"
mkdir -p "$chunks_symlink_dir"

# Loop over the hex values for the first half
for i in $(seq 0 32767); do
  hex=$(printf "%04x" $i)
  # Create directory in chunks1
  mkdir -p "$chunks1_dir/$hex"
  # Create a symbolic link in chunks
  ln -s "$chunks1_dir/$hex" "$chunks_symlink_dir/$hex"
done

# Loop over the hex values for the second half
for i in $(seq 32768 65535); do
  hex=$(printf "%04x" $i)
  # Create directory in chunks2
  mkdir -p "$chunks2_dir/$hex"
  # Create a symbolic link in chunks
  ln -s "$chunks2_dir/$hex" "$chunks_symlink_dir/$hex"
done

echo "Directories and symbolic links have been created successfully."

The above mentioned script is also available here on GitHub.

Final step: Adding the datastore manually

Now all that’s left is adding the datastore manually to our Proxmox Backup Server config. Just add a new entry to your /etc/proxmox-backup/datastore.cfg:

datastore: nas1-pbs-grave
        path /mnt/nas1/pbs-grave

That’s it: Our Datastore is working fine 🙂

Join my Discord Server for feedback or support. Just check out the channel #programming-help 🙂

Leave a Reply Cancel reply

Your email address will not be published. Required fields are marked *

This site is protected by reCAPTCHA and the Google Privacy Policy and Terms of Service apply.

This site uses Akismet to reduce spam. Learn how your comment data is processed.

Recent Posts

  • Solution: Creating a Datastore on Proxmox Backup Server fails at “Chunkstore create” (“File exists” or “Too many links”)
  • Don’t disable dependency-reduced-pom.xml
  • Maven Multi-Module setup for supporting different NMS versions
  • How to read or block Spigot’s console output
  • Why IntelliJ complains about your pom.xml file

Recent Comments

  1. mfnalex on Creating custom heads in Spigot 1.18.1+
  2. Timon Coucke on Maven Multi-Module setup for supporting different NMS versions
  3. 3ricL on Creating custom heads in Spigot 1.18.1+
  4. Ryan Leach on Why the GPL does NOT directly apply to all Spigot plugins
  5. Azzy on NMS: Use Mojang maps for your Spigot plugins with Maven or Gradle

Archives

  • December 2024
  • August 2023
  • July 2023
  • June 2023
  • February 2023
  • January 2023
  • October 2022
  • August 2022
  • March 2022
  • December 2021
  • October 2021
  • August 2021
©2025 JEFF Media Developer Blog
We use cookies on our website to give you the most relevant experience by remembering your preferences and repeat visits. By clicking “Accept All”, you consent to the use of ALL the cookies. However, you may visit "Cookie Settings" to provide a controlled consent.
Cookie SettingsAccept All
Manage consent

Privacy Overview

This website uses cookies to improve your experience while you navigate through the website. Out of these, the cookies that are categorized as necessary are stored on your browser as they are essential for the working of basic functionalities of the website. We also use third-party cookies that help us analyze and understand how you use this website. These cookies will be stored in your browser only with your consent. You also have the option to opt-out of these cookies. But opting out of some of these cookies may affect your browsing experience.
Necessary
Always Enabled
Necessary cookies are absolutely essential for the website to function properly. These cookies ensure basic functionalities and security features of the website, anonymously.
CookieDurationDescription
cookielawinfo-checkbox-analytics11 monthsThis cookie is set by GDPR Cookie Consent plugin. The cookie is used to store the user consent for the cookies in the category "Analytics".
cookielawinfo-checkbox-functional11 monthsThe cookie is set by GDPR cookie consent to record the user consent for the cookies in the category "Functional".
cookielawinfo-checkbox-necessary11 monthsThis cookie is set by GDPR Cookie Consent plugin. The cookies is used to store the user consent for the cookies in the category "Necessary".
cookielawinfo-checkbox-others11 monthsThis cookie is set by GDPR Cookie Consent plugin. The cookie is used to store the user consent for the cookies in the category "Other.
cookielawinfo-checkbox-performance11 monthsThis cookie is set by GDPR Cookie Consent plugin. The cookie is used to store the user consent for the cookies in the category "Performance".
viewed_cookie_policy11 monthsThe cookie is set by the GDPR Cookie Consent plugin and is used to store whether or not user has consented to the use of cookies. It does not store any personal data.
Functional
Functional cookies help to perform certain functionalities like sharing the content of the website on social media platforms, collect feedbacks, and other third-party features.
Performance
Performance cookies are used to understand and analyze the key performance indexes of the website which helps in delivering a better user experience for the visitors.
Analytics
Analytical cookies are used to understand how visitors interact with the website. These cookies help provide information on metrics the number of visitors, bounce rate, traffic source, etc.
Advertisement
Advertisement cookies are used to provide visitors with relevant ads and marketing campaigns. These cookies track visitors across websites and collect information to provide customized ads.
Others
Other uncategorized cookies are those that are being analyzed and have not been classified into a category as yet.
SAVE & ACCEPT