After a long period of contemplation and indecision, I upgraded the hardware on my Ubuntu Home Server last weekend. I’ll write a separate post about my adventures there (mostly self-inflicted).
As part of the upgrade, I also clean installed Ubuntu Server 14.04 LTS. I picked up mhddfs by doing sudo apt-get install mhddfs to continue the same excellent mhddfs+snapraid setup that I have been using for the past few years to manage my storage – you can find details here. Everything seemed to be working fine but today morning, I found that all my samba shares on the server had vanished!
I tried to figure out what was going on and found that the mhddfs mount point was no longer accessible – cd /storage showed the error The transport endpoint is not connected.
I quickly did ps auxw | grep mhddfs and saw that mhddfs was no longer running. /var/log/kern.log showed that mhddfs had quit with a segfault – uh-oh!
mhddfs[11861]: segfault at 0 ip 0000000000404750 sp 00007f1100ff8a40 error 4 in mhddfs[400000+b000]
A quick search for “mhddfs transport endpoint is not connected” lead me to this q&a and in turn to this fix for the issue here. While this is a fine fix, I was intrigued to figure out why the crash was happening in the first place – the fact that it started from 1.39 and that it seems to be fixed by replacing the uthash code with a different implementation narrowed down the places to look a lot.
Looking at this commit that was the only change from 1.38 to 1.39, I didn’t find anything obviously wrong at first. But then I decided to take a closer look at HASH_CLEAR and it struck me that the issue is a use after free – the code in question here would access the node after it had been free’d due to the call to HASH_CLEAR.
I’ve forked the github repo for mhddfs and patched it with a fix – running that for the past few hours and so far everything seems to work fine. Here’s the link to the fix and here’s the pre-built binary for Ubuntu Server 14.04 LTS
I’ll update this post in a couple of weeks on if/how my patch is working.
Update 9/27/2015 My home server has been running mhddfs with this change without issues for 5 days now.
Update 10/11/2015 My home server has been running mhddfs with this change without issues for 20 days now.
Could you provide info on how to deploy your patch?
Run
which mhddfsto find out where the mhddfs binary is on your system and replace it with the pre-built binary. Reboot your system – probably the easiest way to do it.FYI – I installed your git repo for mhhdfs on my gentoo box and had the following error appear within a few minutes during an rsync.
” Transport endpoint is not connected (107) “
Thanks for reporting and sorry to hear that. Did you compile from sources or did you install the binary? FWIW, it is running stably for me for over a month now.
Has this been merged into the main branch yet?
OK, this might be a little too further down the road, but I was wondering how did it eventually work out after all this time.
It worked great and absolutely no issues. However, because I was bored and wanted to try something new + the fact that it is actively developed and supported, I am now using mergerfs (https://github.com/trapexit/mergerfs) instead of mhddfs. It also has the advantage of handling permission issues (which I could never get quite right with mhddfs) well. Overall, I would say drop mhddfs and move to mergerfs.