After a long period of contemplation and indecision, I upgraded the hardware on my Ubuntu Home Server last weekend. I’ll write a separate post about my adventures there (mostly self-inflicted).
As part of the upgrade, I also clean installed Ubuntu Server 14.04 LTS. I picked up mhddfs by doing sudo apt-get install mhddfs to continue the same excellent mhddfs+snapraid setup that I have been using for the past few years to manage my storage – you can find details here. Everything seemed to be working fine but today morning, I found that all my samba shares on the server had vanished!
I tried to figure out what was going on and found that the mhddfs mount point was no longer accessible – cd /storage showed the error The transport endpoint is not connected.
I quickly did ps auxw | grep mhddfs and saw that mhddfs was no longer running. /var/log/kern.log showed that mhddfs had quit with a segfault – uh-oh!
mhddfs[11861]: segfault at 0 ip 0000000000404750 sp 00007f1100ff8a40 error 4 in mhddfs[400000+b000]
A quick search for “mhddfs transport endpoint is not connected” lead me to this q&a and in turn to this fix for the issue here. While this is a fine fix, I was intrigued to figure out why the crash was happening in the first place – the fact that it started from 1.39 and that it seems to be fixed by replacing the uthash code with a different implementation narrowed down the places to look a lot.
Looking at this commit that was the only change from 1.38 to 1.39, I didn’t find anything obviously wrong at first. But then I decided to take a closer look at HASH_CLEAR and it struck me that the issue is a use after free – the code in question here would access the node after it had been free’d due to the call to HASH_CLEAR.
I’ve forked the github repo for mhddfs and patched it with a fix – running that for the past few hours and so far everything seems to work fine. Here’s the link to the fix and here’s the pre-built binary for Ubuntu Server 14.04 LTS
I’ll update this post in a couple of weeks on if/how my patch is working.
Update 9/27/2015 My home server has been running mhddfs with this change without issues for 5 days now.
Update 10/11/2015 My home server has been running mhddfs with this change without issues for 20 days now.