If you have any spare hard drives lying around and you’re a data squirrel like me, you’ve looked at the Drobo or Windows Home Server (WHS) with envy. \u00a0Drobo implements a file system \u00a0called BeyondFS, which lets you add and swap hard drives at will; the system will reconfigure itself depending in the space available and the number of hard disks installed, implementing mirroring when it can without regard to the actual size of the hard disks installed. \u00a0WHS can do the same, on a file level. While these are sweet high-tech toys, they are not free. Even after moving here and recycling dumping the worst of my old hardware, I have plenty left just lying around. \u00a0And I hoard data faster than I can hoard money. \u00a0Is there a way to build\u00a0a Drobo with old hardware and free software?<\/p>\n
I’ve looked at many solutions such as\u00a0 unRaid and Freenas, \u00a0but they need dedicated servers and I wanted to run on my old Ubuntu server, since it also runs my VMs. I’ve also looked at Btrfs, believing that being a freshly minted file system it would support a logical volume manager internally (or at least intelligently manage volumes of different sizes). But it doesn’t; the current load balancer is very basic. It supports Raid0, Raid1 and Raid10. And for the moment, at least (as of 2.6.35), it is limited by the smallest volume in the Raid array (so you get the same capacity as in Raid1 mirroring, I find Raid5 more appealing). \u00a0In the process I’ve discovered that VMWare really doesn’t like running on top of Btrfs – suspending my VMs would leave them in an unrecoverable state with no helpful error message or log entry.<\/p>\n
In the end, I went with LVM on top of Raid5. \u00a0Although it is not as easy to understand and configure as a Drobo or WHS, I can add and remove hard disks at will (provided I leave enough for the Raid5) and grow or shrink the file system even while it is live. And I can run VMWare on that Ubuntu server without a hitch.<\/p>\n
As far as I can tell the only other free option would be to use ZFS, but that would mean switching to FreeBSD (and VirtualBox for my VMs). \u00a0I guess I could take another old box out of storage and muck around. \u00a0However as far as I can tell – correct me if I’m wrong! – once you define a ZFS array you cannot yet add devices, only replace disks with bigger disks, which is not as flexible as I want. So I’m not ready to invest too much time in the attempt.<\/p>\n
Now, LVM+Raid5 isn’t as simple to configure and use as a Drobo or WHS, but it works, has decent performance, and is totally free. You might have some trouble getting all the commands right the first time you define it and you’ll wish it had a working GUI, but once I got the configuration right, it worked like a charm. \u00a0Even adding a device and growing the volume, while a multi-step process, proved easy enough. \u00a0Best of all, since LVM takes charge of load balancing on the array, you can add and remove disks WHILE it is in use. Unfortunately, there is still one thing that it doesn’t do; manage different-sized disks without waste. \u00a0Installed on top of a raid 5 array, the usage of any disk is limited to the smallest partition in the array (to get the final capacity, just ignore one of your partitions when adding up). However you can add as many partitions as you like to grow the array (Taking care not to use two partitions on the same disk).<\/p>\n
Thus I ended up with a LVM on RAID5 array, \u00a0almost easily expanded to infinity and with some data safety. When Btrfs implements Raid5 (possibly in 2.6.39) and you can do an on-line re-balance, it will be worth another look, as you would not need LVM. The file system would grow without a problem as long as you do add one volume at a time.<\/p>\n
All this exploration has left me feeling that there’s a need for a file system that is space-conscious and redundant (for select files). \u00a0Would it be possible to use WrapFS to do this? \u00a0Is anyone aware of a project or file system that already does this?<\/p>\n
So for now it looks like I can’t make a DIY Drobo, but at least I got to experience the magic of LVM.<\/p>\n","protected":false},"excerpt":{"rendered":"
In which I try to find a Drobo or WHS-like file system and fail gracefully<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[5],"tags":[13,14,18,31],"class_list":["post-53","post","type-post","status-publish","format-standard","hentry","category-software-engineering","tag-btrfs","tag-diy","tag-file-system","tag-raid"],"aioseo_notices":[],"_links":{"self":[{"href":"http:\/\/enigmachine.com\/wpress\/wp-json\/wp\/v2\/posts\/53","targetHints":{"allow":["GET"]}}],"collection":[{"href":"http:\/\/enigmachine.com\/wpress\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"http:\/\/enigmachine.com\/wpress\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"http:\/\/enigmachine.com\/wpress\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"http:\/\/enigmachine.com\/wpress\/wp-json\/wp\/v2\/comments?post=53"}],"version-history":[{"count":0,"href":"http:\/\/enigmachine.com\/wpress\/wp-json\/wp\/v2\/posts\/53\/revisions"}],"wp:attachment":[{"href":"http:\/\/enigmachine.com\/wpress\/wp-json\/wp\/v2\/media?parent=53"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"http:\/\/enigmachine.com\/wpress\/wp-json\/wp\/v2\/categories?post=53"},{"taxonomy":"post_tag","embeddable":true,"href":"http:\/\/enigmachine.com\/wpress\/wp-json\/wp\/v2\/tags?post=53"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}