Skip to content

Commit

Permalink
Merge pull request #290 from trapexit/epff
Browse files Browse the repository at this point in the history
add existing path first found policy. closes #289
  • Loading branch information
trapexit authored Jul 12, 2016
2 parents a375f33 + a93ab6c commit 920f7fd
Show file tree
Hide file tree
Showing 5 changed files with 168 additions and 3 deletions.
5 changes: 3 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -86,6 +86,7 @@ Due to FUSE limitations **ioctl** behaves differently if its acting on a directo
| Policy | Description |
|--------------|-------------|
| all | Search category: acts like **ff**. Action category: apply to all found. Create category: for **mkdir**, **mknod**, and **symlink** it will apply to all found. **create** works like **ff**. It will exclude readonly drives and those with free space less than **minfreespace**. |
| epff | Given the order of the drives, as defined at mount time or when configured via the xattr interface, act on the first one found where the path already exists. For **create** cateogry it will exclude readonly drives and those with free space less than **minfreespace** (unless there is no other option). Falls back to **ff**. |
| eplfs (existing path, least free space) | If the path exists on multiple drives use the one with the least free space. For **create** category it will exclude readonly drives and those with free space less than **minfreespace**. Falls back to **lfs**. |
| eplus (existing path, least used space) | If the path exists on multiple drives the the one with the least used space. For **create** category it will exclude readonly drives and those with free space less than **minfreespace**. Falls back to **lus**. |
| epmfs (existing path, most free space) | If the path exists on multiple drives use the one with the most free space. For **create** category it will exclude readonly drives and those with free space less than **minfreespace**. Falls back to **mfs**. |
Expand All @@ -97,7 +98,7 @@ Due to FUSE limitations **ioctl** behaves differently if its acting on a directo
| newest (newest file) | Pick the file / directory with the largest mtime. For **create** category it will exclude readonly drives and those with free space less than **minfreespace** (unless there is no other option). |
| rand (random) | Calls **all** and then randomizes. |

**eplfs**, **eplus**, and **epmf** are path preserving policies. As the descriptions above explain they will only consider drives where the path being accessed exists. Non-path preserving policies will clone paths as necessary.
**epff**, **eplfs**, **eplus**, and **epmf** are path preserving policies. As the descriptions above explain they will only consider drives where the path being accessed exists. Non-path preserving policies will clone paths as necessary.

#### Defaults ####

Expand All @@ -113,7 +114,7 @@ Due to FUSE limitations **ioctl** behaves differently if its acting on a directo

Originally mergerfs would return EXDEV whenever a rename was requested which was cross directory in any way. This made the code simple and was technically complient with POSIX requirements. However, many applications fail to handle EXDEV at all and treat it as a normal error or they only partially support EXDEV (don't respond the same as `mv` would). Such apps include: gvfsd-fuse v1.20.3 and prior, Finder / CIFS/SMB client in Apple OSX 10.9+, NZBGet, Samba's recycling bin feature.

* If using a **create** policy which tries to preserve directory paths (epmfs,eplfs)
* If using a **create** policy which tries to preserve directory paths (epff,eplfs,eplus,epmfs)
* Using the **rename** policy get the list of files to rename
* For each file attempt rename:
* If failure with ENOENT run **create** policy
Expand Down
30 changes: 29 additions & 1 deletion man/mergerfs.1
Original file line number Diff line number Diff line change
Expand Up @@ -203,6 +203,17 @@ It will exclude readonly drives and those with free space less than
\f[B]minfreespace\f[].
T}
T{
epff
T}@T{
Given the order of the drives, as defined at mount time or when
configured via the xattr interface, act on the first one found where the
path already exists.
For \f[B]create\f[] cateogry it will exclude readonly drives and those
with free space less than \f[B]minfreespace\f[] (unless there is no
other option).
Falls back to \f[B]ff\f[].
T}
T{
eplfs (existing path, least free space)
T}@T{
If the path exists on multiple drives use the one with the least free
Expand Down Expand Up @@ -284,6 +295,12 @@ T}@T{
Calls \f[B]all\f[] and then randomizes.
T}
.TE
.PP
\f[B]epff\f[], \f[B]eplfs\f[], \f[B]eplus\f[], and \f[B]epmf\f[] are
path preserving policies.
As the descriptions above explain they will only consider drives where
the path being accessed exists.
Non\-path preserving policies will clone paths as necessary.
.SS Defaults
.PP
.TS
Expand Down Expand Up @@ -337,7 +354,7 @@ Such apps include: gvfsd\-fuse v1.20.3 and prior, Finder / CIFS/SMB
client in Apple OSX 10.9+, NZBGet, Samba\[aq]s recycling bin feature.
.IP \[bu] 2
If using a \f[B]create\f[] policy which tries to preserve directory
paths (epmfs,eplfs)
paths (epff,eplfs,eplus,epmfs)
.IP \[bu] 2
Using the \f[B]rename\f[] policy get the list of files to rename
.IP \[bu] 2
Expand Down Expand Up @@ -850,6 +867,17 @@ Drives can fail and all other data will continue to be accessable.
Yes.
It will be represented immediately in the pool as the policies would
describe.
.SS Why do I get an "out of space" error even though the system says
there\[aq]s lots of space left?
.PP
Please reread the sections above about policies, path preserving, and
the \f[B]moveonenospc\f[] option.
If the policy is path preserving and a drive is almost full and the
drive the policy would pick then the writing of the file may fill the
drive and receive ENOSPC errors.
That is expected with those settings.
If you don\[aq]t want that: enable \f[B]moveonenospc\f[] and don\[aq]t
use a path preserving policy.
.SS It\[aq]s mentioned that there are some security issues with mhddfs.
What are they? How does mergerfs address them?
.PP
Expand Down
2 changes: 2 additions & 0 deletions src/policy.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -30,6 +30,7 @@ namespace mergerfs
buildvector<Policy,true>
(POLICY(invalid,DOESNT_PRESERVE_PATH))
(POLICY(all,DOESNT_PRESERVE_PATH))
(POLICY(epff,PRESERVES_PATH))
(POLICY(eplfs,PRESERVES_PATH))
(POLICY(eplus,PRESERVES_PATH))
(POLICY(epmfs,PRESERVES_PATH))
Expand All @@ -47,6 +48,7 @@ namespace mergerfs

CONST_POLICY(invalid);
CONST_POLICY(all);
CONST_POLICY(epff);
CONST_POLICY(eplfs);
CONST_POLICY(eplus);
CONST_POLICY(epmfs);
Expand Down
3 changes: 3 additions & 0 deletions src/policy.hpp
Original file line number Diff line number Diff line change
Expand Up @@ -43,6 +43,7 @@ namespace mergerfs
invalid = -1,
BEGIN = 0,
all = BEGIN,
epff,
eplfs,
eplus,
epmfs,
Expand Down Expand Up @@ -96,6 +97,7 @@ namespace mergerfs

static int invalid(CType,cstrvec&,const char *,cuint64_t,cstrptrvec&);
static int all(CType,cstrvec&,const char*,cuint64_t,cstrptrvec&);
static int epff(CType,cstrvec&,const char *,cuint64_t,cstrptrvec&);
static int eplfs(CType,cstrvec&,const char *,cuint64_t,cstrptrvec&);
static int eplus(CType,cstrvec&,const char *,cuint64_t,cstrptrvec&);
static int epmfs(CType,cstrvec&,const char *,cuint64_t,cstrptrvec&);
Expand Down Expand Up @@ -169,6 +171,7 @@ namespace mergerfs

static const Policy &invalid;
static const Policy &all;
static const Policy &epff;
static const Policy &eplfs;
static const Policy &eplus;
static const Policy &epmfs;
Expand Down
131 changes: 131 additions & 0 deletions src/policy_epff.cpp
Original file line number Diff line number Diff line change
@@ -0,0 +1,131 @@
/*
Copyright (c) 2016, Antonio SJ Musumeci <[email protected]>
Permission to use, copy, modify, and/or distribute this software for any
purpose with or without fee is hereby granted, provided that the above
copyright notice and this permission notice appear in all copies.
THE SOFTWARE IS PROVIDED "AS IS" AND THE AUTHOR DISCLAIMS ALL WARRANTIES
WITH REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED WARRANTIES OF
MERCHANTABILITY AND FITNESS. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR
ANY SPECIAL, DIRECT, INDIRECT, OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES
WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN AN
ACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF
OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE.
*/

#include <errno.h>

#include <string>
#include <vector>

#include "fs.hpp"
#include "fs_path.hpp"
#include "policy.hpp"

using std::string;
using std::vector;
using mergerfs::Category;

static
int
_epff_create(const vector<string> &basepaths,
const char *fusepath,
const uint64_t minfreespace,
vector<const string*> &paths)
{
string fullpath;
const string *fallback;

fallback = NULL;
for(size_t i = 0, ei = basepaths.size(); i != ei; i++)
{
bool readonly;
uint64_t spaceavail;
uint64_t _spaceused;
const string *basepath = &basepaths[i];

fs::path::make(basepath,fusepath,fullpath);

if(!fs::exists(fullpath))
continue;
if(!fs::info(*basepath,readonly,spaceavail,_spaceused))
continue;
if(readonly)
continue;
if(fallback == NULL)
fallback = basepath;
if(spaceavail < minfreespace)
continue;

paths.push_back(basepath);

return POLICY_SUCCESS;
}

if(fallback == NULL)
return POLICY_FAIL_ENOENT;

paths.push_back(fallback);

return POLICY_SUCCESS;
}

static
int
_epff_other(const vector<string> &basepaths,
const char *fusepath,
vector<const string*> &paths)
{
string fullpath;

for(size_t i = 0, ei = basepaths.size(); i != ei; i++)
{
const string *basepath = &basepaths[i];

fs::path::make(basepath,fusepath,fullpath);

if(!fs::exists(fullpath))
continue;

paths.push_back(basepath);

return POLICY_SUCCESS;
}

return POLICY_FAIL_ENOENT;
}

static
int
_epff(const Category::Enum::Type type,
const vector<string> &basepaths,
const char *fusepath,
const uint64_t minfreespace,
vector<const string*> &paths)
{
if(type == Category::Enum::create)
return _epff_create(basepaths,fusepath,minfreespace,paths);

return _epff_other(basepaths,fusepath,paths);
}


namespace mergerfs
{
int
Policy::Func::epff(const Category::Enum::Type type,
const vector<string> &basepaths,
const char *fusepath,
const uint64_t minfreespace,
vector<const string*> &paths)
{
int rv;

rv = _epff(type,basepaths,fusepath,minfreespace,paths);
if(POLICY_FAILED(rv))
rv = Policy::Func::ff(type,basepaths,fusepath,minfreespace,paths);

return rv;
}
}

0 comments on commit 920f7fd

Please sign in to comment.