ipfs add -r --nocopy Error: Failed to get block for zb2rh... The system cannot find the path specified.

[ivan386]

Version information:

go-ipfs version: 0.4.11-rc2-
Repo version: 6
System version: amd64/windows
Golang version: go1.9

Type: Bug

Severity: High

Description:

I try to add folder with --nocopy folder and get error.

Reproduce:

some_folder\test.txt

ipfs add -r -nocopy some_folder

rename "some_folder" to "some_folder_renamed"

ipfs add -r -nocopy some_folder_renamed

Error: Failed to get block for zb2rh...: open C:...\some_folder\test.txt: The system cannot find the path specified.

[whyrusleeping]

When adding content with the experimental filestore and the --nocopy option, moving files after you have added them will break things.

cc @kevina whats the recovery strategy here?

[kevina]

@whyrusleeping we don't have one, in the short to mid term I would like to see several things

1. Prove an --overwrite option that will replace existing filestore entries (I have talked about this elsewhere: Implement basic filestore 'no-copy' functionality #3629 (comment))
2. Merge filestore rm filestore rm #3815
3. Implement filestore clean to remove invalid entries

Note that (1) is useful because it can fix existing entries without having to worry about pinning. (2) is another way to manually clean up and is a prerequisite of (3).

In the long term I would like to allow multiple filestore entries per block. I have implemented this in my own implementation but will likely require a decent degree of work to get into your implementation.

[kevina]

Note that if --overright is implemented @ivan386 code will just work.

[NiKiZe]

Hitting this as well and it seems that you can;
ipfs pin rm hash
That might tell you it has a parent then unpin that.
And finally run ipfs repo gc, then rerun the add command.
(ipfs pin rm might not be needed?)

Is that correct? Is there any better way until filestore clean or filestore rm is available?

[ivan386]

Try filestore verify

[NiKiZe]

Try filestore verify

That does indeed remove the error
What I'm working on is ipfs/notes#296
And I expect this to be around 4-500GB of data, so running a full verify after each rsync will not really be acceptable. (we know this is all experimental but just mentioning the issue).

What I however did discover is that ipfs repo gc actually works just fine to resolve issues with missing files quite quickly. I don't understand how this does solve it, but it "does for me", Is it a bug that it works? is it causing more hidden issues that I have not yet seen?

Update: This only helps to get ipfs add to not error out, nothing does actually seen to be cleaned, not with verify either, It does seem to be an misunderstanding on my part, See ipfs/notes#296 (comment)

[Stebalien]

Make sure your files are pinned. If they aren't, a GC may fix this issue because it may delete references to the problematic blocks.

[ochilan]

Hello! I'm currently looking into using IPFS for publishing videos through DTube. Organizing my videos in the regular filesystem and adding them to IPFS using --nocopy seems to be a natural thing to me. Since it has been a year since plans to add functions like moving and removing files in filestore have popped up, I wanted to ask if there is any progress.

I also have two related questions:

1. Am I understanding correctly that a workaround is to "pin rm" the nonexistent hash and subsequently run a gc? In a quick test, this fixed the issue of not being able to re-add the file using a different filename.

2. If I choose to host some files on IPFS now, but not using filestore just yet, would it be wise to use "--raw-leaves" when adding files even when not using filestore? I'm asking because using raw leaves results in different hashes than without using them. If I migrate to filestore at a later point in time (which requires raw leaves), would the probability be greater that the hashes will stay the same? If not, migrating would require editing possibly many, many (DTube) postings to point to the correct hashes.