files in subdirectory not synced, if subdirectory shared with different user than directory

[tessus]

Steps to reproduce

1. create directory A
2. create sub-directory B in A
3. share A (can edit: create/update/delete) with group X
4. share B (can edit: create/update/delete) with user Y

Expected behaviour

If user Y uploads a file into B, it should be synced.

Actual behaviour

If user Y uploads a file into B, people in group X can see the file in the web interface, but it is not synced to the clients.

If a user from the group X unchecks B in the 'Choose What to Sync' dialog, clicks 'Ok' and checks it again, the files in B are synced. This has to be done every time user Y uploads a file in B.

Server configuration

• ownCloud 7.0.4
• Linux
• Apache httpd 2.4.10
• PHP 5.6.4
• no external storage
• no encryption
• no apps in use except the ones activated by default

Client configuration

• MacOSX 10.9.5 and 10.10.1
• client 1.7.1 (build 1655)

[karlitschek]

@schiesbn

[PVince81]

I suspect that the etag either doesn't change or the etag change isn't propagated properly.

[PVince81]

also CC @icewind1991 for etag propagation

[tessus]

Any update on this?

Is there a chance of data loss? What happens, if user Y re-uploads a file into B and a user from group X does the same thing. The group X never had the new file and changed an old version of the file...

Can you direct me to the code that has to do with etags and propagation? Maybe I can find something.

[PVince81]

I don't think there will be data loss. In the worst case the sync client will create a conflict file (which is a copy of the local/remote file (I forgot which one))

Etag propagation is all over the place in core, so not easy to track.

[PVince81]

@tessus is user Y in group X ? Ill assume you meant they are separate

[PVince81]

I had a quick test with cadaver and could see that when user Y uploads into "b", the etag of "b" and "a" properly changed, which should normally be detected by the sync clients.

But to be sure I'll check with the sync client as well.

[PVince81]

To make it clearer (in case you want to try it out too), here are my steps:

1. Login as "root" (the admin)
2. Create a user "u1" (no group)
3. Create a user "u2" and assign that user to a new group "g1"
4. Create a folder "a" and subfolder "b"
5. Share "a" with "g1"
6. Share "a/b" with "u1"
7. Start cadaver (CLI webdav client) for "u1"
8. Check properties of "b"

dav:/owncloud/remote.php/webdav/> propget b
Fetching properties for `b':
http://owncloud.org/ns id = 00000009oc11qbmm0afb
http://owncloud.org/ns permissions = SRDNVCK
DAV: getetag = "54c8fbfd2f039"
DAV: getlastmodified = Wed, 28 Jan 2015 15:10:53 GMT
DAV: resourcetype = <DAV:collection></DAV:collection>
DAV: quota-used-bytes = 0
DAV: quota-available-bytes = 1073647616

1. Start another cadaver session for "u2".
2. Do a propget on "a" and "b":

dav:/owncloud/remote.php/webdav/> propget a
Fetching properties for `a':
http://owncloud.org/ns id = 00000008oc11qbmm0afb
http://owncloud.org/ns permissions = SRDNVCK
DAV: getetag = "54c8fbfd2d26d"
DAV: getlastmodified = Wed, 28 Jan 2015 15:10:53 GMT
DAV: resourcetype = <DAV:collection></DAV:collection>
DAV: quota-used-bytes = 0
DAV: quota-available-bytes = 1073647616
dav:/owncloud/remote.php/webdav/> propget a/b
Fetching properties for `a/b':
http://owncloud.org/ns id = 00000009oc11qbmm0afb
http://owncloud.org/ns permissions = SRDNVCK
DAV: getetag = "54c8fbfd2f039"
DAV: getlastmodified = Wed, 28 Jan 2015 15:10:53 GMT
DAV: resourcetype = <DAV:collection></DAV:collection>
DAV: quota-used-bytes = 0
DAV: quota-available-bytes = 1073647616

1. Go back to the session for "u1" then upload a file into "b":

dav:/owncloud/remote.php/webdav/> cd b
dav:/owncloud/remote.php/webdav/b/> put test.txt
Uploading test.txt to `/owncloud/remote.php/webdav/b/test.txt':
Progress: [=============================>] 100.0% of 163 bytes succeeded.

1. With the session from "u2", do again a propget on "a" and "b":

dav:/owncloud/remote.php/webdav/> propget a
Fetching properties for `a':
http://owncloud.org/ns id = 00000008oc11qbmm0afb
http://owncloud.org/ns permissions = SRDNVCK
DAV: getetag = "54c8fc439171d"
DAV: getlastmodified = Wed, 28 Jan 2015 15:12:03 GMT
DAV: resourcetype = <DAV:collection></DAV:collection>
DAV: quota-used-bytes = 163
DAV: quota-available-bytes = 1073639424
dav:/owncloud/remote.php/webdav/> propget a/b
Fetching properties for `a/b':
http://owncloud.org/ns id = 00000009oc11qbmm0afb
http://owncloud.org/ns permissions = SRDNVCK
DAV: getetag = "54c8fc439fcbd"
DAV: getlastmodified = Wed, 28 Jan 2015 15:12:03 GMT
DAV: resourcetype = <DAV:collection></DAV:collection>
DAV: quota-used-bytes = 163
DAV: quota-available-bytes = 1073639424

From what I see the etag changed properly.
You might want to double check the setup to see if it matches and maybe try the same steps, then tell us whether the etag changes for you or not.

I'll try with the sync client next.

[PVince81]

Ok, now I set up a sync client for "u2" (the group member).

I kept the cadaver session of "u1" and then uploaded another file "test2.txt" there.
After a few seconds, the sync client successfully downloads "a/b/test2.txt".

This is all on OC 7.0.4.

Can you tell us how the sync client of "u2" is configured ? I simply used "Sync everything from server" in the wizard. Maybe that user picked something else which might affect syncing behavior ?

[tessus]

@PVince81 Thanks for the info. Here are my answers and some ideas:

• user Y (u1) is not in group X (g1)
• all my sync clients use "Sync everything from server"
• I don't know, if it makes any difference, but user Y uploads with the web interface (no sync client or webdav)
• user (u2) in group X (g1) runs 2 sync clients on different machines

I noticed something strange and not quite reproducable: sometimes the file user u1 uploaded is synced to one of the clients of user u2. so machine A receives the new file and machine B doesn't.
This could help narrow it down: I think that the client which used the uncheck/check workaround last (which I described in the actual behavior), is the one who receives subsequent updates.

So maybe this bug is actually a bit more complex and only occurs, if more than 1 sync client is used on different machines for the same user in the group.

[PVince81]

Hmmm, can you describe the setup of these two sync clients ? Is one running on Windows and the other one on Mac ?

[tessus]

Nope, both (1.7.1 build 1655) are running on MacOSX 10.9.5 and both are setup to sync the entire directory structure.

Edit: Nope was the answer to your second question. Sure. is the answer to your first.

[tessus]

I think I have additional information that could help sorting this out:

I couldn't keep this setup any longer, because I needed my data to be up-to-date.

So I basically moved directory b to the same level as a. In course of this new setup I noticed that this problem might not have to do with subfolders, but with the web upload.
It seems that if somebody uploads a file using the web interface, the changes are not propagated, thus not synced.

[avoine]

I suspect this bug is still valid because I have the same bug here with ownCloud 7.0.4 and owncloud-client 1.7.1 on Ubuntu.
I suspect an etag problem too because like @PVince81 said as soon
as you modify something in the subfolder the changes are propagated and it became visible on the client.

[PVince81]

@avoine could you try the steps like I did at #13333 (comment) and tell me whether the result is different for you ?

We still need more information about how to reproduce the issue.

[tessus]

@PVince81 I think the problem has to do with uploading via the web interface.
At least I haven't seen this problem before when all people involved used the sync client.

[tessus]

@PVince81 it seems that this also happens, if you upload new files via the sync client. it doesn't happen all the time, but most of the time. you upload something with one client, and the other client never receives these files.

is there any update on this? I believe this is rather a serious issue. if you don't get new files or work with older versions. this might lead to data loss.

[PVince81]

Unfortunately no update. If we had a case where we could reproduce this consistently, then we might be able to debug it.

Did you try running the steps from #13333 (comment) ? At least it would help confirm that something is wrong with etag propagation on your setup.

[tessus]

Nope, I haven't tried that, because I'm using MacOSX as clients. But I can try to install cadaver from MacPorts. Also, I don't have the other user's password, so I can't really do that.

But right now I have the following issue with the same user, so if you tell me what you need, I can get it for you.

• one user
• uploaded 2 files via sync client on machine A to directory X
• 2 files are visible via web client
• sync client on machine B does not sync these 2 files

[PVince81]

Hmmm, so this happens even when it's not shared files ?

What you could try:

1. Login as that user with cadaver
2. mkdir test to create a folder test
3. Do a propget . (root) and also propget test and write down the etags
4. Start the sync client for that user
5. Put a file into "test" locally and wait for sync, the sync client will upload it
6. Back to cadaver, do again a propget . and propget test. At this point the etag of the root and the folder should have changed.

If not, then we need to find out why.

[tessus]

Yes, they have changed, but I will also run this test on my other machine at home tonight.

Anything I can do find out why I'm not having the 2 files I mentioned earlier on this machine (although clearly available via web)? Can I force a new etag creation in a certain directory or on certain files?

dav:/remote.php/webdav/> mkdir test
Creating `test': succeeded.
dav:/remote.php/webdav/> propget .
Fetching properties for `.':
id = 00000118oc2352d53521
permissions = RDNVCK
getetag = "54eb744db023e"
getlastmodified = Mon, 23 Feb 2015 18:41:17 GMT
resourcetype = <DAV:collection></DAV:collection>
quota-used-bytes = 504523666
quota-available-bytes = 10232894574
dav:/remote.php/webdav/> propget test
Fetching properties for `test':
id = 00002974oc2352d53521
permissions = RDNVCK
getetag = "54eb744d887c0"
getlastmodified = Mon, 23 Feb 2015 18:41:17 GMT
resourcetype = <DAV:collection></DAV:collection>
quota-used-bytes = 0
quota-available-bytes = 10232894574
dav:/remote.php/webdav/> propget .
Fetching properties for `.':
id = 00000118oc2352d53521
permissions = RDNVCK
getetag = "54eb750c8b19e"
getlastmodified = Tue, 10 Feb 2015 19:29:10 GMT
resourcetype = <DAV:collection></DAV:collection>
quota-used-bytes = 537232599
quota-available-bytes = 10200185641
dav:/remote.php/webdav/> propget test
Fetching properties for `test':
id = 00002974oc2352d53521
permissions = RDNVCK
getetag = "54eb750c91df6"
getlastmodified = Tue, 10 Feb 2015 19:29:10 GMT
resourcetype = <DAV:collection></DAV:collection>
quota-used-bytes = 32708933
quota-available-bytes = 10200185641

[PVince81]

I see both the etag and mtime changed, so the next time the sync client runs it should catch these changes.
Also not sure about your two files that are not downloaded.

One possibly explanation is that maybe the sync client runs into an exception before reaching that part, or it runs into a timeout during the discovery phase. Did the sync client show any error in the activity log ? You could check the sync client log for that run (see https://forum.owncloud.org/viewtopic.php?f=17&t=6526)

[tessus]

Hmm, my sync clients run all the time. I'm not sure, if it make sense to restart them with a logfile option after the fact.
But next time I see that behavior I restart them with the logfile option and post the log.

[PVince81]

Do you mean the issue isn't happening consistently, so that restarting the sync clients might eliminate the issue ?

[tessus]

Yes, it isn't happening consistently, but when it happens, restarting the sync client doesn't help.

[PVince81]

We might be onto something here with etag propagation: #14596

What we observed above was that the etag was propagated up to the owner's folder, which is correct. But in #14596 I observed that the etag is not propagated up to the root, which may or may not cause the issue you're seeing.

[tessus]

Ah, very nice. This looks promising. At least it's worth a try to test.

Is there a fix I can test on my server? A few lines I can change? (I still use 7.0.4, because I saw a bug that involves upgrade issues, which might only be fixed in 8.1-next).

[PVince81]

My theory is that if the shared folder is deeper than 1-2 levels, the etag will not propagate high enough.
I haven't looked at the code yet, still evaluating the scope of the bug.

[PVince81]

Confirmed.

If the shared folder is in the owner's tree here:

• "level1/shared": if recipient uploads to "shared", it will sync fine
• "level1/level2/shared": will only update level2's etag. Will not sync.

Please move over to #14596 and take a seat.

[tessus]

This is awesome. This explains why our earlier scenario to reproduce the issue didn't work. My directories where I experienced this problem were indeed several levels down.

Moving over to #14596, taking a front row seat. Argh, where's my popcorn.

[PVince81]

Etag issue was moved here: #14726

#14596 was for size propagation. They're kind of separate.

Sorry to move your seat to another room 😉

[tessus]

No problem, I have taken a seat there already.

Btw, I'm happy to test any fixes on my 7.0.4 server.