Sharing etag propagation tests #136

jvillafanez · 2015-11-18T14:16:48Z

Only for users, includes resharing and uploads to several places

Based on owncloud/core#14764 (comment)

I have to improve the error messages. Any ideas are welcomed.

There are a couple of potential issues detected (if the tests are correct):

unsharing doesn't change the etag of the share owner
uploading a file inside a reshare doesn't propagate the etag change to the original owner. Owner shares to recip1, recip1 shares to recip2, recip2 uploads a file, owner doesn't get the change

@nickvergessen @DeepDiver1975 @PVince81

Only for users, includes resharing and upload to several places

PVince81 · 2015-11-18T14:21:32Z

CC @icewind1991

nickvergessen · 2015-11-18T15:01:03Z

Looking quickly at it, seems fine and seems like it's doing the job.

PVince81 · 2015-11-18T16:11:36Z

unsharing doesn't change the etag of the share owner

This is expected. There is no actual file change to be synced for the owner.

uploading a file inside a reshare doesn't propagate the etag change to the original owner. Owner shares to recip1, recip1 shares to recip2, recip2 uploads a file, owner doesn't get the change

Hmm, could be a bug. @icewind1991 is that the part where the owner needs to relogin ? (we haven't figured out whether it was a login hook or setup hook)

PVince81 · 2015-11-18T16:19:40Z

I ran this against master and @icewind1991's new branch owncloud/core#20439 that reworks etag propagation. Both have the same issues.

master:

2015-11-18 17:16:47,878 - ERROR - owner - etag for / previous ["564ca46f3a263"] new ["564ca46f3a263"] 'etag for / previous [%s] new [%s]' % (root_etag2, root_etag3)) failed in owner() ["/home/vincent/work/workspace/smashbox/lib/oc-tests/test_sharePropagationInside.py" at line 111]
2015-11-18 17:16:50,382 - ERROR - owner - etag for / previous ["564ca47184423"] new ["564ca47184423"] 'etag for / previous [%s] new [%s]' % (root_etag4, root_etag5)) failed in owner() ["/home/vincent/work/workspace/smashbox/lib/oc-tests/test_sharePropagationInside.py" at line 129]
2015-11-18 17:16:50,383 - ERROR - owner - etag for /test previous ["564ca47184d39"] new ["564ca47184d39"] 'etag for /test previous [%s] new [%s]' % (test_etag2, test_etag3)) failed in owner() ["/home/vincent/work/workspace/smashbox/lib/oc-tests/test_sharePropagationInside.py" at line 131]
2015-11-18 17:16:50,779 - ERROR - owner - etag for / previous ["564ca47184423"] new ["564ca47184423"] 'etag for / previous [%s] new [%s]' % (root_etag5, root_etag6)) failed in owner() ["/home/vincent/work/workspace/smashbox/lib/oc-tests/test_sharePropagationInside.py" at line 137]
2015-11-18 17:16:50,780 - ERROR - owner - etag for /test previous ["564ca47184d39"] new ["564ca47184d39"] 'etag for /test previous [%s] new [%s]' % (test_etag3, test_etag4)) failed in owner() ["/home/vincent/work/workspace/smashbox/lib/oc-tests/test_sharePropagationInside.py" at line 139]
2015-11-18 17:16:52,154 - ERROR - owner - 5 error(s) reported

the branch:

2015-11-18 17:18:14,867 - ERROR - owner - etag for / previous ["e82ccee4f225c2753ffffa6525cdcd85"] new ["e82ccee4f225c2753ffffa6525cdcd85"] 'etag for / previous [%s] new [%s]' % (root_etag2, root_etag3)) failed in owner() ["/home/vincent/work/workspace/smashbox/lib/oc-tests/test_sharePropagationInside.py" at line 111]
2015-11-18 17:18:17,269 - ERROR - owner - etag for / previous ["524974106ea5b46e8bb7058bfecbc62a"] new ["524974106ea5b46e8bb7058bfecbc62a"] 'etag for / previous [%s] new [%s]' % (root_etag4, root_etag5)) failed in owner() ["/home/vincent/work/workspace/smashbox/lib/oc-tests/test_sharePropagationInside.py" at line 129]
2015-11-18 17:18:17,269 - ERROR - owner - etag for /test previous ["564ca4c86d3f5"] new ["564ca4c86d3f5"] 'etag for /test previous [%s] new [%s]' % (test_etag2, test_etag3)) failed in owner() ["/home/vincent/work/workspace/smashbox/lib/oc-tests/test_sharePropagationInside.py" at line 131]
2015-11-18 17:18:17,668 - ERROR - owner - etag for / previous ["524974106ea5b46e8bb7058bfecbc62a"] new ["524974106ea5b46e8bb7058bfecbc62a"] 'etag for / previous [%s] new [%s]' % (root_etag5, root_etag6)) failed in owner() ["/home/vincent/work/workspace/smashbox/lib/oc-tests/test_sharePropagationInside.py" at line 137]
2015-11-18 17:18:17,668 - ERROR - owner - etag for /test previous ["564ca4c86d3f5"] new ["564ca4c86d3f5"] 'etag for /test previous [%s] new [%s]' % (test_etag3, test_etag4)) failed in owner() ["/home/vincent/work/workspace/smashbox/lib/oc-tests/test_sharePropagationInside.py" at line 139]
2015-11-18 17:18:18,923 - ERROR - owner - 5 error(s) reported

DeepDiver1975 · 2015-11-18T16:30:24Z

@PVince81 @jvillafanez @nickvergessen how long is that test execution time? THX

PVince81 · 2015-11-18T16:35:15Z

it was 33 seconds on my laptop with SSD and smashdir in tmpfs.

icewind1991 · 2015-11-18T16:36:56Z

we haven't figured out whether it was a login hook or setup hook

FS setup

DeepDiver1975 · 2015-11-18T16:43:54Z

it was 33 seconds on my laptop with SSD and smashdir in tmpfs.

THX - candidate for jenkins ci job

jvillafanez · 2015-11-18T17:13:03Z

Unsharing now checks that the etag remains the same, so one of the points is fixed.

I've added a bit more information about the checks so it should be easier to know what the test is doing

The test_sharePropagation is passing in my environment, but the test_sharePropagationInside is failing

2015-11-18 17:57:06,518 - ERROR - owner - recipient2 uploads to /test/test3.txt etag for / previous ["564cade1c60e3"] new ["564cade1c60e3"] 'etag for / previous [%s] new [%s]' % (root_etag2, root_etag3)) failed in owner() ["/home/jvillafanez/src/smashbox/lib/oc-tests/test_sharePropagationInside.py" at line 104]
2015-11-18 17:57:10,062 - ERROR - owner - recipient 1 uploads to /test/sub/test5.txt etag for / previous ["564cade4f3f6c"] new ["564cade4f3f6c"] 'etag for / previous [%s] new [%s]' % (root_etag4, root_etag5)) failed in owner() ["/home/jvillafanez/src/smashbox/lib/oc-tests/test_sharePropagationInside.py" at line 122]
2015-11-18 17:57:10,064 - ERROR - owner - recipient 1 uploads to /test/sub/test5.txt etag for /test previous ["564cade507b77"] new ["564cade507b77"] 'etag for /test previous [%s] new [%s]' % (test_etag2, test_etag3)) failed in owner() ["/home/jvillafanez/src/smashbox/lib/oc-tests/test_sharePropagationInside.py" at line 124]
2015-11-18 17:57:10,504 - ERROR - owner - recipient 4 uploads to /sub/test6.txt through reshare etag for / previous ["564cade4f3f6c"] new ["564cade4f3f6c"] 'etag for / previous [%s] new [%s]' % (root_etag5, root_etag6)) failed in owner() ["/home/jvillafanez/src/smashbox/lib/oc-tests/test_sharePropagationInside.py" at line 130]
2015-11-18 17:57:10,508 - ERROR - owner - recipient 4 uploads to /sub/test6.txt through reshare etag for /test previous ["564cade507b77"] new ["564cade507b77"] 'etag for /test previous [%s] new [%s]' % (test_etag3, test_etag4)) failed in owner() ["/home/jvillafanez/src/smashbox/lib/oc-tests/test_sharePropagationInside.py" at line 132]
2015-11-18 17:57:12,103 - ERROR - owner - 5 error(s) reported
SUMMARY:smash.:Elapsed time: 79s (0:01:19.567585)

I'm checking the tests for errors, but I'm not finding anything weird in the tests

jvillafanez · 2015-11-18T17:33:06Z

Ok, it seems I forgot to add the workers 😶 Fixing....

jvillafanez · 2015-11-18T18:14:20Z

2015-11-18 18:51:29,672 - ERROR - recipient4 - recipient 1 uploads to /test/sub/test5.txt etag for / previous ["564cba9aac038"] new ["564cba9aac038"] 'etag for / previous [%s] new [%s]' % (root_etag4, root_etag5)) failed in recipient4() ["/home/jvillafanez/src/smashbox/lib/oc-tests/test_sharePropagationInside.py" at line 384]
2015-11-18 18:51:29,672 - ERROR - recipient4 - recipient 1 uploads to /test/sub/test5.txt etag for /sub previous ["564cba9a26736"] new ["564cba9a26736"] 'etag for /sub previous [%s] new [%s]' % (sub_etag2, sub_etag3)) failed in recipient4() ["/home/jvillafanez/src/smashbox/lib/oc-tests/test_sharePropagationInside.py" at line 386]
2015-11-18 18:51:37,062 - ERROR - recipient2 - recipient 2 unshares reshare etag for / previous ["564cbaa70703c"] new ["564cbaa715f5c"] 'etag for / previous [%s] new [%s]' % (root_etag6, root_etag7)) failed in recipient2() ["/home/jvillafanez/src/smashbox/lib/oc-tests/test_sharePropagationInside.py" at line 282]

@icewind1991 Could you confirm that the following cases (probably causing the above errors) have been taking into account?:

Case 1:

U1 creates /test/sub/
U1 shares /test to U2 and U3
U3 reshares /test/sub to U4
U2 uploads to /test/sub/test.txt

My guess: etag propagates to U2, U3 and U1 but doesn't propagate through the reshare in U3 to U4. It should go U2 -> U1 -> U3 -> U4

Case 2:

U1 creates /test/sub/
U1 shares /test to U2
U2 reshares /test/sub to U3
U2 unshares

The etag in /test for U2 remains the same but somehow the etag changes in /

PVince81 · 2015-11-20T11:52:16Z

@jvillafanez @icewind1991 does it happen also if you manually test with the web UI / Webdav ? (with cadaver you can do "propget")

icewind1991 · 2015-11-20T12:45:44Z

Can't reproduce case 2 on master manually, maybe propagation was triggered by something unrelated

icewind1991 · 2015-11-20T12:50:37Z

Can't reproduce case 1 on master manually, all root etags change after uploading to test/sub as u2

jvillafanez · 2015-11-20T12:58:42Z

For the first two errors, I was uploading the file to /test instead of /test/sub.... I'll recheck the last one.

Anyway, I'm going to change the uploads from the owncloudcmd to pyocclient, I'm having some errors as if the files weren't being uploaded.

nickvergessen · 2015-11-20T13:21:48Z

Anyway, I'm going to change the uploads from the owncloudcmd to pyocclient, I'm having some errors as if the files weren't being uploaded.

No, don't change it.
Just use an older client. The problem is the new client is bugged:
owncloud/client#4160

nickvergessen · 2015-11-20T13:22:22Z

For the moment you can add a sleep(3); after creating the files, but that should be removed before the branch is merged.

nickvergessen · 2015-11-20T13:23:09Z

Please move the tests to lib/owncloud/
That is where we put our tests, lib/oc-tests/ is from cern and should not be touched by us.

jvillafanez · 2015-11-20T14:34:07Z

I've followed @nickvergessen to add a sleep before syncing, tests are almost there.

What's left is the issue with the unshare. I think there might be something wrong with the test cleanup because first run works but then starts failing.

nickvergessen · 2015-11-20T15:26:33Z

lib/owncloud/test_sharePropagation.py

+
+    step(3, 'Upload file')
+    createfile(os.path.join(d, 'test', 'test.txt'), '1', count=1000, bs=10)
+    time.sleep(3)


owncloud/client#4160 the issue has been fixed. So you can remove it when you use a git clone of the client or wait for the next daily build tomorrow

I'd rather wait until 2.1 is out before removing the sleeps

well then we need to remember to remove it,
but I asked klaas for a beta2 soon, then we can remove it

icewind1991 · 2015-11-23T13:28:47Z

owncloud/core#20439 Fixes the unshare tests

icewind1991 · 2015-11-23T13:30:52Z

owncloud/core#20439 + added sleep makes everything pass for me

icewind1991 · 2015-11-23T15:07:20Z

@PVince81 @nickvergessen merge? any failing tests seem to be bugs in master and there's already a pending fix

nickvergessen · 2015-11-23T15:14:21Z

I don't really like merging with the sleep, client is fixed with daily builds and I requested a new beta.

But I guess we can still merge and just create a PR that removes all the sleeps and merge it, once beta2 is there

PVince81 · 2015-11-23T15:24:10Z

Does smashbox run against the stable* versions ? If it does, then the test will fail these where the (non-backportable) fix is missing.

nickvergessen · 2015-11-23T15:25:25Z

Yes it does, but we can teach our selves to ignore them on version x.y

PVince81 · 2015-11-23T15:25:42Z

👍

PVince81 · 2015-11-24T08:53:45Z

@jvillafanez please remove the sleep statements. The CI server uses an older working sync client version so the tests should run fine there already.

jvillafanez · 2015-11-24T09:48:55Z

Ready!

PVince81 · 2015-11-24T09:49:57Z

👍

nickvergessen · 2015-11-24T10:11:19Z

lib/owncloud/test_sharePropagation.py

+| 17          | verify etag is the same | verify etag is the same | verify propagation   |
+-------------+-------------------------+-------------------------+----------------------+
+
+Remove the sleep(x) once https://github.com/owncloud/client/issues/4160 has a resolution


PVince81 · 2015-11-24T16:03:00Z

@nickvergessen merge ?

nickvergessen · 2015-11-25T07:51:21Z

lib/owncloud/test_sharePropagation.py

+    check_users(4)
+
+    reset_rundir()
+    reset_server_log_file()


reset_rundir() reset_server_log_file()

can be removed, smashbox is automatically doing that before each test since end of september

nickvergessen · 2015-11-25T08:12:55Z

merge!

Sharing etag propagation tests

Sharing etag propagation tests

7a9acc9

Only for users, includes resharing and upload to several places

Fix unshare test to check it doesn't change the etag, improve errors

fb87c92

Add missing workers and fix unshare propagation tests

9c0b3d6

This was referenced Nov 19, 2015

Integration tests for etag propagation owncloud/core#20521

Closed

Take submount etag into account for folder etags owncloud/core#20439

Merged

Move tests to owncloud, add sleep before uploads and fix issue

0318c96

nickvergessen reviewed Nov 20, 2015
View reviewed changes

jvillafanez added 2 commits November 23, 2015 14:35

Fix locking issue (only upload one file instead of two)

9094cbf

Fix missing sleep

e06bf83

Remove extra user and remove sleeps

787926d

nickvergessen reviewed Nov 24, 2015
View reviewed changes

Remove messages that are no longer needed

0319dcf

jvillafanez mentioned this pull request Nov 24, 2015

Added tests for etag propagation sharing with groups #137

Merged

nickvergessen reviewed Nov 25, 2015
View reviewed changes

Make the tests as light weight as possible

79006cb

nickvergessen added a commit that referenced this pull request Nov 25, 2015

Merge pull request #136 from owncloud/share_propagation_tests

937dac2

Sharing etag propagation tests

nickvergessen merged commit 937dac2 into master Nov 25, 2015

nickvergessen deleted the share_propagation_tests branch November 25, 2015 08:12

nickvergessen added the TestCase label Nov 25, 2015

Sharing etag propagation tests #136

Sharing etag propagation tests #136

Conversation

jvillafanez commented Nov 18, 2015

PVince81 commented Nov 18, 2015

nickvergessen commented Nov 18, 2015

PVince81 commented Nov 18, 2015

PVince81 commented Nov 18, 2015

DeepDiver1975 commented Nov 18, 2015

PVince81 commented Nov 18, 2015

icewind1991 commented Nov 18, 2015

DeepDiver1975 commented Nov 18, 2015

jvillafanez commented Nov 18, 2015

jvillafanez commented Nov 18, 2015

jvillafanez commented Nov 18, 2015

PVince81 commented Nov 20, 2015

icewind1991 commented Nov 20, 2015

icewind1991 commented Nov 20, 2015

jvillafanez commented Nov 20, 2015

nickvergessen commented Nov 20, 2015

nickvergessen commented Nov 20, 2015

nickvergessen commented Nov 20, 2015

jvillafanez commented Nov 20, 2015

nickvergessen Nov 20, 2015

Choose a reason for hiding this comment

jvillafanez Nov 23, 2015

Choose a reason for hiding this comment

nickvergessen Nov 23, 2015

Choose a reason for hiding this comment

icewind1991 commented Nov 23, 2015

icewind1991 commented Nov 23, 2015

icewind1991 commented Nov 23, 2015

nickvergessen commented Nov 23, 2015

PVince81 commented Nov 23, 2015

nickvergessen commented Nov 23, 2015

PVince81 commented Nov 23, 2015

PVince81 commented Nov 24, 2015

jvillafanez commented Nov 24, 2015

PVince81 commented Nov 24, 2015

nickvergessen Nov 24, 2015

Choose a reason for hiding this comment

PVince81 commented Nov 24, 2015

nickvergessen Nov 25, 2015

Choose a reason for hiding this comment

nickvergessen commented Nov 25, 2015