You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
I am observing data in my S3 bucket (AWS) inconsistent with content on my Nextcloud server. See repro steps for more details, I did came across this issue nextcloud/server#41917 which may or may not be related, my symptoms are slightly different and this is not a new issue - I observed it for two months, I was waiting for nextcloud/fulltextsearch_elasticsearch#326 to be fixed before reporting this on the off chance it was related.
Steps to reproduce
Mount S3 external folder.
Create a new blank file test.md via NextCloud UI, click save, exit the editor.
Look into Details -> Versions, there will be Initial version and Current version, both 0 B.
Look on S3 - you will also see two distinct versions, the first will always be Standard storage class and second (current) will be Intelligent-Tiering - not sure if this is AWS or NextCloud doing, I would expect them both be Intelligent-Tiering since my bucket is configured for it. This is a first sign of trouble, that there are two versions from get-go and that one version is a wrong storage class.
Open test.md in UI, add first line content, save the file, exit the editor.
Observe in Details -> Version that there is now a third version Current version correctly opening. The first two empty versions do not open, they are presenting a loading spinner forever and nothing pops up in the logs.
Observe on the S3 bucket that you have a new third (current) version with a correct content of first line.
Open test.md in UI, add second line content, save the file, exit the editor.
Observe in Details -> Version that there is now a new version, with correct new content.
Observe on the S3 bucket that you have a new (current) version with a correct content.
Open test.md in UI, add third line content, save the file, exit the editor.
Observe in Details -> Version that there is now a new version, with correct new content.
Observe on the S3 bucket that you still have old version as current with only two lines, and third line nowhere to be seen. I left it like that overnight thinking it may eventually sync, but it didn't. No errors in logs.
Adding more changes will trigger sync again. It is some kind of a race condition, sometimes I get inconsistent data on odd lines, sometimes - on even, sometimes it just works. Sometimes it is missing to create local versions too.
Additional details
I did not notice causality between how fast I am making changes and inconsistencies, it may occur on singular isolated edits. Although I did notice correlation, it is occurring more often on fast edits, which is a typical use case for a Markdown KB (add a thought, save, edit, save etc - and you never know if your latest edit actually made it to S3).
You may notice under disabled apps I have files_versions_s3 - I tried to install it to see if it makes any difference, it didn't. It was having this behavior before I ever installed it.
Expected behavior
Expected to see clear linear version history both in NextCloud and S3, and most importantly - expecting to always see latest and consistent content on S3. In my home use scenario, S3 should be the most reliable storage for most important documents and KB, because my home lab hardware may always be stolen or destroyed, and traditional scheduled backups are never latest. This bug is preventing me from fully trusting NextCloud over Google Drive. I may never be sure that my off-site data is consistent.
Installation method
Official All-in-One appliance
Nextcloud Server version
27
Operating system
Debian/Ubuntu
PHP engine version
PHP 8.1
Web server
Other
Database engine version
PostgreSQL
Is this bug present after an update or on a fresh install?
Fresh Nextcloud Server install
Are you using the Nextcloud Server Encryption module?
Looks like maybe #4823 (fixed by #4824) and/or #3404 (fixed by #4286). The former should have been released with NC V27.1.4 (it was backported in #4968), but I see no version bump in the shipped Text app (from v3.8.0) in a stock NC v27.1.4 installation so maybe something got messed up there. I don't think #4286 was backported, but it should be IMO.
So, in any case, this may be a Text app only issue, but until confirmed I'll leave here in this repo (plus it's a shipped app so we definitely want it to work well!)
Hmmm it didn't occur to me that the issue may be with UI not with S3 storage. Indeed both local and S3 versions are getting messed up in the end. Let me test this with a) md files on local disk and b) files other than md files.
Ok yes, I can definitely reproduce the same with SMB external storage as well. #3404 gave me the key - it is increasingly likely to reproduce if I keep making changes and saves without exiting text editor.
Doesn't look like I can reproduce it on S3 external storage by uploading new versions of files via UI, or editing it on my machine mounted as webdav. Good news I don't have to worry about integrity of my non-text files. Bad news, my KB might be corrupted as much as in my own head lol
joshtrichards
changed the title
[Bug]: S3 external storage is not consistent
[Bug]: Text app saving/versioning is not consistent
Dec 2, 2023
Bug description
I am observing data in my S3 bucket (AWS) inconsistent with content on my Nextcloud server. See repro steps for more details, I did came across this issue nextcloud/server#41917 which may or may not be related, my symptoms are slightly different and this is not a new issue - I observed it for two months, I was waiting for nextcloud/fulltextsearch_elasticsearch#326 to be fixed before reporting this on the off chance it was related.
Steps to reproduce
test.md
via NextCloud UI, click save, exit the editor.Initial version
andCurrent version
, both0 B
.Standard
storage class and second (current) will beIntelligent-Tiering
- not sure if this is AWS or NextCloud doing, I would expect them both beIntelligent-Tiering
since my bucket is configured for it. This is a first sign of trouble, that there are two versions from get-go and that one version is a wrong storage class.test.md
in UI, addfirst line
content, save the file, exit the editor.Details
->Version
that there is now a third versionCurrent version
correctly opening. The first two empty versions do not open, they are presenting a loading spinner forever and nothing pops up in the logs.test.md
in UI, addsecond line
content, save the file, exit the editor.Details
->Version
that there is now a new version, with correct new content.test.md
in UI, addthird line
content, save the file, exit the editor.Details
->Version
that there is now a new version, with correct new content.Additional details
I did not notice causality between how fast I am making changes and inconsistencies, it may occur on singular isolated edits. Although I did notice correlation, it is occurring more often on fast edits, which is a typical use case for a Markdown KB (add a thought, save, edit, save etc - and you never know if your latest edit actually made it to S3).
You may notice under disabled apps I have
files_versions_s3
- I tried to install it to see if it makes any difference, it didn't. It was having this behavior before I ever installed it.Expected behavior
Expected to see clear linear version history both in NextCloud and S3, and most importantly - expecting to always see latest and consistent content on S3. In my home use scenario, S3 should be the most reliable storage for most important documents and KB, because my home lab hardware may always be stolen or destroyed, and traditional scheduled backups are never latest. This bug is preventing me from fully trusting NextCloud over Google Drive. I may never be sure that my off-site data is consistent.
Installation method
Official All-in-One appliance
Nextcloud Server version
27
Operating system
Debian/Ubuntu
PHP engine version
PHP 8.1
Web server
Other
Database engine version
PostgreSQL
Is this bug present after an update or on a fresh install?
Fresh Nextcloud Server install
Are you using the Nextcloud Server Encryption module?
Encryption is Disabled
What user-backends are you using?
Configuration report
List of activated Apps
# docker exec --user www-data nextcloud-aio-nextcloud php occ app:list Enabled: - activity: 2.19.0 - admin_audit: 1.17.0 - announcementcenter: 6.7.0 - app_api: 1.3.0 - calendar: 4.5.3 - cfg_share_links: 4.2.0 - checksum: 1.2.2 - circles: 27.0.1 - cloud_federation_api: 1.10.0 - comments: 1.17.0 - contacts: 5.4.2 - dashboard: 7.7.0 - dav: 1.27.0 - deck: 1.11.2 - federatedfilesharing: 1.17.0 - federation: 1.17.0 - files: 1.22.0 - files_antivirus: 5.4.0 - files_archive: 1.1.3 - files_external: 1.19.0 - files_fulltextsearch: 27.0.1 - files_pdfviewer: 2.8.0 - files_reminders: 1.0.0 - files_rightclick: 1.6.0 - files_sharing: 1.19.0 - files_trashbin: 1.17.0 - files_versions: 1.20.0 - firstrunwizard: 2.16.0 - fulltextsearch: 27.0.2 - fulltextsearch_elasticsearch: 27.0.5 - groupfolders: 15.3.1 - imageconverter: 1.3.5 - integration_giphy: 1.0.8 - integration_google: 2.1.0 - logreader: 2.12.0 - lookup_server_connector: 1.15.0 - metadata: 0.19.0 - nextcloud-aio: 0.4.0 - nextcloud_announcements: 1.16.0 - notes: 4.8.1 - notifications: 2.15.0 - notify_push: 0.6.5 - oauth2: 1.15.1 - oidc: 0.7.1 - password_policy: 1.17.0 - photos: 2.3.0 - privacy: 1.11.0 - provisioning_api: 1.17.0 - recommendations: 1.6.0 - related_resources: 1.2.0 - richdocuments: 8.2.3 - secrets: 1.5.2 - serverinfo: 1.17.0 - settings: 1.9.0 - sharebymail: 1.17.0 - side_menu: 3.11.2 - socialsharing_email: 3.0.0 - spreed: 17.1.3 - support: 1.10.0 - survey_client: 1.15.0 - systemtags: 1.17.0 - tasks: 0.15.0 - text: 3.8.0 - theming: 2.2.0 - twofactor_backupcodes: 1.16.0 - twofactor_nextcloud_notification: 3.8.0 - twofactor_totp: 9.0.0 - twofactor_webauthn: 1.3.2 - user_status: 1.7.0 - viewer: 2.1.0 - weather_status: 1.7.0 - workflowengine: 2.9.0 Disabled: - bruteforcesettings: 2.7.0 - contactsinteraction: 1.8.0 (installed 1.8.0) - encryption: 2.15.0 - files_versions_s3: 0.2.1 (installed 0.2.1) - suspicious_login: 5.0.0 - user_ldap: 1.17.0
Nextcloud Signing status
Nextcloud Logs
No response
Additional info
No response
The text was updated successfully, but these errors were encountered: