writer worker caught exception: 10003 Cannot change the size of a document in a capped collection

[raffis]

MongoDB replset slave nodes may crash due a MongoDB bug in capped collections replsets:

There are some workarounds/maybe issue solver:

• It's happening while changing ended timestamp => try to init ended with new UTCDateTime() (not 0)
• Place taskscheduler in the local database (no replset sync),

 2018-12-21T18:54:41.777+0100 I - [repl writer worker 14] Fatal assertion 16359 CannotGrowDocumentInCappedNamespace: Cannot change the size of a document in a capped collection: 71 != 86 at src/mongo/db/repl/sync_tail.cpp 1087
2019-01-21T08:43:21.477+0100 I STORAGE [initandlisten] exception in initAndListen: 10003 Cannot change the size of a document in a capped collection: 71 != 86, terminating
2019-01-21T18:37:32.433+0100 F REPL [repl writer worker 11] writer worker caught exception: 10003 Cannot change the size of a document in a capped collection: 71 != 86 on: { ts: Timestamp 1548092252000|2, h: 8891639960730045349, v: 2, op: "u", ns: "cloudfs.taskscheduler.jobs", o2:
{ _id: ObjectId('5c4592d8a7621c000a270723') }

, o: { $set:
{ status: 3, ended: new Date(1548092252319) }

} }
2019-01-21T18:37:32.433+0100 I - [repl writer worker 11] Fatal assertion 16359 CannotGrowDocumentInCappedNamespace: Cannot change the size of a document in a capped collection: 71 != 86 at src/mongo/db/repl/sync_tail.cpp 1087
2019-01-17T07:23:58.732+0100 F REPL [repl writer worker 6] writer worker caught exception: 10003 Cannot change the size of a document in a capped collection: 71 != 86 on: { ts: Timestamp 1547706238000|2, h: -1287491416624820144, v: 2, op: "u", ns: "cloudfs.taskscheduler.jobs", o2:
{ _id: ObjectId('5c3ecdfd81fa83000704e0f6') }

, o: { $set:
{ status: 3, ended: new Date(1547706238359) }

} }
2019-01-17T07:23:58.770+0100 I - [repl writer worker 6] Fatal assertion 16359 CannotGrowDocumentInCappedNamespace: Cannot change the size of a document in a capped collection: 71 != 86 at src/mongo/db/repl/sync_tail.cpp 1087
2019-01-17T07:52:46.610+0100 I STORAGE [initandlisten] exception in initAndListen: 10003 Cannot change the size of a document in a capped collection: 71 != 86, terminating
2019-01-18T10:36:02.219+0100 F REPL [repl writer worker 0] writer worker caught exception: 10003 Cannot change the size of a document in a capped collection: 71 != 86 on: { ts: Timestamp 1547804162000|2, h: -9155582286729571842, v: 2, op: "u", ns: "cloudfs.taskscheduler.jobs", o2:
{ _id: ObjectId('5c404c817047201a535de75f') }

, o: { $set:
{ status: 3, ended: new Date(1547804162166) }

} }
2019-01-18T10:36:02.219+0100 I - [repl writer worker 0] Fatal assertion 16359 CannotGrowDocumentInCappedNamespace: Cannot change the size of a document in a capped collection: 71 != 86 at src/mongo/db/repl/sync_tail.cpp 1087
2019-01-19T10:36:03.302+0100 F REPL [repl writer worker 13] writer worker caught exception: 10003 Cannot change the size of a document in a capped collection: 71 != 86 on: { ts: Timestamp 1547890562000|3, h: -9125484584779178179, v: 2, op: "u", ns: "cloudfs.taskscheduler.jobs", o2:
{ _id: ObjectId('5c419e02863a75000b7a54b9') }

, o: { $set:
{ status: 3, ended: new Date(1547890562963) }

} }
2019-01-19T10:36:03.315+0100 I - [repl writer worker 13] Fatal assertion 16359 CannotGrowDocumentInCappedNamespace: Cannot change the size of a document in a capped collection: 71 != 86 at src/mongo/db/repl/sync_tail.cpp 1087

[raffis]

https://jira.mongodb.org/browse/SERVER-38724?page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel&focusedCommentId=2126837#comment-2126837

[raffis]

The only useful solution if this happens is to put the taskscheduler collections into the local db.
May need to migrate to changestreams in v4.