"fatal: Malformed file chunk for " error on git fetch

[k88hudson]

I've been getting an error when fetching unified with git cinnabar since yesterday afternoon (I've never seen this problem before fetching multiple times a day). This is what my output looks like:

$ git fetch
Reading 448 changesets
Reading and importing 448 manifests
Reading and importing 804 revisions of 505 filesfatal: Malformed file chunk for 9b8505e01cdcd8bbfad474066d694eba07b21175
fast-import: dumping crash report to .git/fast_import_crash_55116
Reading and importing 981 revisions of 628 files
Traceback (most recent call last):
  File "/Users/mozilla/.mozbuild/git-cinnabar/cinnabar/util.py", line 707, in run
    retcode = func(sys.argv[1:])
  File "/Users/mozilla/.mozbuild/git-cinnabar/git-remote-hg", line 33, in main
    helper.run()
  File "/Users/mozilla/.mozbuild/git-cinnabar/cinnabar/remote_helper.py", line 88, in run
    func(*args)
  File "/Users/mozilla/.mozbuild/git-cinnabar/cinnabar/remote_helper.py", line 347, in import_
    self._branchmap.names())
  File "/Users/mozilla/.mozbuild/git-cinnabar/cinnabar/hg/repo.py", line 788, in getbundle
    apply_bundle(store)
  File "/Users/mozilla/.mozbuild/git-cinnabar/cinnabar/hg/repo.py", line 701, in __call__
    GitHgHelper.store('file', rev_chunk)
  File "/Users/mozilla/.mozbuild/git-cinnabar/cinnabar/helper.py", line 271, in store
    self._helper.stdin.write(obj)
IOError: [Errno 32] Broken pipe
Error in atexit._run_exitfuncs:
Traceback (most recent call last):
  File "/usr/local/Cellar/python@2/2.7.15_1/Frameworks/Python.framework/Versions/2.7/lib/python2.7/atexit.py", line 24, in _run_exitfuncs
    func(*targs, **kargs)
  File "/Users/mozilla/.mozbuild/git-cinnabar/cinnabar/helper.py", line 392, in close
    with self.query('done'):
  File "/usr/local/Cellar/python@2/2.7.15_1/Frameworks/Python.framework/Versions/2.7/lib/python2.7/contextlib.py", line 17, in __enter__
    return self.gen.next()
  File "/Users/mozilla/.mozbuild/git-cinnabar/cinnabar/helper.py", line 108, in query
    wrapper.write('%s\n' % name)
IOError: [Errno 32] Broken pipe
Error in sys.exitfunc:
Traceback (most recent call last):
  File "/usr/local/Cellar/python@2/2.7.15_1/Frameworks/Python.framework/Versions/2.7/lib/python2.7/atexit.py", line 24, in _run_exitfuncs
    func(*targs, **kargs)
  File "/Users/mozilla/.mozbuild/git-cinnabar/cinnabar/helper.py", line 392, in close
    with self.query('done'):
  File "/usr/local/Cellar/python@2/2.7.15_1/Frameworks/Python.framework/Versions/2.7/lib/python2.7/contextlib.py", line 17, in __enter__
    return self.gen.next()
  File "/Users/mozilla/.mozbuild/git-cinnabar/cinnabar/helper.py", line 108, in query
    wrapper.write('%s\n' % name)
IOError: [Errno 32] Broken pipe
fatal: could not read ref refs/cinnabar/refs/heads/bookmarks/beta

This is my git config:

credential.helper=osxkeychain
cinnabar.helper=/Users/mozilla/.mozbuild/git-cinnabar/git-cinnabar-helper
user.name=k88hudson
[email protected]
core.editor=code --wait
filter.lfs.clean=git-lfs clean -- %f
filter.lfs.smudge=git-lfs smudge -- %f
filter.lfs.process=git-lfs filter-process
filter.lfs.required=true
merge.conflictstyle=diff3
core.repositoryformatversion=0
core.filemode=true
core.bare=false
core.logallrefupdates=true
core.ignorecase=true
core.precomposeunicode=true
core.untrackedcache=true
remote.origin.url=hg::https://hg.mozilla.org/mozilla-unified
remote.origin.fetch=+refs/heads/*:refs/remotes/origin/*
branch.branches/default/tip.remote=origin
branch.branches/default/tip.merge=refs/heads/branches/default/tip
branch.bookmarks/central.remote=origin
branch.bookmarks/central.merge=refs/heads/bookmarks/central
fetch.prune=true
remote.try.url=hg::https://hg.mozilla.org/try
remote.try.fetch=+refs/heads/*:refs/remotes/try/*
remote.try.skipdefaultupdate=true
remote.try.pushurl=hg::ssh://hg.mozilla.org/try
remote.try.push=+HEAD:refs/heads/branches/default/tip
remote.inbound.url=hg::ssh://hg.mozilla.org/integration/mozilla-inbound
remote.inbound.fetch=+refs/heads/*:refs/remotes/inbound/*
remote.inbound.skipdefaultupdate=true
remote.inbound.push=+HEAD:refs/heads/branches/default/tip
branch.bookmarks/beta.remote=origin
branch.bookmarks/beta.merge=refs/heads/bookmarks/beta
remote.autoland.url=hg::ssh://hg.mozilla.org/integration/autoland
remote.autoland.fetch=+refs/heads/*:refs/remotes/autoland/*
branch.bookmarks/release.remote=origin
branch.bookmarks/release.merge=refs/heads/bookmarks/release

Any help would be highly appreciated. Thanks!

[glandium]

I've tried to create a clone of mozilla-unified on my mac with the same core.* config items, and I didn't reproduce a broken repo. I'm really interested in getting an archive of that broken .git of yours.

[glandium]

Thanks again for the archive of your .git.

As mentioned on irc, https://hg.mozilla.org/mozilla-unified/rev/be98a5ed931b is when your clone ended up definitely broken, and I was able to reproduce using the state of your clone before that, but it turned out the underpinning of the breakage appeared a few days earlier, when pulling https://hg.mozilla.org/mozilla-unified/rev/1692e4521fb0 and https://hg.mozilla.org/mozilla-unified/rev/f94486bc39182d98646991f8. What happened is that some files were renamed, and the mercurial metadata of those renames were not stored during the pull.

Incidentally, another file rename occurred just shortly before that, and was recorded properly. It was on pulling https://hg.mozilla.org/mozilla-unified/rev/9deb23f8b29b.

Considering the contents of your clone back then, I know with certainty that you were using at least git-cinnabar commit 514b404, but that one works for me.

As mentioned on irc, it would be useful to figure out what version of git-cinnabar you were using back then, and with chance, it's available in the reflog in your git-cinnabar clone. Can you check the output from git reflog show --date=iso and see what commit you might have been on back at the beginning of November?

[k88hudson]

For reference:

• currently using a clone in ~/.mozbuild/git-cinnabar at d5443ce (2018-09-28)
• I have another one at 1144ded (2018-09-27)

[glandium]

I tried all combinations of git-cinnabar and helper versions possible in the range where your repository works (that's about 10k combinations), with a pull of the same heads that made your clone break, and none of them reproduced the issue. :(

[glandium]

The root cause has still not been identified. #214 is another occurrence of the same problem, and I wasn't able to reproduce it with the exact version of git-cinnabar and the helper that were used there. My plan of action at this point is, essentially:

• Release 0.5.1 (soon) with some improvements to git cinnabar fsck and some nagging to run it (once a week, operations that change metadata will ask if you've run fsck recently).
• Going forward, I want to do more improvements to git cinnabar fsck such that it can run automatically to catch the error as it happens, and possibly gather more actionable information (like a copy of the mercurial bundle that was returned by the server)