From 57f35cfd7c6e9c7f7ab59e650280e09f85618e33 Mon Sep 17 00:00:00 2001 From: Taylor Blau Date: Thu, 14 Nov 2024 08:42:09 -0500 Subject: [PATCH 1/2] t5332-multi-pack-reuse.sh: demonstrate duplicate packing failure In the multi-pack reuse code, there are two paths for reusing the on-disk representation of an object, handled by: - builtin/pack-objects.c::write_reused_pack_one() - builtin/pack-objects.c::write_reused_pack_verbatim() The former is responsible for copying the bytes for a single object out of an existing source pack. The latter does the same but for a region of objects aligned at eword_t boundaries. Demonstrate a bug whereby write_reused_pack_verbatim() can be tricked into writing out objects from some source pack, even when those objects were selected from a different source pack in the MIDX bitmap. When the caller wants at least one of the objects in that region, pack-objects will write the same object twice as a result of this bug. In the other case where the caller doesn't want any of the objects in the region of interest, we will write out objects that weren't requested. Demonstrate this bug by creating two packs, where the preferred one of those packs contains a single object which also appears in the main (non-preferred) pack. A separate bug[^1] prevents us from triggering the main bug when the duplicated object is the last one in the main pack, but any earlier object will suffice. We could fix that separate bug, but the following commit will simplify write_reused_pack_verbatim() and only call it on the preferred pack, so doing so would have little point. [^1]: Because write_reused_pack_verbatim() only reuses bits in the range off_t pack_start_off = pack_pos_to_offset(reuse_packfile->p, 0); off_t pack_end_off = pack_pos_to_offset(reuse_packfile->p, pos - reuse_packfile->bitmap_pos); written += pos - reuse_packfile->bitmap_pos; /* We're recording one chunk, not one object. */ record_reused_object(pack_start_off, pack_start_off - (hashfile_total(out) - pack_start)); , or in other words excluding the object beginning at position 'pos - reuse_packfile->bitmap_pos' in the source pack. But since reuse_packfile->bitmap_pos is '1' in the non-preferred pack (accounting for the single-object pack which is preferred), we don't actually copy the bytes from the last object. Helped-by: Jeff King Signed-off-by: Taylor Blau Signed-off-by: Junio C Hamano --- t/t5332-multi-pack-reuse.sh | 22 ++++++++++++++++++++++ 1 file changed, 22 insertions(+) diff --git a/t/t5332-multi-pack-reuse.sh b/t/t5332-multi-pack-reuse.sh index 955ea42769bc0e..d87ea0ae19bccc 100755 --- a/t/t5332-multi-pack-reuse.sh +++ b/t/t5332-multi-pack-reuse.sh @@ -259,4 +259,26 @@ test_expect_success 'duplicate objects' ' ) ' +test_expect_failure 'duplicate objects with verbatim reuse' ' + git init duplicate-objects-verbatim && + ( + cd duplicate-objects-verbatim && + + git config pack.allowPackReuse multi && + + test_commit_bulk 64 && + + # take the first object from the main pack... + git show-index <$(ls $packdir/pack-*.idx) >obj.raw && + sort -nk1 in && + + # ...and create a separate pack containing just that object + p="$(git pack-objects $packdir/pack Date: Thu, 14 Nov 2024 08:42:12 -0500 Subject: [PATCH 2/2] pack-objects: only perform verbatim reuse on the preferred pack When reusing objects from source pack(s), write_reused_pack_verbatim() is responsible for reusing objects whole eword_t's at a time. It works by taking the longest continuous run of objects from the beginning of each source pack that the caller wants, and reuses the entirety of that section from each pack. This is based on the assumption that we don't have any gaps within the region. This assumption relieves us from having to patch any OFS_DELTAs, since we know that there aren't any gaps between any delta and its base in that region. To illustrate why this assumption is necessary, suppose we have some pack P, which has objects X, Y, and Z. If the MIDX's copy of Y was selected from a pack other than P, then the bit corresponding to object Y will appear earlier in the bitmap than the bits corresponding to X and Z. If pack-objects already has or will use the copy of Y from the pack it was selected from in the MIDX, then it is an error to reuse all objects between X and Z in the source pack. Doing so will cause us to reuse Y from a different pack than the one which represents Y in the MIDX, causing us to either: - include the object twice, assuming that the caller wants Y in the pack, or - include the object once, resulting in us packing more objects than necessary. This regression comes from ca0fd69e37 (pack-objects: prepare `write_reused_pack_verbatim()` for multi-pack reuse, 2023-12-14), which incorrectly assumed that there would be no gaps in reusable regions of non-preferred packs. Instead, we can only safely perform the whole-word reuse optimization on the preferred pack, where we know with certainty that no gaps exist in that region of the bitmap. We can still reuse objects from non-preferred packs, but we have to inspect them individually in write_reused_pack() to ensure that any gaps that may exist are accounted for. This allows us to simplify the implementation of write_reused_pack_verbatim() back to almost its pre-multi-pack reuse form, since we can now assume that the beginning of the pack appears at the beginning of the bitmap, meaning that we don't have to account for any bits up to the first word boundary (like we had to special case in ca0fd69e37). The only significant changes from the pre-ca0fd69e37 implementation are: - that we can no longer inspect words up to the end of reuse_packfile_bitmap->word_alloc, since we only want to look at words whose bits all correspond to objects in the given packfile, and - that we return early when given a reuse_packfile which is not preferred, making the call a noop. In the future, it might be possible to restore this optimization if we could guarantee that some reuse packs don't contain any gaps by construction (similar to the "disjoint packs" idea in very early versions of multi-pack reuse). Helped-by: Jeff King Signed-off-by: Taylor Blau Signed-off-by: Junio C Hamano --- builtin/pack-objects.c | 95 ++++++++++++++++--------------------- t/t5332-multi-pack-reuse.sh | 2 +- 2 files changed, 41 insertions(+), 56 deletions(-) diff --git a/builtin/pack-objects.c b/builtin/pack-objects.c index 0fc0680b40252a..7cbed8b256a2d5 100644 --- a/builtin/pack-objects.c +++ b/builtin/pack-objects.c @@ -1100,78 +1100,64 @@ static void write_reused_pack_one(struct packed_git *reuse_packfile, static size_t write_reused_pack_verbatim(struct bitmapped_pack *reuse_packfile, struct hashfile *out, - off_t pack_start, struct pack_window **w_curs) { - size_t pos = reuse_packfile->bitmap_pos; + size_t pos = 0; size_t end; - if (pos % BITS_IN_EWORD) { - size_t word_pos = (pos / BITS_IN_EWORD); - size_t offset = pos % BITS_IN_EWORD; - size_t last; - eword_t word = reuse_packfile_bitmap->words[word_pos]; - - if (offset + reuse_packfile->bitmap_nr < BITS_IN_EWORD) - last = offset + reuse_packfile->bitmap_nr; - else - last = BITS_IN_EWORD; - - for (; offset < last; offset++) { - if (word >> offset == 0) - return word_pos; - if (!bitmap_get(reuse_packfile_bitmap, - word_pos * BITS_IN_EWORD + offset)) - return word_pos; - } - - pos += BITS_IN_EWORD - (pos % BITS_IN_EWORD); + if (reuse_packfile->bitmap_pos) { + /* + * We can't reuse whole chunks verbatim out of + * non-preferred packs since we can't guarantee that + * all duplicate objects were resolved in favor of + * that pack. + * + * Even if we have a whole eword_t worth of bits that + * could be reused, there may be objects between the + * objects corresponding to the first and last bit of + * that word which were selected from a different + * pack, causing us to send duplicate or unwanted + * objects. + * + * Handle non-preferred packs from within + * write_reused_pack(), which inspects and reuses + * individual bits. + */ + return reuse_packfile->bitmap_pos / BITS_IN_EWORD; } /* - * Now we're going to copy as many whole eword_t's as possible. - * "end" is the index of the last whole eword_t we copy, but - * there may be additional bits to process. Those are handled - * individually by write_reused_pack(). + * Only read through the last word whose bits all correspond + * to objects in the given packfile, since we must stop at a + * word boundary. * - * Begin by advancing to the first word boundary in range of the - * bit positions occupied by objects in "reuse_packfile". Then - * pick the last word boundary in the same range. If we have at - * least one word's worth of bits to process, continue on. + * If there is no whole word to read (i.e. the packfile + * contains fewer than BITS_IN_EWORD objects), then we'll + * inspect bits one-by-one in write_reused_pack(). */ - end = reuse_packfile->bitmap_pos + reuse_packfile->bitmap_nr; - if (end % BITS_IN_EWORD) - end -= end % BITS_IN_EWORD; - if (pos >= end) - return reuse_packfile->bitmap_pos / BITS_IN_EWORD; - - while (pos < end && - reuse_packfile_bitmap->words[pos / BITS_IN_EWORD] == (eword_t)~0) - pos += BITS_IN_EWORD; + end = reuse_packfile->bitmap_nr / BITS_IN_EWORD; + if (reuse_packfile_bitmap->word_alloc < end) + BUG("fewer words than expected in reuse_packfile_bitmap"); - if (pos > end) - pos = end; + while (pos < end && reuse_packfile_bitmap->words[pos] == (eword_t)~0) + pos++; - if (reuse_packfile->bitmap_pos < pos) { - off_t pack_start_off = pack_pos_to_offset(reuse_packfile->p, 0); - off_t pack_end_off = pack_pos_to_offset(reuse_packfile->p, - pos - reuse_packfile->bitmap_pos); + if (pos) { + off_t to_write; - written += pos - reuse_packfile->bitmap_pos; + written = (pos * BITS_IN_EWORD); + to_write = pack_pos_to_offset(reuse_packfile->p, written) + - sizeof(struct pack_header); /* We're recording one chunk, not one object. */ - record_reused_object(pack_start_off, - pack_start_off - (hashfile_total(out) - pack_start)); + record_reused_object(sizeof(struct pack_header), 0); hashflush(out); copy_pack_data(out, reuse_packfile->p, w_curs, - pack_start_off, pack_end_off - pack_start_off); + sizeof(struct pack_header), to_write); display_progress(progress_state, written); } - if (pos % BITS_IN_EWORD) - BUG("attempted to jump past a word boundary to %"PRIuMAX, - (uintmax_t)pos); - return pos / BITS_IN_EWORD; + return pos; } static void write_reused_pack(struct bitmapped_pack *reuse_packfile, @@ -1183,8 +1169,7 @@ static void write_reused_pack(struct bitmapped_pack *reuse_packfile, struct pack_window *w_curs = NULL; if (allow_ofs_delta) - i = write_reused_pack_verbatim(reuse_packfile, f, pack_start, - &w_curs); + i = write_reused_pack_verbatim(reuse_packfile, f, &w_curs); for (; i < reuse_packfile_bitmap->word_alloc; ++i) { eword_t word = reuse_packfile_bitmap->words[i]; diff --git a/t/t5332-multi-pack-reuse.sh b/t/t5332-multi-pack-reuse.sh index d87ea0ae19bccc..e9d80186ec6eca 100755 --- a/t/t5332-multi-pack-reuse.sh +++ b/t/t5332-multi-pack-reuse.sh @@ -259,7 +259,7 @@ test_expect_success 'duplicate objects' ' ) ' -test_expect_failure 'duplicate objects with verbatim reuse' ' +test_expect_success 'duplicate objects with verbatim reuse' ' git init duplicate-objects-verbatim && ( cd duplicate-objects-verbatim &&