Shallow, blobless or treeless clone?

[sliekens]

Hello,

I want to optimize our CI build's git clone step to only clone what is needed for dotnet affected to build the affected project graph, what do you recommend? (https://github.blog/open-source/git/get-up-to-speed-with-partial-clone-and-shallow-clone/)

• shallow clone (depth < ...)
• blobless clone
• treeless clone

I tried to make it work with blobless clone, but it crashes with an exception:

Unhandled exception: LibGit2Sharp.NotFoundException: object not found - cannot read header for (3a6b...87)
   at LibGit2Sharp.Core.Ensure.HandleError(Int32 result) in /_/LibGit2Sharp/Core/Ensure.cs:line 154
   at LibGit2Sharp.Core.Ensure.ZeroResult(Int32 result) in /_/LibGit2Sharp/Core/Ensure.cs:line 172
   at LibGit2Sharp.Core.Proxy.git_diff_find_similar(DiffHandle diff, GitDiffFindOptions options) in /_/LibGit2Sharp/Core/Proxy.cs:line 843
   at LibGit2Sharp.Diff.DetectRenames(DiffHandle diffList, CompareOptions compareOptions) in /_/LibGit2Sharp/Diff.cs:line 586
   at LibGit2Sharp.Diff.BuildDiffList(ObjectId oldTreeId, ObjectId newTreeId, TreeComparisonHandleRetriever comparisonHandleRetriever, DiffModifiers diffOptions, IEnumerable`1 paths, ExplicitPathsOptions explicitPathsOptions, CompareOptions compareOptions) in /_/LibGit2Sharp/Diff.cs:line 575
   at LibGit2Sharp.Diff.Compare[T](Tree oldTree, Tree newTree, IEnumerable`1 paths, ExplicitPathsOptions explicitPathsOptions, CompareOptions compareOptions) in /_/LibGit2Sharp/Diff.cs:line 248
   at LibGit2Sharp.Diff.Compare[T](Tree oldTree, Tree newTree, IEnumerable`1 paths) in /_/LibGit2Sharp/Diff.cs:line 173
   at DotnetAffected.Core.GitChangesProvider.GetChangesBetweenTrees[T](Repository repository, Tree fromTree, Tree toTree, IEnumerable`1 files) in /_/src/DotnetAffected.Core/GitChangesProvider.cs:line 127
   at DotnetAffected.Core.GitChangesProvider.GetChangesForRange[T](Repository repository, String from, String to) in /_/src/DotnetAffected.Core/GitChangesProvider.cs:line 103
   at DotnetAffected.Core.GitChangesProvider.GetChangedFiles(String directory, String from, String to) in /_/src/DotnetAffected.Core/GitChangesProvider.cs:line 24
   at DotnetAffected.Core.Processor.AffectedProcessorBase.DiscoverChangedFiles(AffectedProcessorContext context) in /_/src/DotnetAffected.Core/Processor/AffectedProcessorBase.cs:line 57
   at DotnetAffected.Core.Processor.AffectedProcessorBase.Process(AffectedProcessorContext context) in /_/src/DotnetAffected.Core/Processor/AffectedProcessorBase.cs:line 22
   at DotnetAffected.Core.AffectedExecutor.Execute() in /_/src/DotnetAffected.Core/AffectedExecutor.cs:line 41
   at Affected.Cli.Commands.InvocationContextExtensions.ExecuteAffectedExecutor(InvocationContext ctx) in /_/src/dotnet-affected/Commands/Binding/InvocationContextExtensions.cs:line 38
   at Affected.Cli.Commands.AffectedRootCommand.<>c.<<-ctor>b__4_0>d.MoveNext() in /_/src/dotnet-affected/Commands/AffectedRootCommand.cs:line 39
--- End of stack trace from previous location ---
   at System.CommandLine.Invocation.AnonymousCommandHandler.InvokeAsync(InvocationContext context)
   at System.CommandLine.Invocation.InvocationPipeline.<>c__DisplayClass4_0.<<BuildInvocationChain>b__0>d.MoveNext()
--- End of stack trace from previous location ---
   at System.CommandLine.Builder.CommandLineBuilderExtensions.<>c__DisplayClass17_0.<<UseParseErrorReporting>b__0>d.MoveNext()
--- End of stack trace from previous location ---
   at Affected.Cli.ErrorHandlingMiddleware.<>c__DisplayClass0_0.<<UseRenderingErrorHandler>b__0>d.MoveNext() in /_/src/dotnet-affected/CommandLine/ErrorHandlingMiddleware.cs:line 26
--- End of stack trace from previous location ---
   at System.CommandLine.Builder.CommandLineBuilderExtensions.<>c__DisplayClass12_0.<<UseHelp>b__0>d.MoveNext()
--- End of stack trace from previous location ---
   at System.CommandLine.Builder.CommandLineBuilderExtensions.<>c__DisplayClass22_0.<<UseVersionOption>b__0>d.MoveNext()
--- End of stack trace from previous location ---
   at System.CommandLine.Builder.CommandLineBuilderExtensions.<>c__DisplayClass19_0.<<UseTypoCorrections>b__0>d.MoveNext()
--- End of stack trace from previous location ---
   at System.CommandLine.Builder.CommandLineBuilderExtensions.<>c.<<UseSuggestDirective>b__18_0>d.MoveNext()
--- End of stack trace from previous location ---
   at System.CommandLine.Builder.CommandLineBuilderExtensions.<>c__DisplayClass16_0.<<UseParseDirective>b__0>d.MoveNext()
--- End of stack trace from previous location ---
   at System.CommandLine.Builder.CommandLineBuilderExtensions.<>c.<<RegisterWithDotnetSuggest>b__5_0>d.MoveNext()
--- End of stack trace from previous location ---
   at System.CommandLine.Builder.CommandLineBuilderExtensions.<>c__DisplayClass8_0.<<UseExceptionHandler>b__0>d.MoveNext()

Thanks

[leonardochaia]

Hi @sliekens , I honestly don't know but would like to find out as well. Haven't read the full article you've liked yet but this what i know as of now:
What we do is mainly a git diff between the --from and --to arguments, alternatively if they are not provided we compare the current changes against HEAD. So we would need a clone statement that would allow us to do just that.

In CI we usually do a fetch-depth: 0 as recommended in the action's readme

[sliekens]

Hey @leonardochaia,

I think fetch-depth default is 1 (shallow clone)? While fetch-depth: 0 does a full clone. I guess the readme recommends 0 because the tool uses the history for checking the diffs. My guess is that a lower value like 100 would also work for small to medium size projects.

I'm interested in getting blobless or treeless clones to work, then the speed of fetch-depth: 0 is okay. In my repo, blobless clone with fetch-depth: 0 takes only 4 seconds, compared to 30 seconds for a full clone.

[leonardochaia]

Hmm I am honestly not sure but I am definitively interested in an answer 🤣
Biggest monorepo I have access to is hosted on Azure DevOps and we do not configure the checkout / fetch at all, so I assume ADO does something different than GH on that regard.

I honestly do not know if it is at all possible or not to make it work with blobless/treeless clone.

[sliekens]

Yeah, ADO does things a bit differently, also depending on when you created the pipeline 🤯. I guess your pipeline was created before Sep 2022.
https://learn.microsoft.com/en-us/azure/devops/pipelines/yaml-schema/steps-checkout?view=azure-pipelines#shallow-fetch

[sliekens]

@leonardochaia could you share how you run dotnet affected in Azure Pipelines? I had some problems getting --from $(System.PullRequest.TargetBranch) to work, reporting that it couldn't find the branch.

[leonardochaia]

Hi @sliekens, yes indeed, we have some pipelines created before 2022 that don't have fetch-depth, and we have some newer ones that do have it set to 0.

In CI it's a good idea to invoke dotnet-affected using commit ids instead of branches, so in ADO, in CI of main we have some in-house bash script that fetches the "last successful build" using the ADO HTTP API.
On PRs we use $BUILD_SOURCEBRANCH and $SYSTEM_PULLREQUEST_TARGETBRANCH and then git rev-parse to obtain a commit id.