[git-buildpackage] `gbp import-orig --pristine-tar` with component tarballs

[Nish Aravamudan]

On Thu, Jun 29, 2017 at 5:59 PM, Nish Aravamudan
<nish.aravamudan at canonical.com> wrote:
> Hello,
>
> I *think* I may have found a bug in `gbp import-orig --pristine-tar` with
> multiple component tarballs, but I'm not 100% :)

I think I can TL;DR my previous e-mail a bit, hopefully to encourage
some feedback:

I think that when `gbp import-orig --pristine-tar --component=...`
runs, the treeish's stored in the pristine-tar commits are
sub-treeish's of the treeish obtained by untarring the main and all
component tarballs. These treeishs, though, are not stored in git at
all, as they are synthesized by create_pristinetar_commits in order to
invoke `pristine-tar commit` for each tarball. So when you push/pull
the reachable objects from the importer repository, the stored
treeishs become unresolveable by git.

I'm not sure why they are resolveable by git when doing the import
itself. I wonder if there is some local caching going on, but `git
push` doesn't just push everything, it only pushes reachable objects
(aiui).

Does that make sense to anyone else? :)

Thanks,
Nish

> I'm specifically looking (as a test example) src:gosa, e.g.,
> 2.7.4+reloaded2-13. The behavior I'm seeing is that while the machine
> that does the pristine-tar import is able to re-generate the tarballs,
> if I `git push` all my branches and then `git clone` that same
> repository down, the new repository is not able to (the tree IDs in the
> various .id files are not present in the clone repository). I think this
> only happens with multiple component tarballs, because `gbp import-orig`
> does:
>
> scripts/import_orig.py - main
>    ...
>    commit = repo.commit_dir(source.unpacked,
>                             ...
>                             branch=import_branch,
>                             ...
>    ...
>
> where source.unpacked is the fully unpacked main + component tarballs
>
>    source, tmpdir = unpack_tarballs(sourcepackage, source, version, component_tarballs, options)
>
> This commit is later tagged by the --upstream-tag value, so even if we
> were to throw away an upstream branch (e.g., not push it), in theory we
> have the corresponding treeish via this commit.
>
> It's relevant to note that the tree of this commit does not correspond
> to any actual tarball, but the combination of all tarballs (as needed to
> build the package).
>
> However,
>
> deb/git.py - DebianGitRepository::create_pristinetar_commits
>
> does:
>
>         components = [c for (c, t) in component_tarballs]
>         main_tree = self.tree_drop_dirs(upstream_tree, components)
>
>         for component, name in component_tarballs:
>             subtree = self.tree_get_dir(upstream_tree, component)
>             if not subtree:
>                 raise GitRepositoryError("No tree for '%s' found in '%s' to create pristine tar commit from" % (component, upstream_tree))
>             gbp.log.debug("Creating pristine tar commit '%s' from '%s'" % (component, subtree))
>             self.pristine_tar.commit(name, subtree)
>         self.pristine_tar.commit(tarball, main_tree)
>
> These pristine-tar commits, which get put on the pristine-tar branch
> will, potentially [0], correspond to trees that are not present on the
> upstream branch at all, because they are for each tarball's contents (as
> represented by the set of main_tree and all subtrees).
>
> I am not sure how this would ever work -- I know the trees as contained
> by the .id files in the pristine-tar branches must be resolveable by
> git. But I also assume the intentions is that one can do this
> --component import and push like usual and be able to reproduce all the
> tarballs. Perhaps I am missing something obvious!
>
> I completely understand that this is an experimental and new feature,
> but I would love to leverage it in our Ubuntu source package importer
> (as I don't want to deal with all the corner cases gbp already does in
> our code for handling pristine-tar :) -- so if I am on the right track
> and patches would be welcome, then I am happy to work on something! [1]
>
> Thanks,
> -Nish
>
> [0] I say potentially, because I think if there are no component
> tarballs, the main_tree value will correspond exactly to the tree of the
> commit from main earlier.
>
> [1] I see
> https://github.com/agx/git-buildpackage/commit/956982c52500ff52bdab6bb811b86ad6fedbaa19
> was recently committed, and I'm reading it now to see if it's covering
> exactly this case. I think it does not, because that seems to mostly
> just be a wrapper around create_pristinetar_commits, which will end up
> storing trees in the .id files for pristine-tar that are not reachable
> in the repository.
>
> --
> Nishanth Aravamudan
> Ubuntu Server
> Canonical Ltd