[git-buildpackage] `gbp import-orig --pristine-tar` with component tarballs

[Guido Günther]

Hi,
On Wed, Jul 05, 2017 at 12:47:49PM -0700, Nish Aravamudan wrote:
> On Thu, Jun 29, 2017 at 5:59 PM, Nish Aravamudan
> <nish.aravamudan at canonical.com> wrote:
> > Hello,
> >
> > I *think* I may have found a bug in `gbp import-orig --pristine-tar` with
> > multiple component tarballs, but I'm not 100% :)
> 
> I think I can TL;DR my previous e-mail a bit, hopefully to encourage
> some feedback:
> 
> I think that when `gbp import-orig --pristine-tar --component=...`
> runs, the treeish's stored in the pristine-tar commits are
> sub-treeish's of the treeish obtained by untarring the main and all
> component tarballs. These treeishs, though, are not stored in git at
> all, as they are synthesized by create_pristinetar_commits in order to
> invoke `pristine-tar commit` for each tarball. So when you push/pull
> the reachable objects from the importer repository, the stored
> treeishs become unresolveable by git.
> 
> I'm not sure why they are resolveable by git when doing the import
> itself. I wonder if there is some local caching going on, but `git
> push` doesn't just push everything, it only pushes reachable objects
> (aiui).

Can you provide steps to reproduce your problem? As I said: the unref'ed
trees are recreated on the fly so no need to push them. They're not even
referenced by commits.
Cheers,
 -- Guido

> Does that make sense to anyone else? :)
> 
> Thanks,
> Nish
> 
> > I'm specifically looking (as a test example) src:gosa, e.g.,
> > 2.7.4+reloaded2-13. The behavior I'm seeing is that while the machine
> > that does the pristine-tar import is able to re-generate the tarballs,
> > if I `git push` all my branches and then `git clone` that same
> > repository down, the new repository is not able to (the tree IDs in the
> > various .id files are not present in the clone repository). I think this
> > only happens with multiple component tarballs, because `gbp import-orig`
> > does:
> >
> > scripts/import_orig.py - main
> >    ...
> >    commit = repo.commit_dir(source.unpacked,
> >                             ...
> >                             branch=import_branch,
> >                             ...
> >    ...
> >
> > where source.unpacked is the fully unpacked main + component tarballs
> >
> >    source, tmpdir = unpack_tarballs(sourcepackage, source, version, component_tarballs, options)
> >
> > This commit is later tagged by the --upstream-tag value, so even if we
> > were to throw away an upstream branch (e.g., not push it), in theory we
> > have the corresponding treeish via this commit.
> >
> > It's relevant to note that the tree of this commit does not correspond
> > to any actual tarball, but the combination of all tarballs (as needed to
> > build the package).
> >
> > However,
> >
> > deb/git.py - DebianGitRepository::create_pristinetar_commits
> >
> > does:
> >
> >         components = [c for (c, t) in component_tarballs]
> >         main_tree = self.tree_drop_dirs(upstream_tree, components)
> >
> >         for component, name in component_tarballs:
> >             subtree = self.tree_get_dir(upstream_tree, component)
> >             if not subtree:
> >                 raise GitRepositoryError("No tree for '%s' found in '%s' to create pristine tar commit from" % (component, upstream_tree))
> >             gbp.log.debug("Creating pristine tar commit '%s' from '%s'" % (component, subtree))
> >             self.pristine_tar.commit(name, subtree)
> >         self.pristine_tar.commit(tarball, main_tree)
> >
> > These pristine-tar commits, which get put on the pristine-tar branch
> > will, potentially [0], correspond to trees that are not present on the
> > upstream branch at all, because they are for each tarball's contents (as
> > represented by the set of main_tree and all subtrees).
> >
> > I am not sure how this would ever work -- I know the trees as contained
> > by the .id files in the pristine-tar branches must be resolveable by
> > git. But I also assume the intentions is that one can do this
> > --component import and push like usual and be able to reproduce all the
> > tarballs. Perhaps I am missing something obvious!
> >
> > I completely understand that this is an experimental and new feature,
> > but I would love to leverage it in our Ubuntu source package importer
> > (as I don't want to deal with all the corner cases gbp already does in
> > our code for handling pristine-tar :) -- so if I am on the right track
> > and patches would be welcome, then I am happy to work on something! [1]
> >
> > Thanks,
> > -Nish
> >
> > [0] I say potentially, because I think if there are no component
> > tarballs, the main_tree value will correspond exactly to the tree of the
> > commit from main earlier.
> >
> > [1] I see
> > https://github.com/agx/git-buildpackage/commit/956982c52500ff52bdab6bb811b86ad6fedbaa19
> > was recently committed, and I'm reading it now to see if it's covering
> > exactly this case. I think it does not, because that seems to mostly
> > just be a wrapper around create_pristinetar_commits, which will end up
> > storing trees in the .id files for pristine-tar that are not reachable
> > in the repository.
> >
> > --
> > Nishanth Aravamudan
> > Ubuntu Server
> > Canonical Ltd
> _______________________________________________
> git-buildpackage mailing list
> git-buildpackage at lists.sigxcpu.org
> http://lists.sigxcpu.org/mailman/listinfo/git-buildpackage
>