[git-buildpackage] `gbp import-orig --pristine-tar` with component tarballs

[Guido Günther]

On Thu, Jul 06, 2017 at 09:45:51AM -0700, Nish Aravamudan wrote:
> On Thu, Jul 6, 2017 at 9:27 AM, Guido Günther <agx at sigxcpu.org> wrote:
> > On Thu, Jul 06, 2017 at 09:16:24AM -0700, Nish Aravamudan wrote:
> >> On 06.07.2017 [08:25:14 +0200], Guido Günther wrote:
> >> > Hi,
> >> > On Wed, Jul 05, 2017 at 12:47:49PM -0700, Nish Aravamudan wrote:
> >> > > On Thu, Jun 29, 2017 at 5:59 PM, Nish Aravamudan
> >> > > <nish.aravamudan at canonical.com> wrote:
> >> > > > Hello,
> >> > > >
> >> > > > I *think* I may have found a bug in `gbp import-orig --pristine-tar` with
> >> > > > multiple component tarballs, but I'm not 100% :)
> >> > >
> >> > > I think I can TL;DR my previous e-mail a bit, hopefully to encourage
> >> > > some feedback:
> >> > >
> >> > > I think that when `gbp import-orig --pristine-tar --component=...`
> >> > > runs, the treeish's stored in the pristine-tar commits are
> >> > > sub-treeish's of the treeish obtained by untarring the main and all
> >> > > component tarballs. These treeishs, though, are not stored in git at
> >> > > all, as they are synthesized by create_pristinetar_commits in order to
> >> > > invoke `pristine-tar commit` for each tarball. So when you push/pull
> >> > > the reachable objects from the importer repository, the stored
> >> > > treeishs become unresolveable by git.
> >> > >
> >> > > I'm not sure why they are resolveable by git when doing the import
> >> > > itself. I wonder if there is some local caching going on, but `git
> >> > > push` doesn't just push everything, it only pushes reachable objects
> >> > > (aiui).
> >> >
> >> > Can you provide steps to reproduce your problem?
> >>
> >> Of course, sorry for not providing that earlier! It turns out that after more
> >> reviewing, it's not the component tarballs that are unreproducible, but the orig
> >> tarball itself (in the case of gosa).
> >>
> >> Test case:
> >>
> >> $ dir=$(mktemp -d)
> >> $ cd $dir
> >> $ pull-debian-source -d gosa
> >> $ mkdir gosa
> >> $ cd gosa
> >> $ git init .
> >> # there are more components, but this is sufficient
> >> $ gbp import-orig --pristine-tar --component=webdav --component=uw-imap --component=systems ../gosa_2.7.4+reloaded2.orig.tar.gz
> >> What will be the source package name? [gosa]
> >> What is the upstream version? [2.7.4+reloaded2]
> >> gbp:info: Importing '../gosa_2.7.4+reloaded2.orig.tar.gz' to branch 'upstream'...
> >> gbp:info: Source package is gosa
> >> gbp:info: Upstream version is 2.7.4+reloaded2
> >> gbp:info: Successfully imported version 2.7.4+reloaded2 of ../gosa_2.7.4+reloaded2.orig.tar.gz
> >> $ pristine-tar list
> >> gosa_2.7.4+reloaded2.orig-webdav.tar.gz
> >> gosa_2.7.4+reloaded2.orig-uw-imap.tar.gz
> >> gosa_2.7.4+reloaded2.orig-systems.tar.gz
> >> gosa_2.7.4+reloaded2.orig.tar.gz
> >> $ pristine-tar checkout gosa_2.7.4_reloaded2.orig.tar.gz
> >> pristine-tar: successfully generated gosa_2.7.4+reloaded2.orig.tar.gz
> >> $ diff -q gosa_2.7.4_reloaded2.orig.tar.gz ../gosa_2.7.4_reloaded2.orig.tar.gz
> >> $ echo $?
> >> 0
> >> $ git push <somewhere, e.g., git+ssh://nacc@git.launchpad.net/~nacc/+git/gosa-import> master upstream pristine-tar
> >> $ cd ..
> >> $ mkdir gosa_test
> >> $ git clone <somewhere> gosa_test
> >> $ cd gosa_test
> >> $ pristine-tar list
> >> gosa_2.7.4+reloaded2.orig-webdav.tar.gz
> >> gosa_2.7.4+reloaded2.orig-uw-imap.tar.gz
> >> gosa_2.7.4+reloaded2.orig-systems.tar.gz
> >> gosa_2.7.4+reloaded2.orig.tar.gz
> >> $ pristine-tar checkout gosa_2.7.4_reloaded2.orig.tar.gz
> >> fatal: ambiguous argument 'ef482620c30c35481200d24897babedc7087718a^{tree}': unknown revision or path not in the working tree.
> >> Use '--' to separate paths from revisions, like this:
> >> 'git <command> [<revision>...] -- [<file>...]'
> >> fatal: Not a valid object name
> >> tar: This does not look like a tar archive
> >> tar: Exiting with failure status due to previous errors
> >> pristine-tar: command failed: git archive --format=tar ef482620c30c35481200d24897babedc7087718a\^\{tree\} | (cd '/tmp/pristine-tar.Fu741PYlX4' && tar x)
> >
> > I see you're problem now. You're trying to rebuild the tarballs with
> > plain pristine-tar not with "gbp build-package". That won't work since
> > pristine-tar knows nothing about component tarballs. Try with
> 
> Ah! Why do the component tarballs work, though? It's only the main
> orig that fails.

Because the trees for the component tarballs are part of the upstream
tree that is referenced by the upstream commit.  A tree that is
identical to the upstream tree _without_ the component tarballs is
is not ref'ed by any commit.

> 
> >   gbp buildpackag --git-builder=/bin/true --git-pristine-tar
> 
> And so to reproduce each component tarball, I will need to pass
> --git-component= with each component name, and (it appears), have to
> have the upstream/<version> tag be reachable? That is, I get:
> 
> $ gbp buildpackage --git-builder=/bin/true --git-pristine-tar
> --git-ignore-branch --git-component=gofon --git-component=gofax
> gbp:error: Couldn't find upstream tree
> 'upstream/2.7.4+reloaded2^{tree}' to create orig tarball via
> pristine-tar

Yeah, the upstream tag is used to create the pristine-tar delta against
so you need it on the remote end too. We need this to create the orig
tarball _without_ the component dirs. Since we don't have the "full"
tree in a pristine-tar commit (it would waste lots of space) we need to
reference the tag.

The restriction that you need to pass in the component tarball names
will go away in the future. It will then pick them from pristine-tar by
default (but it must be overrideable in case a tarball disappears).

Cheers,
 -- Guido

> 
> > and it will work. "gbp export-orig"[1] will be there too soonish
> > (debconf latest) and do the right thing for you (and it will also create
> > tarballs when not using pristine-tar).
> 
> Thank you for your guidance!
>