[git-buildpackage] `gbp import-orig --pristine-tar` with component tarballs

[Nish Aravamudan]

On Thu, Jul 6, 2017 at 10:20 AM, Guido Günther <agx at sigxcpu.org> wrote:
> On Thu, Jul 06, 2017 at 09:45:51AM -0700, Nish Aravamudan wrote:
>> On Thu, Jul 6, 2017 at 9:27 AM, Guido Günther <agx at sigxcpu.org> wrote:
>> > On Thu, Jul 06, 2017 at 09:16:24AM -0700, Nish Aravamudan wrote:
>> >> On 06.07.2017 [08:25:14 +0200], Guido Günther wrote:
>> >> > Hi,
>> >> > On Wed, Jul 05, 2017 at 12:47:49PM -0700, Nish Aravamudan wrote:
>> >> > > On Thu, Jun 29, 2017 at 5:59 PM, Nish Aravamudan
>> >> > > <nish.aravamudan at canonical.com> wrote:
>> >> > > > Hello,
>> >> > > >
>> >> > > > I *think* I may have found a bug in `gbp import-orig --pristine-tar` with
>> >> > > > multiple component tarballs, but I'm not 100% :)
>> >> > >
>> >> > > I think I can TL;DR my previous e-mail a bit, hopefully to encourage
>> >> > > some feedback:
>> >> > >
>> >> > > I think that when `gbp import-orig --pristine-tar --component=...`
>> >> > > runs, the treeish's stored in the pristine-tar commits are
>> >> > > sub-treeish's of the treeish obtained by untarring the main and all
>> >> > > component tarballs. These treeishs, though, are not stored in git at
>> >> > > all, as they are synthesized by create_pristinetar_commits in order to
>> >> > > invoke `pristine-tar commit` for each tarball. So when you push/pull
>> >> > > the reachable objects from the importer repository, the stored
>> >> > > treeishs become unresolveable by git.
>> >> > >
>> >> > > I'm not sure why they are resolveable by git when doing the import
>> >> > > itself. I wonder if there is some local caching going on, but `git
>> >> > > push` doesn't just push everything, it only pushes reachable objects
>> >> > > (aiui).
>> >> >
>> >> > Can you provide steps to reproduce your problem?
>> >>
>> >> Of course, sorry for not providing that earlier! It turns out that after more
>> >> reviewing, it's not the component tarballs that are unreproducible, but the orig
>> >> tarball itself (in the case of gosa).
>> >>
>> >> Test case:
>> >>
>> >> $ dir=$(mktemp -d)
>> >> $ cd $dir
>> >> $ pull-debian-source -d gosa
>> >> $ mkdir gosa
>> >> $ cd gosa
>> >> $ git init .
>> >> # there are more components, but this is sufficient
>> >> $ gbp import-orig --pristine-tar --component=webdav --component=uw-imap --component=systems ../gosa_2.7.4+reloaded2.orig.tar.gz
>> >> What will be the source package name? [gosa]
>> >> What is the upstream version? [2.7.4+reloaded2]
>> >> gbp:info: Importing '../gosa_2.7.4+reloaded2.orig.tar.gz' to branch 'upstream'...
>> >> gbp:info: Source package is gosa
>> >> gbp:info: Upstream version is 2.7.4+reloaded2
>> >> gbp:info: Successfully imported version 2.7.4+reloaded2 of ../gosa_2.7.4+reloaded2.orig.tar.gz
>> >> $ pristine-tar list
>> >> gosa_2.7.4+reloaded2.orig-webdav.tar.gz
>> >> gosa_2.7.4+reloaded2.orig-uw-imap.tar.gz
>> >> gosa_2.7.4+reloaded2.orig-systems.tar.gz
>> >> gosa_2.7.4+reloaded2.orig.tar.gz
>> >> $ pristine-tar checkout gosa_2.7.4_reloaded2.orig.tar.gz
>> >> pristine-tar: successfully generated gosa_2.7.4+reloaded2.orig.tar.gz
>> >> $ diff -q gosa_2.7.4_reloaded2.orig.tar.gz ../gosa_2.7.4_reloaded2.orig.tar.gz
>> >> $ echo $?
>> >> 0
>> >> $ git push <somewhere, e.g., git+ssh://nacc@git.launchpad.net/~nacc/+git/gosa-import> master upstream pristine-tar
>> >> $ cd ..
>> >> $ mkdir gosa_test
>> >> $ git clone <somewhere> gosa_test
>> >> $ cd gosa_test
>> >> $ pristine-tar list
>> >> gosa_2.7.4+reloaded2.orig-webdav.tar.gz
>> >> gosa_2.7.4+reloaded2.orig-uw-imap.tar.gz
>> >> gosa_2.7.4+reloaded2.orig-systems.tar.gz
>> >> gosa_2.7.4+reloaded2.orig.tar.gz
>> >> $ pristine-tar checkout gosa_2.7.4_reloaded2.orig.tar.gz
>> >> fatal: ambiguous argument 'ef482620c30c35481200d24897babedc7087718a^{tree}': unknown revision or path not in the working tree.
>> >> Use '--' to separate paths from revisions, like this:
>> >> 'git <command> [<revision>...] -- [<file>...]'
>> >> fatal: Not a valid object name
>> >> tar: This does not look like a tar archive
>> >> tar: Exiting with failure status due to previous errors
>> >> pristine-tar: command failed: git archive --format=tar ef482620c30c35481200d24897babedc7087718a\^\{tree\} | (cd '/tmp/pristine-tar.Fu741PYlX4' && tar x)
>> >
>> > I see you're problem now. You're trying to rebuild the tarballs with
>> > plain pristine-tar not with "gbp build-package". That won't work since
>> > pristine-tar knows nothing about component tarballs. Try with
>>
>> Ah! Why do the component tarballs work, though? It's only the main
>> orig that fails.
>
> Because the trees for the component tarballs are part of the upstream
> tree that is referenced by the upstream commit.  A tree that is
> identical to the upstream tree _without_ the component tarballs is
> is not ref'ed by any commit.

Ah of course, as subtrees effectively. Sorry for being a bit dense!

>> >   gbp buildpackag --git-builder=/bin/true --git-pristine-tar
>>
>> And so to reproduce each component tarball, I will need to pass
>> --git-component= with each component name, and (it appears), have to
>> have the upstream/<version> tag be reachable? That is, I get:
>>
>> $ gbp buildpackage --git-builder=/bin/true --git-pristine-tar
>> --git-ignore-branch --git-component=gofon --git-component=gofax
>> gbp:error: Couldn't find upstream tree
>> 'upstream/2.7.4+reloaded2^{tree}' to create orig tarball via
>> pristine-tar
>
> Yeah, the upstream tag is used to create the pristine-tar delta against
> so you need it on the remote end too. We need this to create the orig
> tarball _without_ the component dirs. Since we don't have the "full"
> tree in a pristine-tar commit (it would waste lots of space) we need to
> reference the tag.

Got it, I think we can do that without too much hassle in our code.

> The restriction that you need to pass in the component tarball names
> will go away in the future. It will then pick them from pristine-tar by
> default (but it must be overrideable in case a tarball disappears).

Yep understood. Thank you for your excellent work on gbp!

-Nish