[git-buildpackage] git-buildpackage-rpm patchset 2015-11-20

Wed Dec 2 22:00:45 CET 2015

Hi Markus,
On Wed, Dec 02, 2015 at 01:58:17PM +0200, Markus Lehtonen wrote:
> On 01/12/15 19:35, "git-buildpackage on behalf of Guido Günther" <git-buildpackage-bounces at lists.sigxcpu.org on behalf of agx at sigxcpu.org> wrote:
> 
> >Hi Markus,
> >On Mon, Nov 30, 2015 at 09:51:24AM +0200, Markus Lehtonen wrote:
> >> Hi,
> >> 
> >> 
> >> 
> >> On 27/11/15 17:38, "Guido Günther" <agx at sigxcpu.org> wrote:
> >> 
> >> >Hi Markus,
> >> >On Fri, Nov 27, 2015 at 04:43:56PM +0200, Markus Lehtonen wrote:
> >> >[..snip..]
> >> >> > > and I think we
> >> >> > > need to rethink the strategy for this submodule. For the deb component
> >> >> > > tests I'm keeping everything in HEAD and set up everything from
> >> >> > > there. The rpm tests need different commits to function which causes/caused
> >> >> > > some breakackge here when pulling in more component tests so I think we
> >> >> > > would be better of needing only a single commit like in the deb
> >> >> > > case. Would that be doable?
> >> >> > 
> >> >> > I haven't figured out any better way – although I admit that this is not
> >> >> > the most elegant solution. The other branches in the testdata repository
> >> >> > basically represent various test packages (i.e. test repositories can be
> >> >> > constructed from those). I didn't find any nice and easy way to
> >> >> > serialize the test repositories (to construct them at test time) either.
> >> >> > And additional git submodules (per-testpackage submodule) don't work
> >> >> > either because the test repositories usually require multiple branches
> >> >> > (upstream, packaging, pristine-tar) and this is not supported in git
> >> >> > submodules which work on single HEAD commit. Better ideas are welcome,
> >> >> > of course.
> >> >> 
> >> >> I played around with one idea I had earlier and came up with a solution
> >> >> to the problem of storing test package git repositories. Please see the
> >> >> 'feature/component-tests' branch in my Github repository
> >> >> https://github.com/marquiz/git-buildpackage-rpm
> >> >>
> >> >> Now, all test data is available in submodule 'HEAD' - test package git
> >> >> repositories are serialized as patches (and some additional data) in
> >> >> separate subdiretories. The branch also contains two patches for
> >> >> buildpackage-rpm (cherry-picked from the 'feature/buildpackage-rpm'
> >> >> branch) that are needed for the buildpackage-rpm unit tests to pass. It
> >> >> also contains one "optional" patch that makes names of the per-testcase
> >> >> temporary directories more user friendly.
> >> >
> >> >Awesome! This makes the rpm and deb component tests use the same
> >> >"strategy" (the rpm ones being far more complete). Running the tests
> >> >fails on me with:
> >> >
> >> >nosetests -x
> >> >ERROR: Basic test of native pkg
> >> >----------------------------------------------------------------------
> >> >Traceback (most recent call last):
> >> >  File "/var/scratch/src/git-buildpackage/git-buildpackage/tests/component/rpm/test_buildpackage_rpm.py", line 116, in test_native_build
> >> >    self.init_test_repo('gbp-test-native')
> >> >  File "/var/scratch/src/git-buildpackage/git-buildpackage/tests/component/rpm/__init__.py", line 58, in init_test_repo
> >> >    dirname = os.path.basename(cls.orig_repos[pkg_name].path)
> >> >KeyError: 'gbp-test-native'
> >> >
> >> >though. This is the first error. I have the full log attached.
> >> 
> >> Oh, sorry, there was erroneous glob in my patch. I pushed a fixed version into my branch. It should work now, please test.
> >
> >
> >Down to one failure now after merging features/component-tests
> >
> >.....................E...................................................................................................................................................................................................................
> >======================================================================
> >ERROR: Create an upstream zip archive
> >----------------------------------------------------------------------
> >Traceback (most recent call last):
> >  File "/usr/lib/python2.7/dist-packages/nose/case.py", line 197, in runTest
> >    self.test(*self.arg)
> >  File "/var/scratch/src/git-buildpackage/git-buildpackage/tests/04_test_submodules.py", line 136, in test_create_zip_archives
> >    '', '', '', 'zip')
> >  File "/var/scratch/src/git-buildpackage/git-buildpackage/gbp/scripts/common/buildpackage.py", line 82, in git_archive_submodules
> >    CatenateZipArchive(main_archive)(submodule_archive)
> >  File "/var/scratch/src/git-buildpackage/git-buildpackage/gbp/command_wrappers.py", line 299, in __call__
> >    Command.__call__(self, [target])
> >  File "/var/scratch/src/git-buildpackage/git-buildpackage/gbp/command_wrappers.py", line 146, in __call__
> >    raise CommandExecFailed(self._format_err())
> >CommandExecFailed: Couldn't append "/tmp/tmpKHcFcV/submodule.zip" to "/tmp/tmpKHcFcV/main.zip": execution failed: [Errno 2] No such file or directory
> >-------------------- >> begin captured logging << --------------------
> >gbp: ERROR: Couldn't append "/tmp/tmpKHcFcV/submodule.zip" to "/tmp/tmpKHcFcV/main.zip": execution failed: [Errno 2] No such file or directory
> >--------------------- >> end captured logging << ---------------------
> >
> >----------------------------------------------------------------------
> 
> Hmm, this must be because you do not have zipmerge installed. I changed the
> "common/buildpackage: support for different archive formats"
> patch in my branch so that zipmerge is now part of Build-Depends

Great, this fixes it.

> >> >Two further things:
> >> >
> >> >* please add gbp_*/ to .gitignore in the data repo so we don't get any
> >> >  untracked content after running the tests.
> >> 
> >> Hmm, no gbp_* temporary directories should be created under the data
> >> (tests/component/rpm/data) directory. They should be created in the
> >> cwd of the user which should be topdir of gbp.
> >
> ># git status
> >On branch locals/rpm-testdata
> >Changes not staged for commit:
> >  (use "git add <file>..." to update what will be committed)
> >  (use "git checkout -- <file>..." to discard changes in working directory)
> >  (commit or discard the untracked or modified content in submodules)
> >
> >	modified:   tests/component/rpm/data (untracked content)
> >
> >cd tests/component/rpm/data/
> >$ git status
> >HEAD detached at 09a446d
> >Untracked files:
> >  (use "git add <file>..." to include in what will be committed)
> >
> >	gbp_TestGbpRpm_smDdsW/
> >	gbp_TestGbpRpm_wKF6Z3/
> >	gbp_TestPqRpm_UOAsQW/
> >
> >nothing added to commit but untracked files present (use "git add" to track)
> >
> >Adding gbp_* to .gitignore in that repo would fix this.
> 
> I think those must be leftovers from my previous erroneous patch. The
> tests should not contaminate the testdata submodule - if it does
> something is wrong. Thus, I would not like to add that to .gitignore
> as it may shadow some problems.

With the current patches there's not gbp_* around anymore as well.

> >> >* It seems manage.py uses lots of code from gbp. Wouldn't it be better
> >> >  to reuse this. We can do this at a later time though, having the test
> >> >  data for the component tests correct is much more important.
> >> 
> >> Yeah, there's a lot of similar code - doing similar things, variable names etc (although it's technically written from scratch). A lot of it is copied from my previous 'bootstrap.py' from the same repository. I specifically wrote it independently, in order to cut the circular dependency between the data repository and gbp api. The amount of overlapping code is not that big IMO, although there is more than I probably anticipated, initially. We can revisit this later, if needed.
> >> 
> >> 
> >> >> If you're not going to merge this in the near future, please update the
> >> >> rpm test data submodule into a valid commit. Currently, it points into
> >> >> some local commit of yours or something as I pointed out in my earlier
> >> >> email.
> >> >
> >> >I'd be happy to pull this in in general. There are some details with
> >> >
> >> >    common/buildpackage: support for different archive formats
> >> >
> >> >which I need to work out but that can be done afterwards as well.
> >> 
> >> Is there something specific? At least it shouldn't affect (deb
> >> )buildpackage at all because zip archives are not supported there.
> >
> >Mostly misreading the code on my end too early in the morning. I'll have
> >to run the performance tests to see if git_archive_submodules regresses
> >once merged but that should be about it.
> 
> OK.

Merged now. I fixed the case where we don't have the submodules checked
out too.

Cheers
 -- Guido