Subject: dgit does not handle debian/.gitignore missing from source package
Date: Sun, 09 Sep 2018 13:57:21 -0300
Package: dgit
Version: 6.11
Severity: normal
notmuch had until recently "tar-ignore" in debian/source/options so
that a default set of VCS related things is dropped from the source
package, and (still has) a debian/.gitignore. This means that dgit is
sad because the source packagee does not match the git view, and
e.g. push-source fails.
-- System Information:
Debian Release: buster/sid
APT prefers testing
APT policy: (500, 'testing')
Architecture: amd64 (x86_64)
Foreign Architectures: i386
Kernel: Linux 4.17.0-3-amd64 (SMP w/8 CPU cores)
Locale: LANG=en_CA.UTF-8, LC_CTYPE=en_CA.UTF-8 (charmap=UTF-8), LANGUAGE=en_CA:en (charmap=UTF-8)
Shell: /bin/sh linked to /bin/dash
Init: systemd (via /run/systemd/system)
LSM: AppArmor: enabled
Versions of packages dgit depends on:
ii apt 1.6.4
ii ca-certificates 20170717
ii coreutils 8.30-1
ii curl 7.61.0-1
ii devscripts 2.18.4
ii dpkg-dev 1.19.0.5
ii dput-ng [dput] 1.21
ii git [git-core] 1:2.18.0-1
ii git-buildpackage 0.9.10
ii libdpkg-perl 1.19.0.5
ii libjson-perl 2.97001-1
ii liblist-moreutils-perl 0.416-1+b3
ii libtext-glob-perl 0.10-1
ii libtext-iconv-perl 1.7-5+b6
ii libwww-perl 6.35-2
ii perl [libdigest-sha-perl] 5.26.2-7
Versions of packages dgit recommends:
ii openssh-client [ssh-client] 1:7.8p1-1
Versions of packages dgit suggests:
ii cowbuilder 0.87+b1
ii pbuilder 0.229.3
ii sbuild 0.77.0-4
-- no debconf information
Acknowledgement sent
to Ian Jackson <[email protected]>:
Extra info received and forwarded to list.
(Sun, 09 Sep 2018 17:09:08 GMT) (full text, mbox, link).
Subject: Re: Bug#908417: dgit does not handle debian/.gitignore missing from source package
Date: Sun, 9 Sep 2018 18:07:46 +0100
David Bremner writes ("Bug#908417: dgit does not handle debian/.gitignore missing from source package"):
> Package: dgit
> Version: 6.11
> Severity: normal
>
> notmuch had until recently "tar-ignore" in debian/source/options so
> that a default set of VCS related things is dropped from the source
> package, and (still has) a debian/.gitignore. This means that dgit is
> sad because the source packagee does not match the git view, and
> e.g. push-source fails.
I'm sorry to be dim, but I don't understand what you think the bug in
dgit is ? dgit's design principle is that the source package and git
tree are idnntical. Are you arguing that this is not correct ?
Or is the problem that dgit cannot be used to nmu a package with
tar-ignore in debian/source/options ? dgit passes an explicit -I
option to dpkg-source which I think should override
debian/source/options. And if the uploader did not use dgit push then
the .gitignore would be missing anyway.
Ian.
Acknowledgement sent
to Ian Jackson <[email protected]>:
Extra info received and forwarded to list.
(Sun, 09 Sep 2018 17:21:06 GMT) (full text, mbox, link).
Subject: Re: Bug#908417: dgit does not handle debian/.gitignore missing from source package
Date: Sun, 9 Sep 2018 18:19:10 +0100
Ian Jackson writes ("Re: Bug#908417: dgit does not handle debian/.gitignore missing from source package"):
> I'm sorry to be dim, but I don't understand what you think the bug in
> dgit is ? dgit's design principle is that the source package and git
> tree are idnntical. Are you arguing that this is not correct ?
>
> Or is the problem that dgit cannot be used to nmu a package with
> tar-ignore in debian/source/options ? dgit passes an explicit -I
> option to dpkg-source which I think should override
> debian/source/options. And if the uploader did not use dgit push then
> the .gitignore would be missing anyway.
Also, it's possible that --quilt=dpm could be used to work around the
problem, although that seems much less desirable than removing
tar-exclude from debian/source/options...
(I guess you weren't using --quilt-gbp since that would make a patch
for .gitignore in the dgit view.)
Ian.
--
Ian Jackson <[email protected]> These opinions are my own.
If I emailed you from an address @fyvzl.net or @evade.org.uk, that is
a private address which bypasses my fierce spamfilter.
Hello,
On Sun 09 Sep 2018 at 06:07PM +0100, Ian Jackson wrote:
> I'm sorry to be dim, but I don't understand what you think the bug in
> dgit is ? dgit's design principle is that the source package and git
> tree are idnntical. Are you arguing that this is not correct ?
Based on discussion on IRC, I believe that the problem is that the
options that dgit passes to dpkg-source, while sufficient to ensure that
.gitignore is included in the resultant source package, are not
sufficient to ensure that debian/.gitignore is included. I.e. they do
not properly override tar-ignore.
David, could you provide steps to reproduce? I.e. a repository and a
git commit hash.
--
Sean Whitton
Acknowledgement sent
to David Bremner <[email protected]>:
Extra info received and forwarded to list. Copy sent to Ian Jackson <[email protected]>.
(Sun, 09 Sep 2018 23:57:02 GMT) (full text, mbox, link).
Subject: Re: Bug#908417: dgit does not handle debian/.gitignore missing from source package
Date: Sun, 09 Sep 2018 20:54:44 -0300
Sean Whitton <[email protected]> writes:
> Based on discussion on IRC, I believe that the problem is that the
> options that dgit passes to dpkg-source, while sufficient to ensure that
> .gitignore is included in the resultant source package, are not
> sufficient to ensure that debian/.gitignore is included. I.e. they do
> not properly override tar-ignore.
>
> David, could you provide steps to reproduce? I.e. a repository and a
> git commit hash.
Hash: 8feec188a0bb3ed5dcda2c0e22c7645ca0d8f618
Repo: https://git.notmuchmail.org/git/notmuch
If you look at branch release, you can see the dgit created merge after
that commit, and before the actual archive tag.
Acknowledgement sent
to Ian Jackson <[email protected]>:
Extra info received and forwarded to list.
(Mon, 10 Sep 2018 09:18:06 GMT) (full text, mbox, link).
Subject: Re: Bug#908417: dgit does not handle debian/.gitignore missing from source package
Date: Mon, 10 Sep 2018 10:15:50 +0100
Control: retitle -1 do something about tar-ignore in d/s/options
David Bremner writes ("Re: Bug#908417: dgit does not handle debian/.gitignore missing from source package"):
> Sean Whitton <[email protected]> writes:
> > Based on discussion on IRC, I believe that the problem is that the
> > options that dgit passes to dpkg-source, while sufficient to ensure that
> > .gitignore is included in the resultant source package, are not
> > sufficient to ensure that debian/.gitignore is included. I.e. they do
> > not properly override tar-ignore.
Oh. I see.
I hvve just RTFM dpkg-source(1) again (but not yet UTSL) and there
doesn't seem to be a way to reset the list of -I options.
I think the best I can do, therefore, is to detect this situation and
print a warning. It has to be only a warning, because if the rogue -I
does not match any files in the tree then it is harmless - and this
situation will arise when NMUing a non-dgit-pushed package with such a
rogue -I.
(Of course a package with such a rogue -I is probably excluding parts
of the source code from the .dsc and is therefore a DFSG violation and
maybe a copyleft violation, although in the case of .gitignore quite a
minimal one.)
It might be a good idea to have a passlist of known-good options and
warn on any others. But tar-ignore should be handled specially
because the warning can explain the likely consequences (and the fix,
although the fix - removing the option - is obvious and harmless).
It would be possible in principle to detect whether any files in the
git tree would match the rogue ignore - but not without reimplementing
dpkg-source's matching logic.
Ideally dgit should tolerate a tar-ignore which matches exactly .git
in the toplevel but that's even more work because it would involve
predicting the things which a regexp matches.
David, would a warning have been sufficient, to avoid this being a
significant inconvenience to you ? Something like this perhsps:
debian/source/options contains a tar-ignore option.
If this option matches anything, dpkg-source's source
package will differs from your git tree, and dgit push will fail.
tar-ignore is not needed with dgit push, so you can remove it.
> > David, could you provide steps to reproduce? I.e. a repository and a
> > git commit hash.
>
> Hash: 8feec188a0bb3ed5dcda2c0e22c7645ca0d8f618
> Repo: https://git.notmuchmail.org/git/notmuch
>
> If you look at branch release, you can see the dgit created merge after
> that commit, and before the actual archive tag.
Thanks. I will look at this.
Ian.
--
Ian Jackson <[email protected]> These opinions are my own.
If I emailed you from an address @fyvzl.net or @evade.org.uk, that is
a private address which bypasses my fierce spamfilter.
Changed Bug title to 'do something about tar-ignore in d/s/options' from 'dgit does not handle debian/.gitignore missing from source package'.
Request was from Ian Jackson <[email protected]>
to [email protected].
(Mon, 10 Sep 2018 09:18:06 GMT) (full text, mbox, link).
Acknowledgement sent
to David Bremner <[email protected]>:
Extra info received and forwarded to list. Copy sent to Ian Jackson <[email protected]>.
(Tue, 11 Sep 2018 11:15:03 GMT) (full text, mbox, link).
Subject: Re: Bug#908417: dgit does not handle debian/.gitignore missing from source package
Date: Tue, 11 Sep 2018 08:12:54 -0300
Ian Jackson <[email protected]> writes:
> David, would a warning have been sufficient, to avoid this being a
> significant inconvenience to you ? Something like this perhsps:
>
> debian/source/options contains a tar-ignore option.
> If this option matches anything, dpkg-source's source
> package will differs from your git tree, and dgit push will fail.
> tar-ignore is not needed with dgit push, so you can remove it.
It would have saved me some time. I don't really think "so you can
remove it" follows from "is not needed with dgit push". People use
dpkg-source outside dgit, and may have legitimate reasons for including
a tar-ignore option. In the case of notmuch it is used to prevent a
seperately downloadable performance corpus (400MB) from being included
in the debian source package. There might well be other ways to
accomplish the same goals, but every change required is a barrier to
using dgit.
d
Acknowledgement sent
to David Bremner <[email protected]>:
Extra info received and forwarded to list. Copy sent to Ian Jackson <[email protected]>.
(Thu, 13 Sep 2018 10:21:03 GMT) (full text, mbox, link).
Subject: Re: Bug#908417: dgit does not handle debian/.gitignore missing from source package
Date: Thu, 13 Sep 2018 07:16:03 -0300
David Bremner <[email protected]> writes:
> Ian Jackson <[email protected]> writes:
>
>
>> David, would a warning have been sufficient, to avoid this being a
>> significant inconvenience to you ? Something like this perhsps:
>>
>> debian/source/options contains a tar-ignore option.
>> If this option matches anything, dpkg-source's source
>> package will differs from your git tree, and dgit push will fail.
>> tar-ignore is not needed with dgit push, so you can remove it.
>
> It would have saved me some time. I don't really think "so you can
> remove it" follows from "is not needed with dgit push". People use
> dpkg-source outside dgit, and may have legitimate reasons for including
> a tar-ignore option. In the case of notmuch it is used to prevent a
> seperately downloadable performance corpus (400MB) from being included
> in the debian source package. There might well be other ways to
> accomplish the same goals, but every change required is a barrier to
> using dgit.
As another example, since I removed tar-ignore from notmuch, I have
W: notmuch source: diff-contains-git-control-dir .git/dgit/unpack/notmuch-0.27/.git
E: notmuch source: source-contains-unsafe-symlink .git/dgit/unpack/notmuch-0.27/.git/objects
(from lintian)
Acknowledgement sent
to Ian Jackson <[email protected]>:
Extra info received and forwarded to list.
(Thu, 13 Sep 2018 10:54:03 GMT) (full text, mbox, link).
Subject: Re: Bug#908417: dgit does not handle debian/.gitignore missing from source package
Date: Thu, 13 Sep 2018 11:50:44 +0100
David Bremner writes ("Bug#908417: dgit does not handle debian/.gitignore missing from source package"):
> As another example, since I removed tar-ignore from notmuch, I have
>
> W: notmuch source: diff-contains-git-control-dir .git/dgit/unpack/notmuch-0.27/.git
> E: notmuch source: source-contains-unsafe-symlink .git/dgit/unpack/notmuch-0.27/.git/objects
Certainly that is wrong.
I think this would happen if you ran dpkg-source -b (or some tool that
calls it but is ignorant of git) in a git working tree without
relevant -I options. I think my answer is "don't do that then".
I will file a bug against dpkg requesting a command line option which
resets the tar-ignore list.
Even if and when that is all done, you will still have the problem
that source packages built in whatever way generated the errors above
will still be wrong: they will wrongly lack debian/.gitignore.
Ian.
--
Ian Jackson <[email protected]> These opinions are my own.
If I emailed you from an address @fyvzl.net or @evade.org.uk, that is
a private address which bypasses my fierce spamfilter.
Acknowledgement sent
to Ian Jackson <[email protected]>:
Extra info received and forwarded to list.
(Thu, 13 Sep 2018 11:33:02 GMT) (full text, mbox, link).
Subject: Re: Bug#908417: dgit does not handle debian/.gitignore missing from source package
Date: Thu, 13 Sep 2018 12:30:49 +0100
Ian Jackson writes ("Re: Bug#908417: dgit does not handle debian/.gitignore missing from source package"):
> I will file a bug against dpkg requesting a command line option which
> resets the tar-ignore list.
I've just filed
#908742 Want way to reset tar-ignore list
> Even if and when that is all done, you will still have the problem
> that source packages built in whatever way generated the errors above
> will still be wrong: they will wrongly lack debian/.gitignore.
It occurs to me that I think you can give a value for the tar-ignore
option. So maybe you can say tar-ignore=.git ?
Probably, then, you want to put into debian/source/options the output
from
dgit print-dpkg-source-ignores
which is currently
-i(?:^|/)\.git(?:/|$) -I.git
It might change if we find bugs in it. That has happened once
already. But if your package doesn't contain files for which that
rune is buggy, I think you should be OK.
HTH.
I hvae also filed
Bug#908747 Default -I and -i option should not exclude .<vcs>ignore
about the root cause of all of this. I expect that to be
controversial and slow to fix.
Regards,
Ian.
--
Ian Jackson <[email protected]> These opinions are my own.
If I emailed you from an address @fyvzl.net or @evade.org.uk, that is
a private address which bypasses my fierce spamfilter.
Acknowledgement sent
to David Bremner <[email protected]>:
Extra info received and forwarded to list. Copy sent to Ian Jackson <[email protected]>.
(Thu, 13 Sep 2018 11:51:03 GMT) (full text, mbox, link).
Subject: Re: Bug#908417: dgit does not handle debian/.gitignore missing from source package
Date: Thu, 13 Sep 2018 08:49:22 -0300
Ian Jackson <[email protected]> writes:
> David Bremner writes ("Bug#908417: dgit does not handle debian/.gitignore missing from source package"):
>> As another example, since I removed tar-ignore from notmuch, I have
>>
>> W: notmuch source: diff-contains-git-control-dir .git/dgit/unpack/notmuch-0.27/.git
>> E: notmuch source: source-contains-unsafe-symlink .git/dgit/unpack/notmuch-0.27/.git/objects
>
> Certainly that is wrong.
>
> I think this would happen if you ran dpkg-source -b (or some tool that
> calls it but is ignorant of git) in a git working tree without
> relevant -I options. I think my answer is "don't do that then".
Sure. My point was that the tar-ignore option that I had to remove was
in lieu of the relevant -I options. So the issue is a bit more subtle
than your proposed wording would suggest, and seems to me to be more
related to the use of debian/source/options versus command line options.
> I will file a bug against dpkg requesting a command line option which
> resets the tar-ignore list.
>
> Even if and when that is all done, you will still have the problem
> that source packages built in whatever way generated the errors above
> will still be wrong: they will wrongly lack debian/.gitignore.
They're wrong for dgit, and maybe for DFSG, but not for building
throwaway source packages that are not distributed.
Acknowledgement sent
to David Bremner <[email protected]>:
Extra info received and forwarded to list. Copy sent to Ian Jackson <[email protected]>.
(Thu, 13 Sep 2018 11:54:02 GMT) (full text, mbox, link).
Subject: Re: Bug#908417: dgit does not handle debian/.gitignore missing from source package
Date: Thu, 13 Sep 2018 08:51:56 -0300
Ian Jackson <[email protected]> writes:
> It occurs to me that I think you can give a value for the tar-ignore
> option. So maybe you can say tar-ignore=.git ?
>
> Probably, then, you want to put into debian/source/options the output
> from
> dgit print-dpkg-source-ignores
> which is currently
> -i(?:^|/)\.git(?:/|$) -I.git
> It might change if we find bugs in it. That has happened once
> already. But if your package doesn't contain files for which that
> rune is buggy, I think you should be OK.
Thanks, that's a helpful suggestion.
d
Acknowledgement sent
to Ian Jackson <[email protected]>:
Extra info received and forwarded to list.
(Thu, 13 Sep 2018 13:36:02 GMT) (full text, mbox, link).
Subject: Re: Bug#908417: dgit does not handle debian/.gitignore missing from source package
Date: Thu, 13 Sep 2018 14:32:46 +0100
David Bremner writes ("Re: Bug#908417: dgit does not handle debian/.gitignore missing from source package"):
> They're wrong for dgit, and maybe for DFSG, but not for building
> throwaway source packages that are not distributed.
Not to detract from your other comments, but:
For a throwaway source package I think you don't care if it has a .git
directory and your whole revision history ?
Regards,
Ian.
--
Ian Jackson <[email protected]> These opinions are my own.
If I emailed you from an address @fyvzl.net or @evade.org.uk, that is
a private address which bypasses my fierce spamfilter.
Acknowledgement sent
to David Bremner <[email protected]>:
Extra info received and forwarded to list. Copy sent to Ian Jackson <[email protected]>.
(Sun, 16 Sep 2018 13:57:03 GMT) (full text, mbox, link).
Subject: Re: Bug#908417: dgit does not handle debian/.gitignore missing from source package
Date: Sun, 16 Sep 2018 10:53:28 -0300
Ian Jackson <[email protected]> writes:
> David Bremner writes ("Re: Bug#908417: dgit does not handle debian/.gitignore missing from source package"):
>> They're wrong for dgit, and maybe for DFSG, but not for building
>> throwaway source packages that are not distributed.
>
> Not to detract from your other comments, but:
>
> For a throwaway source package I think you don't care if it has a .git
> directory and your whole revision history ?
>
Fair point. In my case I want to run lintian on the thing I'm not
uploading to Debian, and the .git stuff introduces a bunch of noise.
d