Debian Bug report logs - #999677
RFP: popcon-stats-data -- Debian's Popularity Contest statistics

Full log

Message #35 received at [email protected] (full text, mbox, reply):

Received: (at 999677) by bugs.debian.org; 17 Nov 2021 12:21:43 +0000
From [email protected] Wed Nov 17 12:21:43 2021
X-Spam-Checker-Version: SpamAssassin 3.4.2-bugs.debian.org_2005_01_02
	(2018-09-13) on buxtehude.debian.org
X-Spam-Level: 
X-Spam-Status: No, score=-15.8 required=4.0 tests=BAYES_00,FROMDEVELOPER,
	HAS_BUG_NUMBER,HEADER_FROM_DIFFERENT_DOMAINS,RCVD_IN_DNSWL_NONE,
	SPF_HELO_NONE,SPF_NEUTRAL,TXREP autolearn=ham autolearn_force=no
	version=3.4.2-bugs.debian.org_2005_01_02
X-Spam-Bayes: score:0.0000 Tokens: new, 12; hammy, 150; neutral, 192; spammy,
	0. spammytokens: hammytokens:0.000-+--popcon, 0.000-+--bookworm,
	0.000-+--H*r:sk:Bill.Al, 0.000-+--H*RU:sk:Bill.Al,
	0.000-+--H*rp:U*Bill.Allombert
Return-path: <[email protected]>
Received: from smtp5-g21.free.fr ([2a01:e0c:1:1599::14]:45810)
	by buxtehude.debian.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256)
	(Exim 4.92)
	(envelope-from <[email protected]>)
	id 1mnJwc-0007ai-T5
	for [email protected]; Wed, 17 Nov 2021 12:21:43 +0000
Received: from seventeen (unknown [IPv6:2a01:e0a:9d1:7200:d3ff:6d19:95c6:5618])
	(Authenticated sender: [email protected])
	by smtp5-g21.free.fr (Postfix) with ESMTPSA id 825925FF9E;
	Wed, 17 Nov 2021 13:21:34 +0100 (CET)
Received: from bill by seventeen with local (Exim 4.94.2)
	(envelope-from <[email protected]>)
	id 1mnJwT-005eNA-Ps; Wed, 17 Nov 2021 13:21:33 +0100
Date: Wed, 17 Nov 2021 13:21:33 +0100
From: Bill Allombert <[email protected]>
To: [email protected],
	Bálint Réczey <[email protected]>,
	999677 <[email protected]>
Subject: Re: Bug#999677: RFP: popcon-stats-data -- Debian's Popularity
 Contest statistics
Message-ID: <YZTzzQCdEmSrNl5Z@seventeen>
Mail-Followup-To: [email protected],
	Bálint Réczey <[email protected]>,
	999677 <[email protected]>
References: <CAK0Odpz9GBZXg3TvSHw5tT-V59kptibm_RnRx_AkTf3Ff33uXA@mail.gmail.com>
 <[email protected]>
 <[email protected]>
 <YZOmLwfa5S9ouCYv@seventeen>
 <[email protected]>
MIME-Version: 1.0
Content-Type: text/plain; charset=iso-8859-1
Content-Disposition: inline
Content-Transfer-Encoding: 8bit
In-Reply-To: <[email protected]>

On Wed, Nov 17, 2021 at 08:05:38AM +0800, Paul Wise wrote:
> On Tue, 2021-11-16 at 13:38 +0100, Bill Allombert wrote:
> 
> > What is the idea exactly ?
> 
> Bálint's idea was to ship popcon data in a popcon-stats-data package in
> the Debian archive. I suggested to instead ship that in the apt
> metadata present in the Packages files.
> 
> > How often the popcon data are going to be refreshed ?
> 
> I would assume with the same frequency as the existing data on the
> popcon.d.o website is refreshed. Anything faster than that would just
> be refreshing unchanged data. Anything slower than that would be
> providing outdated data. Outdated data is fine though, so maybe weekly.
> 
> > Which exact set of data are going to be used ?
> 
> Initially I thought similar to the QA per-package popcon data:
> 
> https://qa.debian.org/popcon.php?package=iotop
> 
> Package: iotop
> Popcon: 30314 7962 21197 1143 12
> 
> If I massage the by_inst file into the same format as this, I calculate
> that the extra Popcon fields would add 3.7 MB to the Packages files and
> that data would change often, making the apt updating process slower.
> So probably the data should go into new files instead and there should
> be a config file snippet to enable downloading them, a tool to query
> and index them and a way for apt clients to get that data.
> 
> Since the Debian repository splits the metadata by suite and component,
> these new statistics should probably do the same. So the raw popcon
> submissions would need to be individually mapped to a suite based on
> the popcon version in the submission, and then each item in the
> submission attributed to that suite/component. For popcon versions that
> don't match a suite, if they match a known Debian version, attribute
> them to the next highest suite and discard submissions with popcon
> versions that were never in Debian, or maybe attribute them to the
> relevant vendor separately. popcon submissions that don't have Debian
> as the vendor probably should be discarded, or maybe attribute them to
> the relevant vendor separately.

So the idea is to have a Popcon file for each suite ?
So let say bookworm is released today. What bookworm/Popcon will contain ?
We release a new popularity-contest package. What sid/Popcon will contain ?
The package migrate to testing; What testing/Popcon will contain ?
As I understand, the metadata for stable are only updated with point
releases. Would that be the same for stable/Popcon ?

I still do not quite see how this would work...
We do not want to provide data generated from a very small subset of
reports for accuracy and privacy reasons.
The current all-popcon-result.gz/stable-popcon-result.gz split is
middle ground between competing constraints.

What not instead write a tool to download all-popcon-result.gz or
stable-popcon-result.gz when needed, and cache them ?
This can then be processed by a tool that makes suggestions.

Cheers,
-- 
Bill. <[email protected]>

Imagine a large red swirl here. 

Send a report that this bug log contains spam.