News for hackers. Stuff that matters.
Re:HomeBrew Analytics – top 1000 packages installed over las... by woodruffw on Sunday September 03, 2017 @08:41PM (#15164828)
Homebrew maintainer here. That language could probably be more precise - only current maintainers have access to detailed analytics (the details being specified on that same page).

I wasn't part of the creation of that particular page, but one thing we (the maintainers) commonly find ourselves doing is publicly referencing install statistics as justification for removing an unused formula or taking extra care during a version bump. Having public statistics for the top 1000 most popular packages makes those considerations a little bit more transparent.

Edit: I've created a PR to fix the language: https://github.com/Homebrew/brew/pull/3120

Re:HomeBrew Analytics – top 1000 packages installed over las... by zakk on Monday September 04, 2017 @04:09AM (#15166547)
Mildly unrelated question.

Why is telemetry collected through Google? I suppose that’s because they have some particularly convenient APIs, however please consider that many user are concerned by the amount of information Google has about their lifes.

(I know, it’s possible to opt out, but good choices should be the default!)

Re:HomeBrew Analytics – top 1000 packages installed over las... by mikemcquaid on Monday September 04, 2017 @05:05AM (#15166787)
> Why is telemetry collected through Google?

No-one else provides anywhere near the same scale for free. We have over a million monthly active users and every other solution we tried (including FOSS ones) either fell over at that load or we couldn't find anyone willing to provide hosting for us.

We don't send any personally identifiable information to Google. You are tracked by a randomly generated UUID (which you can regenerate at any time) and we tell Google to not store IP addresses.

Re:HomeBrew Analytics – top 1000 packages installed over las... by zakk on Monday September 04, 2017 @05:42AM (#15166944)
First of all, let me say I appreciate your response and your work on Homebrew, a great piece of software, and quite a blessing for Mac users.

> No-one else provides anywhere near the same scale for free.

That's a bit naive. Google is not a charity and provides that service by making a profit out of user data.

> We don't send any personally identifiable information to Google. You are tracked by a randomly generated UUID (which you can regenerate at any time) and we tell Google to not store IP addresses.

Even with no IP, Google can easily cross-references searches and the random UUID, since a typical use case is that a user installs something through Homebrew after Googling it.

Please rethink this bad default choice. And sorry if my reply sounds harsh, but I think that Google tracking by default is an extremely bad choice for your users.

Re:HomeBrew Analytics – top 1000 packages installed over las... by mikemcquaid on Tuesday September 05, 2017 @05:36AM (#15173673)
> That's a bit naive. Google is not a charity and provides that service by making a profit out of user data.

Yep and that's an acceptable trade-off for the maintainers of Homebrew given that we need analytics to do our (volunteer, free-time) job adequately and we do not have financial resources for other alternatives. If you're willing to provide those financial resources indefinitely: get in touch.

> Even with no IP, Google can easily cross-references searches and the random UUID, since a typical use case is that a user installs something through Homebrew after Googling it.

That may be technically possible but I see no evidence that it is the case.

> Please rethink this bad default choice. And sorry if my reply sounds harsh, but I think that Google tracking by default is an extremely bad choice for your users.

We disagree. Please consider another macOS package manager. MacPorts is a good alternative.

Re:HomeBrew Analytics – top 1000 packages installed over las... by octalmage on Monday September 04, 2017 @11:40AM (#15168942)
I also use Google Analytics for tracking usage in my desktop applications (opt out) and it's great. I think most just think about web stats (page views) but I just use the event system. This allows me to identify unused features or possible confusing UX. I also started using it in a project at work and the data has been extremely useful.
Re:HomeBrew Analytics – top 1000 packages installed over las... by zeroxfe on Monday September 04, 2017 @09:18AM (#15168013)
> That's a bit naive. Google is not a charity and provides that service by making a profit out of user data.

So what if they make profit -- they provide a fantastic service that actually works well, and it allows homebrew (and many other systems) to continue to provide it's services for free.

Re:HomeBrew Analytics – top 1000 packages installed over las... by zakk on Monday September 04, 2017 @10:35AM (#15168468)
If you are fine with a company profiting out of your personal information and habits, I am not, and I think is a reasonable choice to assume that people are not fine with that.
Re:HomeBrew Analytics – top 1000 packages installed over las... by sigzero on Monday September 04, 2017 @11:33AM (#15168898)
Then those people won't use Homebrew.
Re:HomeBrew Analytics – top 1000 packages installed over las... by zakk on Monday September 04, 2017 @12:25PM (#15169220)
Sure! But the telemetry is a default setting, and the implications for the user’s privacy are not clear at all.

Opt-in telemetry clearly stating that Google will record the first three bytes of your IP? I’m OK with that!

Re:HomeBrew Analytics – top 1000 packages installed over las... by pdkl95 on Monday September 04, 2017 @06:09AM (#15167059)
> We don't send any personally identifiable information to Google.

Yes, you do. While you use[1] the "Anonymize IP" option, a packet is still sent from the user's IP. Google's business model includes gathering as much data as possible so it's foolish to think that they are throwing data away in this situation. You may disagree and trust Google to honor the "Anonymize IP" option, but trust is not transitive so you shouldn't ever assume users agree (use opt-in in every situation).

However, claiming you don't send pii to Google makes me wonder if you have actually read the documentation for GA? The "Anonymize IP" (aid=1) option is blatant doublespeak. From their own documentation[2]:

> The IP anonymization feature in Analytics sets the last octet of IPv4 user IP addresses ...

They are only masking out the last 8 bits of the address, which are the least interesting bits. You can still discover the ASN from the remaining data. At worst all that option did is add a 1-in-256 guess when correlating your analytics data to the rest of Google's tracking data. That is trivial to overcome Google's massive databases of tracking data.

You even provide a unique additional per-install tracking number that lets Google track users when they move to a different IP address. Once a correlation exists between your analytics data and everything else at Google, your analytics events provide a reliable report about that can allows other tracking data to be correlated to the new IP address.

Why does that option exist? It's possible that it was designed to mislead developers into sending Google tracking information, but their own documentation[2] suggests a different hypothesis:

> This feature is designed to help site owners comply with their own privacy policies or, in some countries, recommendations from local data protection authorities

This is a feature designed to check boxes on compliance requirements, not to provide any provide actual anonymity to users.

[1] https://github.com/Homebrew/brew/blob/fd4fe3b80cab9902437016...

[2] https://support.google.com/analytics/answer/2763052?hl=en

Re:HomeBrew Analytics – top 1000 packages installed over las... by osteele on Monday September 04, 2017 @07:16AM (#15167343)
>> We don't send any personally identifiable information to Google.

> Yes, you do

PII is a term of the art which the GP is using in its standard sense and you are not. https://en.wikipedia.org/wiki/Personally_identifiable_inform...

(This is independent of the deontolic status of your comment.)

Re:HomeBrew Analytics – top 1000 packages installed over las... by damnfine on Monday September 04, 2017 @09:34AM (#15168109)
Redefining a phrase to omit much of its common meaning is what got us here. I appreciate you helping to bridge the gap by translating, I am just bemoaning its nessessity.
Re:HomeBrew Analytics – top 1000 packages installed over las... by dredmorbius on Sunday September 03, 2017 @10:49PM (#15165360)
Beware the Tyranny of the Minimum Viable User.

It may well be that some relatively-little-used tool nontheless has a very significant use. Popularity is one factor within the mix, but only one.

https://redd.it/69wk8y

Re:HomeBrew Analytics – top 1000 packages installed over las... by woodruffw on Sunday September 03, 2017 @11:04PM (#15165415)
You're right, which is why popularity isn't the sole factor in determining whether to remove a formula from the core repository. It's just one of the things we reference to justify our choice, when relevant.

We generally only refer to popularity after a formula comes up on our radar due to failing other sniff tests. To give you an idea for some of them:

1. Multiple subsequent releases without anybody bothering to update the formula

2. Historical problems with the package (flaky builds, complex build systems)

3. Historical problems with the upstream (patches ignored, unwillingness to cooperate with packagers, unreliable download servers)

Re:HomeBrew Analytics – top 1000 packages installed over las... by dredmorbius on Sunday September 03, 2017 @11:16PM (#15165463)
Criticality in workflow might also be a consideration. I'd have to check how homebrew does this, but Debian frequently has separate documentation and debug packages as supplements to core functionality. Those might be essential in cases.

Since homebrew is aimed mostly at technical types and devs, build tools themselves are probably fairly highly featured, but those are easy things to lose in large general-public releases. (Android's abysmally poor shell tools come to mind.)

Sounds like a pretty good basic set. The flaky builds criterion in particular seems like a strong signal of a low-quality upstream.

Re:HomeBrew Analytics – top 1000 packages installed over las... by falcolas on Sunday September 03, 2017 @09:27PM (#15165009)
> as justification for removing an unused formula

This does not, in my mind, help the case at all. This culture of deletion makes no real sense to me; something not being used for a month or a year doesn't mean it's not being used at all. What's the real cost of just leaving those formula around? A slight problem with it not working immediately when it is used? At least then it can be fixed, instead of seeing the "No formula found" text.

The culture of deletion surrounding the long tail of digital artifacts just doesn't make any sense. Yeah, yeah. Get off my lawn, too.

Re:HomeBrew Analytics – top 1000 packages installed over las... by woodruffw on Sunday September 03, 2017 @09:48PM (#15165107)
> What's the real cost of just leaving those formula around?

Maintenance.

When unused and unmaintained projects and formulae pile up in Homebrew, we end up spending a tremendous amount of time and effort patching mostly unused software for a very small part of the userbase. That dis-proportionality hurts the 95% of users who expect timely and well-tested updates to major packages.

We used to provide a "boneyard" tap for unused/unmaintained formulae, but even that led to a lot of requests for support that we simply can't provide. If something is being removed from the core tap, our current recommendation is to put it a personal tap[1].

> The culture of deletion surrounding the long tail of digital artifacts just doesn't make any sense.

Keep in mind that the "artifacts" in question are still available, since Homebrew and all Homebrew taps are just Git underneath. You might not be able to build an old formula for compatibility reasons, but all prior work is available for reference.

[1]: https://github.com/Homebrew/brew/blob/master/docs/How-to-Cre...

Re:HomeBrew Analytics – top 1000 packages installed over las... by falcolas on Monday September 04, 2017 @10:52AM (#15168591)
Somewhat of a tangent, but a lot of these reasons seem to boil down to "we don't have the resources". The HomeBrew project does appear to have a Patreon account, but to find it you have to go to the GitHub repo and scroll to the bottom of the readme. It might help to promote that a bit more on the homepage. It might also help to engage a bit more with the community other than "here's a new release".

IOW, HomeBrew seems to be big enough that it needs to start engaging in some PR.

Re:HomeBrew Analytics – top 1000 packages installed over las... by AlphaSite on Sunday September 03, 2017 @09:39PM (#15165062)
maintain burden. betroth and the fact that git/GitHub really doesn't like huge dirs.
Re:HomeBrew Analytics – top 1000 packages installed over las... by sillysaurus3 on Sunday September 03, 2017 @09:42PM (#15165071)
I've been delighted to install casks using `brew cask install X` where X is some obscure program that I thought brew cask would never have bothered adding. Stuff like that dazzled me about brew cask, so I hope it never goes away even under the guise of maintenance burden.

(It's pretty easy to contribute your own formulas, too! The checklist is pretty small.)

That said, I've occasionally run into a broken `brew cask install X`, so I get that maintenance is a thing. But in that situation it seems best to let someone else notice that and contribute a patch rather than remove it entirely. I understand that might eventually lead to a large percentage of broken casks though.

Re:HomeBrew Analytics – top 1000 packages installed over las... by woodruffw on Sunday September 03, 2017 @10:01PM (#15165161)
As a nit, `brew install` and `brew cask install` use largely different codebases ;)

In the best case, the community does pick up the work of patching and maintaining formulae (and casks). However, that's the best case, and low-volume formulae rarely fall into it.

In my experience, formulae tend to get abandoned by the upstream that originally submitted them, causing them to eventually break when the system or surrounding dependencies change. One potential solution is to have a chain of custody for formula maintenance, but that hasn't worked so well for MacPorts.

Re:HomeBrew Analytics – top 1000 packages installed over las... by contras1970 on Monday September 04, 2017 @10:52AM (#15168596)
> the community does pick up the work of patching and maintaining formulae (and casks). However, that's the best case, and low-volume formulae rarely fall into it.

IMO that's self-inflicted by homebrew's policies. from https://docs.brew.sh/Acceptable-Formulae.html :

> We frown on authors submitting their own work unless it is very popular.

I'd expect "very popular" software to have relatively large populations of users capable of maintaining the formulae (and authors less likely to bother with packaging), while in lesser-known software, the respective authors are most likely to maintain long-time interest.

Re:HomeBrew Analytics – top 1000 packages installed over las... by mikemcquaid on Monday September 04, 2017 @05:07AM (#15166794)
> At least then it can be fixed, instead of seeing the "No formula found" text.

You don't see that text but instead text indicating the formula has been deleted.

> The culture of deletion surrounding the long tail of digital artifacts just doesn't make any sense.

They will be preserved in the Git history for as long as we continue to use Git.

Both the above comments make it seem like you've not really done any research into your own objections, here.

Re:HomeBrew Analytics – top 1000 packages installed over las... by falcolas on Monday September 04, 2017 @10:23AM (#15168377)
The best defense is a good offense. I express concern about your project's use of Google Analytics and then releasing that data to the public, then about the culture of deletion; and I am criticized for my "lack of research". Sorry I touched a nerve; but all of these issues are decisions that are being made, to not expect critiques of your decisions is naive.

Anyways. Having done my research:

> You don't see that text but instead text indicating the formula has been deleted.

If, and only if, you know the exact name of the package. If you do a search, you get the "no formula found". If you attempt to use the braumeister.org web site, it also will not show up unless you explicitly craft the recipe URL.

> They will be preserved in the Git history for as long as we continue to use Git.

How does one go back into git history and revive deleted formula, given how automated the entire process is around ensuring the most recent version of brew is always in use? By your "deleted" help docs, I infer that it comes down to creating your own cask?

The one example I came across quickly is "abi_compliance_checker'. It was deleted because it requires GCC 4.7. Yet Homebrew is still quite capable of installing and using [email protected]. Not that it was broken in a recent build. Not that it was a major maintenance burden - the updates for years consisted of version updates.

This isn't something I need, but it's a great (and quickly found) example of seemingly arbitrary deletion of an otherwise active project.

Re:HomeBrew Analytics – top 1000 packages installed over las... by apostacy on Monday September 04, 2017 @10:29PM (#15172095)
Well I happen to agree with your sentiments. I've been meaning to make a fork of homebrew that doesn't have these problems. You should checkout out tigerbrew.