News for hackers. Stuff that matters.
Re:Prolific Engineers Take Small Bites by minimaxir on Tuesday December 13, 2016 @11:33AM (#13168074)
This post is a good example of my current issues with modern data journalism as content marketing: you can say anything you want without evidence to back it up, but hey, there's a pretty chart, so it doesn't matter. (In this blog post, the data points don't correspond to actual data!)

This is also the primary reason I have switched to Jupyter/R Notebooks for my blog posts: if I make ridiculous claims, people can check my work as evidence. This post doesn't even provide any quantifiable metrics, just "we analyzed millions of commits."

Re:Prolific Engineers Take Small Bites by lamontcg on Tuesday December 13, 2016 @01:46PM (#13169422)
I just got done staring at some code written by a prolific upper quadrant coder, that I'm going to have to noodle on and figure out how to disentangle some bad design decisions. I might be able to do a fairly short commit to refactor and extend, in which case my metrics will be crap. I might wind up looking at all the outstanding issues with this codebase and rewrite every class in this directory (in which case my LoC metrics will improve dramatically). Right now I'm probably one of those "stuck" coders that need "help". Kinda? But most software developers hit problems like this and run away in terror, and looking around me IDK who can really help me that much, I'll run my ideas by some colleagues, but the other ones who work on problems this gnarly are also neck deep in their own issues...
Re:Prolific Engineers Take Small Bites by hinkley on Tuesday December 13, 2016 @06:52PM (#13172497)
Mikado Method might offer you some clues, or at least confirm some of your existing habits.

Basically, start a top-down refactor. When you find the bottom you stop, revert your code, and refactor from the bottom up.

You probably won't find the actual bottom, but in my experience the moment when I was about to give up and call it quits was usually just before I reached the bottom. Taking a break, and starting over from the spot where I stopped, usually the solution became obvious, and far less invasive than I expected.

Re:Prolific Engineers Take Small Bites by daveguy on Tuesday December 13, 2016 @09:19PM (#13173280)
Do I understand this correctly:

You start refactoring at a high level -- essentially set out to rewrite the whole chunk of code in which your refactoring work lies. And focus on the most important until you get to the inner-most structure.

Then when you are to the "sum(a,b)" level you start over with the original code base and refactor bottom up. With the knowledge of how it works all the way up to the top? Re-creating your original refactorings?

It seems interesting. Like tackling a math problem or puzzle from two directions. Are there certain situations in which it is most effective? Architectures? Paradigms?

Re:Prolific Engineers Take Small Bites by hinkley on Tuesday December 13, 2016 @11:08PM (#13173766)
Most refactoring is from the bottom up. Make the change easy, then make the easy change. I said "start a top down refactor" but redesigning from the top or all at once isn't really refactoring. Some of the actions look similar but you get few of the benefits of refactoring.

When the architecture problem has gotten away from you, or only becomes apparent late in the project, it's hard to see the trees for the forest. The only way to decompose the problem is to start somewhere and see what happens. With a top down rewrite the number of bits in flux becomes overwhelming, which is the anxiety I picked up from the previous commenter.

Mikado is just an exploratory development trick to help you find a way to make the change with refactoring. You try rewriting pieces until the number of concerns starts to multiply, you keep going essentially until your brain starts telling you, "this is nuts, you should stop", and you take that detail that broke the camel's back and do just that part, and build up and out from there.

Re:Prolific Engineers Take Small Bites by nerdy on Tuesday December 13, 2016 @03:00PM (#13170352)
Sounds like "Discovery" quadrant more than "Prolific" to me.
Re:Prolific Engineers Take Small Bites by user5994461 on Tuesday December 13, 2016 @12:09PM (#13168395)
> you can say anything you want without evidence to back it up

Actually, you can say anything you want with evidence to back up the opposite of it ;)

I have a hate-love relationship with journalists. It's both incredibly surprising and incredibly frustrating to discover what they end up publishing after an interview.

http://www.smbc-comics.com/?id=1623

Re:Prolific Engineers Take Small Bites by heisenbit on Tuesday December 13, 2016 @01:33PM (#13169262)
That is one of the reasons companies train people speaking to the press. Having a message and being articulate is often not enough.
Re:Prolific Engineers Take Small Bites by user5994461 on Tuesday December 13, 2016 @02:10PM (#13169737)
The thing is, it doesn't matter what you say. They'll just write anything they want ^^
Re:Prolific Engineers Take Small Bites by UseYrIllusion11 on Tuesday December 13, 2016 @02:27PM (#13169923)
They could retitle this "In Defense of Cowboy Coding". Engineers that code this way become the owners of their codebase because nobody else wants to deal with their insane decisions.
Re:Prolific Engineers Take Small Bites by tedmiston on Tuesday December 13, 2016 @03:21PM (#13170569)
Forget cowboy coding, this is cowboy data analysis. At best, it's misleading; at worst, it's dangerous.

This is embarrassing from a YC co.

Re:Prolific Engineers Take Small Bites by sitkack on Tuesday December 13, 2016 @06:22PM (#13172248)
They sell metrics, this blog post has us talking. This is a blog post, a PR piece, not a news paper story.

Source metrics are the new panopticon of Software is Eating the World.

Re:Prolific Engineers Take Small Bites by maxerickson on Tuesday December 13, 2016 @11:45AM (#13168177)
Is this even journalism?
Re:Prolific Engineers Take Small Bites by module0000 on Tuesday December 13, 2016 @12:01PM (#13168321)
> Is this even journalism?

I wondered this also, when the popups in the lower right kept presenting a chat box with "Can we help you with something?"(from gitprime). Do they want to sell me something, or do they want me to read their publication?

Re:Prolific Engineers Take Small Bites by geofft on Tuesday December 13, 2016 @12:56PM (#13168842)
Am I missing the sarcasm at some point in this thread, or something? It's very clear to me that this is not journalism, and shouldn't be read with the expectation that it conforms to journalistic standards, any more than some observation on distributed systems on a personal blog should be read with the expectation that it conforms to scientific standards.

It's a different type of content, which has a different kind of value. Are people trusting the claims the way they'd trust journalism from a known good media organization (or a paper from a known good research organization)?

Re:Prolific Engineers Take Small Bites by maxerickson on Tuesday December 13, 2016 @03:25PM (#13170612)
I wasn't being sarcastic, I was questioning the labeling of the use of charts and numbers as data journalism (This post is a good example of my current issues with modern data journalism as content marketing) in the comment I replied to.

If it had said something like tools of data journalism I wouldn't have been inclined to reply. A small difference, but enough for me.

Re:Prolific Engineers Take Small Bites by colechristensen on Tuesday December 13, 2016 @01:12PM (#13168997)
If it looks like a duck and quacks like a duck, it should follow journalistic standards. The fact that it obviously isn't is irrelevant. If you're going to pretend to be a journalist, do it right or don't do it at all.
Re:Prolific Engineers Take Small Bites by geofft on Tuesday December 13, 2016 @01:58PM (#13169582)
So, should I not post things about distributed systems to my blog without rigorous peer review, lest I get accused of pretending to be a scientist?

(I'm genuinely confused about how this person is pretending to be a journalist. It doesn't look like journalism at all to me, starting from the part where it's on some startup's blog. What am I missing?)

Re:Prolific Engineers Take Small Bites by joatmon-snoo on Tuesday December 13, 2016 @02:28PM (#13169939)
I wouldn't argue the journalism point, but it's very clear that the blog post is an attempt to (1) present solid statistics and (2) bolster their own credibility because "they have good statistics".

Except what we really have is a lack of data backing up the fancy non-informative charts (that clearly in no way shape or form correspond to the so-called millions of data points).

Re:Prolific Engineers Take Small Bites by colechristensen on Tuesday December 13, 2016 @03:12PM (#13170489)
It needs to be very clear from what and how you're presenting information the quality, source, and rigor of your data.

This blog post is pretending to be true.

I want to contrast it with this: another startup(ish) blog post about their own data

https://www.backblaze.com/blog/hard-drive-reliability-stats-...

That is a wonderful, useful, trustworthy source of interesting data.

The post source seems like made-up marketing fluff pretending to be real information. I'm looking at graphs I entirely believe were made up, I have nothing to support the conclusions, and I'm very concerned that a less wary reader would think it's true.

Re:Prolific Engineers Take Small Bites by minimaxir on Tuesday December 13, 2016 @02:26PM (#13169906)
There's a difference, IMO, between a blog post from a person in their spare time and a blog post by a startup whose main purpose is to get people to sign up for it.
Re:Prolific Engineers Take Small Bites by rexpop on Tuesday December 13, 2016 @03:00PM (#13170356)
Less of a difference, IMO, than that between either of those blogs and a reputable journalistic outfit.
Re:Prolific Engineers Take Small Bites by TeMPOraL on Tuesday December 13, 2016 @02:34PM (#13170028)
If you publish something, it's your responsibility to ensure what you write is true, to the extent reasonable within your skills and time available to you.

Doing less is harmful to your own reputation at best, but can be harmful to whoever believes in the things you got wrong.

Now that applies to mistakes. But for those who intentionally write and publish misinformation - I hope there's a special circle in hell waiting for them.

Re:Prolific Engineers Take Small Bites by rexpop on Tuesday December 13, 2016 @03:01PM (#13170363)
> skills and time available to you

Lots of plausible deniability, here.

Re:Prolific Engineers Take Small Bites by CamperBob2 on Tuesday December 13, 2016 @08:42PM (#13173110)
When a page starts popping up unsolicited chat boxes, the only conclusion I tend to reach is that they want me to go away. I'm happy to oblige.
Re:Prolific Engineers Take Small Bites by thebent on Tuesday December 13, 2016 @11:44AM (#13168151)
Would love to better expose our data set. It’s a bit tricky however: we’re specifically interested in enterprise software engineering (the patterns of which differ radically from the open source world).

In order to dig into enterprise data, we sort of need to have a ToS that only allows us to talking abstractly about aggregate data.

Any thoughts on how we might navigate that?

Re:Prolific Engineers Take Small Bites by minimaxir on Tuesday December 13, 2016 @11:47AM (#13168194)
You don't have to expose the raw dataset explicitly, but you do need to describe your methodology (moreso than an arbitrary "impact" score; give a formula) and give actual numbers for the analysis, and not just argue a "strong correlation" with a highly arbitrary matrix of attributes.
Re:Prolific Engineers Take Small Bites by arethuza on Tuesday December 13, 2016 @11:55AM (#13168271)
It's a marketing article - trying to explain their product to potential customers, not a scientific publication.

I'm not saying that you are necessarily wrong but I do wonder if making an article like this more like a formal publication will actually make it better or worse at it's real goal which is presumably to sell more.

Re:Prolific Engineers Take Small Bites by minimaxir on Tuesday December 13, 2016 @12:01PM (#13168324)
Yes, the standards of an marketing article are nonacademic, but you can't make a article containing a data driven argument and skimp on both the data and the argument.
Re:Prolific Engineers Take Small Bites by mannykannot on Tuesday December 13, 2016 @10:25PM (#13173540)
There is no downside to a legitimate marketing article to make the underlying data and analysis available to those who want to drill down. If it is valid, you stand a chance of persuading the skeptics.

More importantly, calling something a marketing article does not immunize it from critical analysis, and especially not on a forum such as HN.

Re:Prolific Engineers Take Small Bites by rexpop on Tuesday December 13, 2016 @03:03PM (#13170384)
I don't know if it's respectable to doll up a marketing article like journalism, let alone scientific research.
Re:Prolific Engineers Take Small Bites by jabagonuts on Tuesday December 13, 2016 @12:09PM (#13168387)
They go into slightly more detail on factors that influence the impact rating in a linked post. It still doesn't explain what formula or weightings they use. https://blog.gitprime.com/impact-a-better-way-to-measure-cod...
Re:Prolific Engineers Take Small Bites by iamwil on Tuesday December 13, 2016 @02:09PM (#13169719)
How do you publish them? I had always heard that publishing notebooks was a pain in the ass.
Re:Prolific Engineers Take Small Bites by minimaxir on Tuesday December 13, 2016 @02:24PM (#13169882)
GitHub can render Jupyter Notebooks natively.

For R Notebooks (which are HTML files), you can use GitHub Pages, which is super easy to set up nowadays, as the static files can now be located in the master branch.

Re:Prolific Engineers Take Small Bites by taeric on Tuesday December 13, 2016 @03:10PM (#13170472)
Github pages can also work with exported org pages rather easily. (For an example, here is one of my blog posts: http://taeric.github.io/Sudoku.html)
Re:Prolific Engineers Take Small Bites by tedmiston on Tuesday December 13, 2016 @03:23PM (#13170596)
That's misleading though — GitHub can render pre-run Jupyter notebooks but does not allow you to actually run them. IMO that's only half the battle.
Re:Prolific Engineers Take Small Bites by iamwil on Wednesday December 14, 2016 @12:53PM (#13178027)
When would you want Jupyter notebooks that you'd want to run, rather than just pre-render?
Re:Prolific Engineers Take Small Bites by huac on Tuesday December 13, 2016 @03:16PM (#13170524)
It's also easy to convert R notebooks to PDF, which are easily viewable on GitHub, if you don't value the interactivity highly.
Re:Prolific Engineers Take Small Bites by tedmiston on Tuesday December 13, 2016 @03:27PM (#13170630)
You need a host that serves Jupyter notebooks to actually run them vs render a pre-run notebook, which is what GitHub, Gist, etc do.

A few hosts that currently support that are PythonAnywhere and as of this month, Azure [1].

[1]: https://notebooks.azure.com

Re:Prolific Engineers Take Small Bites by mannykannot on Tuesday December 13, 2016 @11:01PM (#13173719)
They talk about 'small bites', but if their first chart means anything, its linearity shows that commits tend to be of the same size. As to whether those commits represent progress or repeated attempts to get it right, we have no idea.

As for their size measure, they are vague, but the comment about 'normal' commits raises the question of whether they have crafted a metric for which the linearity is built-in by definition.