JasonPunyon

Thanks for taking it for a spin! I'm working on why this is slow now.

You may remember a carbon copy of this event from a year ago. https://meta.stackexchange.com/questions/389922/june-2023-da...

Discussion from then https://news.ycombinator.com/item?id=36257523

If anyone wants their data back in a way they can use it, it's right here https://seqlite.puny.engineering

And I'd be remiss if I didn't point out that their trade dress is MIT licensed. https://stackoverflow.design

Have fun.

Thanks so much, that’s really kind.

Good catch! Fix going up now.

Nice post!

Yep, my process is similar. It goes...

  - decompress (users|posts)  
  - split into batches of 10,000  
  - xsltproc the batch into sql statements  
  - pipe the batches of statements into sqlite in parallel using flocks for coordination

On my M1 Max it takes about 40 minutes for the whole network. Then I compress each database with brotli which takes about 5 hours.

This site is on a Cloudflare R2 bucket because (and only because) they have free egress. While not datacenter sized some of these files are large. Just opening up your credit card to 10 cents a gigabyte will be a bad time anywhere else.

I mean fine, write an article that overweights recent events to weave a tale of the decline of Stack Exchange...but don't leave out the moderator strike, the data dump fiasco, or the CEO that simply doesn't get it. Jon Ericson's writing on the issue is much more informative. https://jlericson.com/

PCA/SVD aim for maximizing explained variance, not preserving distance. They tend to "preserve" large distances at the expense of smaller ones, but that's not an explicit goal, nor can you bound the distortion. The [answer here](https://stats.stackexchange.com/a/176801/60) gives a pretty good intuition about why.

You can also compare them on computational complexity, where random projection (O(numPoints * numOriginalDimensions * numProjectedDimensions) smokes PCA or SVD which are cubic in the number of original dimensions.

And then there's simplicity. The random projection method turns on sampling from a normal distribution and then doing a matrix multiplication. There's a whole lot more about PCA to understand (standardizing your data, calculating the covariance matrix, eigenvector decomposition). I doubt I could implement it correctly myself, and I surely couldn't do it in high dimension.

Nothing is dying. Our data dumps are live at https://archive.org/details/stackexchange as they've always been.

Not everyone wants to work at Google. I know I don't.

This is the standard stuff from our company page (http://stackoverflow.com/jobs/companies/stack-overflow)...

20 days vacation (to start)

Flextime

No Copay Health Insurance

Whatever workstation you want (PC/MAC/Build Your Own), Monitors out the wazoo. Seriously, I have 128GB of RAM and a GTX 1080, we don't skimp here.

$3,500 a year + 3 days conference budget

Gym Reimbursement Tuition Reimbursement

In NYC and Denver we have in house chefs

Unlimited sick time

Fully Paid Parental Leave

...but our people team is pretty empowered to make you happy.

And yeah, there's equity.

We have offices in NYC, London, and Denver. If I'm remembering correctly, I can count the number of developers who've left the company on one hand. We currently have ~20 engineering staff in the NYC office. Retention doesn't seem to be a huge problem.

I'd also like to point out that...we're a largely remote company. We don't index our salaries to the most expensive time and place to hire developers ever. We don't have to support Bay Area rent, or duel it out with AppAmaMicroGoog to keep our people from getting poached. We hire people everywhere in the world and through that lens, we're competitive. I live like a king in Buffalo, NY.

What about the millions of people who've never written anything on Stack Overflow?

Hey, I'm the Jason Punyon drob's gushing about in the post. I wanted to tell you that this is 100% totally realistic, everyone on the Data team at Stack Overflow besides Dave Robinson is living proof.

In the beginning the Data Science Team at Stack Overflow was just me and Kevin Montrose. We're both ~30, we had some skills that were data-ish, but they were general critical thinking and math skills. By no means were we statisticians or data scientists by trade like Dave Robinson. However, the data at Stack Overflow was very amenable to analysis, and we were able to ship a personalized machine learning product after a year of work.

That year was plagued by (my, mostly) engineering failures (http://jasonpunyon.com/blog/2015/02/12/providence-failure-is...) and a bit of unfortunate lack-of-empiricism, not "data science" problems. And post launch of Providence, the engineering remains the bigger issue. It's just hard to get this shit right. Our record on getting experiments right the first time is spotty, and at Stack Overflow levels of traffic you might be dumping a week or more of time on a single screwup. We've had to rerun many experiments because of various things going wrong with the code (screwed up the A/B testing code, not counting the right things, visual issues with the ads we didn't catch during design, etc etc etc). There's a litany of things you've gotta get right engineering wise before data even comes into the picture.

So if you're gonna take the plunge, know that it's 100% possible but 100% difficult, and maybe not for the reasons you think.

Again, up to the individual teams to decide what's right for them. The most globally applicable principle we have is "Make sure to use your brain."

Currently most teams don't subscribe to a particular methodology, though there is a Scrum effort happening on one team.

Long time dev at the company here.

Yes, Yes, and No.

#1 kind of happens naturally as we're all working on different things. There are "a lot" (< 100) of people slinging code/design/SRE/IT, but only a few work in the same areas at the same time and rapid good communication generally happens among those subsets of the overall team. We also seem to have perpetuated a tendency toward good citizenship, so we generally talk to people before traipsing through areas of the code we aren't familiar with.

#2 Absolutely. This is some basic principle stuff. We don't hire people to not trust them.

#3 Related to #2: Automated acceptance testing is done as is deemed appropriate by the person developing the system. I've been on teams that valued and developed automated testing more, and less. My personal experience at the company has been it is neither necessary nor sufficient for success.

Much more important than any pre-deploy automated testing is our effort to monitor what's deployed (both in terms of software/hardware metrics and business goals). Bosun (http://bosun.org), developed by our SRE team, gives us some pretty great introspection/alerting abilities. I'd be incredibly sad to not have it. Bosun monitoring combined with the ability to have a build out in <5 minutes keeps me pretty happy.

A new site may come to replace it, but what of the 10MM+ questions and scads more answers created by the Stack Overflow community over the last 7 years? Without the CC license all that hard work gets lost in the case SO goes rogue.

We use that license because it protects the content from us. No matter who comes along to run Stack Overflow in the future, Stack Overflow can't do something like put up a paywall and lock it up. Someone else'll just be able to host a copy.