News for hackers. Stuff that matters.
Re:How We Built r/Place by firebones on Thursday April 13, 2017 @10:45PM (#14112398)
I wish I had enough spare cash to take you up on a bet that you could implement this in less than a weekend. Maybe HN can crowdfund paying you for two days at $1000/day to replicate, versus your time and a $3000 rebate if you can't.
Re:How We Built r/Place by josephg on Thursday April 13, 2017 @11:42PM (#14112583)
My loved ones are out of town, so you're on! I'll do it for the fun of it and the experience. And because I still haven't built anything on top of kafka, but there's no time like now to fix that.

The challenge is: Meet the main requirements of r/place: 1000x1000 image, a web based editor, 333+ edits / second, with an architecture that can scale to 100k simultaneous users (although that part will be hard to actually test). I won't implement mobile support or notifications, and I won't implement any sort of user access control (thats out of scope). The challenge is to do it & get it hosted online before I go to bed on Saturday (so, I'm allowing myself some slop there). But its already 1:30pm on Friday, so I think that easily qualifies as "less than a weekend".

As a stretch I'm going to write all the actual code in nodejs while aiming for well into the thousands / tens of thousands of writes per second territory.

I'm willing to accept some fun stakes if anyone wants to propose them (ice bucket challenge level stuff). I'll live-tweet progress here - https://twitter.com/josephgentle

(Edit: fixed days)

Re:How We Built r/Place by josephg on Friday April 14, 2017 @11:46AM (#14115110)
12 hours in and the site is up: https://josephg.com/sp/

Polish and perf tuning tomorrow.

Code: https://github.com/josephg/sephsplace (warning: contains evil)

Re:How We Built r/Place by josephg on Saturday April 15, 2017 @04:02AM (#14119869)
Complete and working. I haven't tested the load for real, but judging by the performance numbers it should be able to handle 100k connections fine with 2 beefy backend servers. (40% of one of my laptop's cores handles 15k connections with 400 edits per second. So about 8 cores should handle 100k users no sweat.) The bottleneck is fanout, so until kafka starts hitting subscribe limits it should scale linearly with frontend servers. (And kafka is a long way from hitting that limit because I've added write coalescing.)

I'll post a writeup about it all once I've eaten some real food and had a rest. But for now, enjoy!

Re:How We Built r/Place by socmag on Saturday April 15, 2017 @05:34AM (#14120066)
I like what you are doing a lot. It doesn't matter that it isn't a one to one comparison, the point is you are taking on the challenge and enjoying it. Probably actually you are learning some stuff along the way as well.

It would be kind of neat if the place experiment could become something of a micro-benchmark for online customer facing distributed platforms.

Keep up the good work man, it's looking good.

Don't forget to hydrate and consume pizza.

Re:How We Built r/Place by renlo on Friday April 14, 2017 @12:32PM (#14115392)
Very cool man. For the real-time updates you keep the browser-client requests open, right? Any reason you did it that way?
Re:How We Built r/Place by josephg on Friday April 14, 2017 @12:52PM (#14115513)
Well, the server needs to message the client with updates somehow. Either the client keeps a connection open or it has to periodically poll the client. Keeping a connection open is faster and it uses less system resources.
Re:How We Built r/Place by always_good on Friday April 14, 2017 @11:47AM (#14115120)
12 hour amphetamine binge can produce some good work!
Re:How We Built r/Place by developer2 on Friday April 14, 2017 @11:01AM (#14114844)
>> I won't implement mobile support

Mobile support was a huge factor for /r/place. To reimplement the project while completely ignoring mobile is basically taking out maybe 30% of the effort required. It's a bit of a copout to claim "can implement in a weekend", while discounting one of the main reasons why it is nearly impossible to hammer out solo in a single weekend.

The other "big ticket item" is integrating such a system into an existing live stack. It's very convenient that you get to pick the tools you think are most efficient for the job from scratch; it's quite another to restrict yourself to an existing set of tech.

>> an architecture that can scale to 100k simultaneous users (although that part will be hard to actually test)

Also this. Reddit had a single opportunity to deploy live. You can't throw out many days of required dev/staging benchmarking for a system that must work on first deploy. You're not "done" after a weekend unless your first deploy both works 100% and scales as required.

tldr; Were I to do what you are doing, it would be out of interest and to put it live for others to use. What seems disingenuous is to do so with the primary goal of proving that "see, child's play - it's not difficult at all!". The goal should be to build something that works, not to rush the job as some way to show off and take something away from Reddit's efforts.

Re:How We Built r/Place by josephg on Friday April 14, 2017 @11:45AM (#14115100)
Absolutely! And I'm not going to claim otherwise. I said above I think I can only implement it in a weekend if I can work to my technical strengths. Doing otherwise would mean learning, which is super slow compared to typing. Just implementing to a closed spec cuts out maybe half of the work of something like this, because you don't need to figure out the requirements or waste time going down blind alleys.

From my post above:

> But I couldn't implement r/place that quickly if reddit didn't already do all the work deciding on the scope of the problem, and what the end result should look like.

Re:How We Built r/Place by developer2 on Friday April 14, 2017 @12:40PM (#14115447)
Upvoted you. I'm a very cynical person, and thus I focused on the "weekend" aspect as being more of an attempt to refute the claim that Reddit had to put in quite a bit of effort to accomplish what they did, rather than you simply limiting how much time you're willing to sink into it.

If anything, this only increases my motivation to replicate the project myself, whether it's during a weekend or two full weeks. It's interesting enough and at the right level of complexity - kind of simple, but not too simple - to make it a fun side project.

Re:How We Built r/Place by didibus on Friday April 14, 2017 @03:01AM (#14113060)
Yes!! Love me a good show. I'll be following your tweets.
Re:How We Built r/Place by SomeStupidPoint on Friday April 14, 2017 @12:01AM (#14112631)
Wait... Where are you that it's Saturday already?

I thought I knew timezones... But that's 40+ hours ahead of me.

Re:How We Built r/Place by josephg on Friday April 14, 2017 @12:08AM (#14112650)
Errr... oh, that thing is happening where its a public holiday so I'm a bit unmoored from reality. I'll get it done by Saturday night, not Sunday night.
Re:How We Built r/Place by employee8000 on Friday April 14, 2017 @02:52AM (#14113036)
Unless you stick with a single Kafka partition (which might not withstand the load) I don't think you can consume the events in exact chronological order.
Re:How We Built r/Place by josephg on Friday April 14, 2017 @03:54AM (#14113204)
A single kafka partition should be able to handle the load - it only needs to write 333 ops/second, and I could coalesce several edits together over half second windows if I wanted to make that more efficient. And I'll only have as many kafka consumers as I have frontend servers. So we're talking about 333 ops/second and like, 5 consumers of those operations. Somehow I think a single kafka partition should be able to manage the load :)

But there's no reason I couldn't use multiple partitions if I had to. To make it fully consistent I'd just need a consistent way to resolve conflicting concurrent writes on different partitions. Either adding a server timestamp to the messages, or using the partition number to pick a winner would work fine for this. It would just make the server logic a little more complicated.

Re:How We Built r/Place by xapata on Friday April 14, 2017 @01:05AM (#14112792)
You can simulate the users for load testing. Have them random walk from an arbitrary start point, changing a pixel every 10-20 minutes.
Re:How We Built r/Place by josephg on Friday April 14, 2017 @01:13AM (#14112818)
Simulating hundreds to thousands of writes per second via post requests will be easy. Simulating 100k simultaneous readers of an eventstream will be much harder. I don't know of any tools which can load test that many readers. Do you have any ideas?
Re:How We Built r/Place by scrollaway on Friday April 14, 2017 @03:51AM (#14113198)
locust can spawn arbitrary scripts; you'd have to write your own eventstream reader, and simulating 100k of them will take a few machines concurrently load-testing, but i think this should be the easiest way to do it.