News for hackers. Stuff that matters.
Re:Lessons from Building and Scaling Reddit's Ad Serving Pla... by katzgrau on Monday September 10, 2018 @08:17AM (#17951212)
As the founder of a SAAS ad serving company, I can say that as slick as (I think) my platform is, it's not a problem I would want to tackle again. There will be a large volume of technical challenges and required features to support.

It'd be like writing Apache or Nginx from scratch. It seems simple on the surface and then you wake up to all kinds of inconvenient and unexpected technical challenges.

I'm not sure Reddit's engineering resources are best allocated to building ad serving infrastructure and software. It was going to be an ongoing headache from the start, and worse, it sounds like it was build with a language and stack that was fairly new to the team.

My prediction is that it will become a high cost and difficult to maintain project that will also prove difficult to hire for when this engineering team gets tired of working on the (currently new and fun, soon to be old and scary) adserver and moves on to other jobs.

My point being that some challenges are best left to third parties who make solving some problem its full time job. I truly do not say that out of bias, just experience building and managing such a complex system.

Re:Lessons from Building and Scaling Reddit's Ad Serving Pla... by grey-area on Monday September 10, 2018 @09:08AM (#17951509)
They were already using a third party service. It was slow and opaque, so they ditched it.

At the scale of reddit, and because this is how they make their money, this is clearly something they want in house and something it would be a competitive advantage to control, if nothing else it gives them a much clearer conduit to their advertisers, and much greater control over things like ad selection.

Re:Lessons from Building and Scaling Reddit's Ad Serving Pla... by supernovae on Monday September 10, 2018 @10:06AM (#17951874)
The problem with Reddit ads isn't their tech, it's their people. The ad approval process is a farce - and human-managed. Oh, and those humans are assholes. You have to go through many denials - for no reason - submit a ticket to get another approver and approved - for no reason stated. It was the most upsetting process I've ever dealt with.
Re:Lessons from Building and Scaling Reddit's Ad Serving Pla... by pjc50 on Monday September 10, 2018 @10:47AM (#17952190)
It seems odd that there's a stringent ad approval process when the site proudly lets Redditors post any garbage they want. I'd have expected anything SFW and not obviously fraudulent to be allowed.
Re:Lessons from Building and Scaling Reddit's Ad Serving Pla... by MoronInAHurry on Monday September 10, 2018 @01:10PM (#17953278)
It's obviously not very stringent. A lot of the ads they run lately are outright scams, including even advertising Scientology: https://i.redd.it/6cd1ricqc8j11.jpg
Re:Lessons from Building and Scaling Reddit's Ad Serving Pla... by setr on Monday September 10, 2018 @01:49PM (#17953556)
It sounds bureaucratic, not stringent. Lots of forms to fill out, and a lot of order, and probably not even many requirements to meet (but a lot of different ways saying you did meet them) but lost its purpose a long ways back.
Re:Lessons from Building and Scaling Reddit's Ad Serving Pla... by yourbandsucks on Monday September 10, 2018 @10:12AM (#17951914)
This is the QA process that yields us precocious-looking stock photo 'MIT grads' trying to sell me a wine subscription on every other pageload?
Re:Lessons from Building and Scaling Reddit's Ad Serving Pla... by zeveb on Monday September 10, 2018 @10:50AM (#17952207)
reddit has ads? I hadn't noticed …

More-pertinently, I wonder how many of the developers of reddit's new ad system themselves have ad blockers installed. Do any of them actually see the results of their work?

Re:Lessons from Building and Scaling Reddit's Ad Serving Pla... by rbobby on Monday September 10, 2018 @12:06PM (#17952744)
Reddit ads were what finally pushed me into install ublock origin about a year ago (redditor since 2007).

They had a misbehaving ad (downloaded a ton of "stuff" over and over and over). I reported it on one of the their support subreddits and was told "don't recognize the add urls. scan your system for a virus". It was a pretty good error report too (screen shots, console log, etc).

Installed ublock origin that day and haven't looked back.

Re:Lessons from Building and Scaling Reddit's Ad Serving Pla... by jedberg on Monday September 10, 2018 @11:22AM (#17952444)
FWIW, when I worked at reddit, I never ran an ad blocker. In fact, I didn't run one for about five years after, until my CPU started spiking on just about every web page I visited.

But even then, I whitelisted reddit.

So my guess is yes, they see the output of their work.

Re:Lessons from Building and Scaling Reddit's Ad Serving Pla... by cco on Monday September 10, 2018 @10:41AM (#17952142)
What were you trying to advertise out of curiosity?
Re:Lessons from Building and Scaling Reddit's Ad Serving Pla... by kadendogthing on Monday September 10, 2018 @10:57AM (#17952263)
This is a pretty emotionally driven comment and would do well to provide some sort of concrete story behind why the ad approval process is a farce and why this particular group of individuals happens to be full of assholes?
Re:Lessons from Building and Scaling Reddit's Ad Serving Pla... by supernovae on Monday September 10, 2018 @12:27PM (#17952914)
I advertised for a travel site in the travel reddits. I never got a single ad approved "out the door" - only after I opened a ticket and asked why they were not approved did someone actually respond and approve it. The denial was within a few minutes of submission and the approval only after I opened a ticket in which case the response was "We're sorry, we've approved the ads" - but no one ever answered the question of editorial review.

It may appear "emotionally driven" because my experience with reddit ads has been terrible. I struggled through it, i followed the example ads, i created very topical ads and when i finally got all approved and done, they just didn't perform well either. I talked with peers in the industry and they all avoid reddit ads for the most part because of similar experiences. I haven't spent any money with ya'll since that bad taste experience.

for me, it would just be easier to site target reddit through google and do adwords for desktop/web traffic.

The editorial review literally feels like some redditor judging your ad on personal opinion vs an editorial review for passing some kind of targetable creative process. That's a terrible way to treat paying customers and i don't that experience on any other market i advertise on.

Re:Lessons from Building and Scaling Reddit's Ad Serving Pla... by katzgrau on Monday September 10, 2018 @12:46PM (#17953071)
I agree that it's important to keep the money-making component of a business free from external dependency to a large degree, but I think there's some liability that's being ignored here. A 3 person team built such an important piece of Reddit in a language they never used in production? I hope the team was at least experienced in ad-tech.
Re:Lessons from Building and Scaling Reddit's Ad Serving Pla... by kaeshiwaza on Monday September 10, 2018 @03:25PM (#17954367)
Maybe they have other future projects with Go ?
Re:Lessons from Building and Scaling Reddit's Ad Serving Pla... by ckdarby on Monday September 10, 2018 @09:04AM (#17951479)
It seems biased. I had a similar situation with a client of mine. They went out looked at 3rd party companies, and everything looked decent on paper.

Internally there were some napkin math and found out while the 3rd parties would save development time the returns of implementing features their customers were asking for now instead of waiting for 3rd party company paid for them to hire 3x the engineering staff the 3rd party company even had.

Size matters, Reddit is at the scale of tiny percentage gains, if they control everything and can drop a standard 90ms down to 10ms and that converts to a couple of percentage points on their revenue it is worth it.

Talking from the experience of building out a system that handled +1B impressions a day, the actual ad serving isn't challenging at all these days. Reporting and ad setup from the user is much more challenging.

Re:Lessons from Building and Scaling Reddit's Ad Serving Pla... by katzgrau on Monday September 10, 2018 @12:01PM (#17952711)
> if they control everything and can drop a standard 90ms down to 10ms

10ms, after request processing, ad selection logic, and faster-than-light packet travel? I think that's far too optimistic of an improvement. As the post said, 30ms was the goal. My adserver delivers in 20-40ms. But since ads are routinely loaded asynchronously, is speed concern truly valid? The adserver can usually make a selection and deliver before the content is fully rendered in the browser.

> Talking from the experience of building out a system that handled +1B impressions a day, the actual ad serving isn't challenging at all these days.

If you developed a system that delivers 1BB impressions per day (most large scale ad networks deliver only a modest multiple of that number) and didn't find it challenging, then it's either a very simple system or you're Elon Musk ;) That's 12k requests per second, generally uncached, hitting all sorts of internal services most likely.

I'm just saying that I think Reddit's best bet is to offload the task to a company that works on these problems full time. Or acquire one like AOL and Facebook. Unless the use cases are so unique and key to the company's future plans, avoid building in-house. And if you have to, perhaps it would be a larger effort than a 3 person team developing the thing in a language that the company never used in production before.

Re:Lessons from Building and Scaling Reddit's Ad Serving Pla... by ckdarby on Monday September 10, 2018 @06:42PM (#17955797)
Welp, guess I'm "Elon Musk" ;).

I know it isn't the actual ad delivery itself but don't have public docs for that :(. Here is the ETL layer to back up claim: https://docs.google.com/presentation/d/13x_wdXA_C9ZymA2gcjsb...

Re:Lessons from Building and Scaling Reddit's Ad Serving Pla... by katzgrau on Monday September 10, 2018 @07:06PM (#17955891)
Total cakewalk. Impressive!
Re:Lessons from Building and Scaling Reddit's Ad Serving Pla... by wink on Wednesday September 12, 2018 @10:50AM (#17969332)
30ms really sounded a bit much for me, if you take network-locality as a precondition (i.e. adserver on same physical sub-1ms-network than webserver).

But yeah, it should be loaded async. Been a while since I worked with ad stuff, but I thought 10ms-20ms sounded more likely.

Re:Lessons from Building and Scaling Reddit's Ad Serving Pla... by katzgrau on Saturday September 15, 2018 @09:52AM (#17994283)
If it was straight ad delivery I'd agree that being closer to 10ms is achievable. But in the process of pulling in ad data, user profile data, running it through a selection algorithm that can be pretty complex for even a simple adserver, calculating and plugging in macros, gzipping, and doing that thousands of times per second per server, the latency tends to creep upward.

More/faster hardware can obviously help, but since everything's async from the browser's perspective you tend to run into higher costs and diminishing returns by pushing for the lowest latency possible.

Re:Lessons from Building and Scaling Reddit's Ad Serving Pla... by ChrisSD on Monday September 10, 2018 @11:10AM (#17952350)
Out of interest, how did those option compare with the napkin maths for paying a 3rd party company to include the needed features in their product?
Re:Lessons from Building and Scaling Reddit's Ad Serving Pla... by jacquesm on Monday September 10, 2018 @08:35AM (#17951316)
They could have gone with Iponweb[1] or something like that but at Reddit's scale it might actually (for a change) make sense to roll their own.

They're one of the largest websites in the world now, so what holds for 'ordinary' websites and their best practices may simply no longer apply to Reddit.

The whole Reddit saga for the last few years is quite the story of turnaround management done well.

[1] https://www.iponweb.com/

Re:Lessons from Building and Scaling Reddit's Ad Serving Pla... by katzgrau on Monday September 10, 2018 @12:36PM (#17952989)
Yeah I was thinking that if what they need or are planning is truly unique or just needs to be homegrown (a la FB's ad system tightly integrated with first party data), maybe it really does make sense. Or maybe any alternative third party company they engaged tried to pitch them a huge/prohibitive contract.

But then I was thinking that if it's going to be so important, why a 3 person team with no experience running Go in production?

Re:Lessons from Building and Scaling Reddit's Ad Serving Pla... by somethingroma on Monday September 10, 2018 @08:21AM (#17951237)
Honestly I think you make a good point and implicitly they could have used those resources instead to improve the user experience (for example, improve the sites efficiency and load times)