News for hackers. Stuff that matters.
Re:How we raised $3M for an open source project by malisper on Saturday June 06, 2020 @12:31AM (#23436842)
For those not familiar with Posthog, it's open an open source product analytics tool. That means you install some javascript on your website. It then collects information on what features people are using on your website and let's people run charts and graphs to ask questions about feature engagement and product usage.

Product analytics is a pretty large space. The three main existing product analytics tools are Mixpanel[0], Amplitude[1], and Heap[2] (I used to work at Heap). Notably, Mixpanel was valued at $865MM as of 2014 and Amplitude was valued at $1B as of two weeks ago.

Posthog is an open source competitor to these tools that you are able to self host. This is interesting for several reasons. If you don't feel comfortable sending data to third party tools, you control the data yourself. In addition, existing product analytics tools can get expensive. After you get off the public pricing, you will likely find yourself paying >$12k/year for one of these tools.

There are also challenges I see for them going forward. In my experience, the person who makes the decision to buy a product analytics tool isn't in engineering. It's usually a product manager or sometimes a marketer. I imagine the fact that it's open source is less interesting to one of the traditional product analytics purchasers. Compare this to a company like Gitlab where the end users are engineers.

Scaling these tools is also a huge challenge. My job at Heap was to basically scale Heap. If you want to self-host a product analytics tool, you will basically be responsible for scaling the system. On top of that, because each Posthog instance is isolated from each other, Posthog won't be able to take advantage of the shared compute power. Imagine a product analytics tool that has 100 customers and 100 servers. If the compute power was shared, each query would make use of all 100 servers. If the servers were isolated from each other, each query would only be able to make use of 1 server. That's basically a 100x slow down in performance. Posthog does sell a managed version which in theory shouldn't have this problem. It will be interesting to see long term whether that's a big driver of revenue for them.

[0] https://mixpanel.com/

[1] https://amplitude.com/

[2] https://heap.io/

Re:How we raised $3M for an open source project by james_impliu on Saturday June 06, 2020 @05:29AM (#23437886)
(I'm the person that wrote the article)

A few thoughts:

As you say, the thing that's really different about us is exactly that we are focussed on engineers not PMs with our tool. PMs are of course welcome to use it too, but it's a little more technical feeling.

We felt the people building the thing should have the context of usage data and this shouldn't only sit in another team - which was a behavior we saw happening quite frequently when we did user interviews early on.

We started with the fundamentals of product analytics with some features we wanted ourselves but are now focussed on features that are much engineering specific. For example, we are about to release (an optional) "inspect element" for usage data: https://github.com/PostHog/posthog/issues/870, so you can pop that up whilst working on localhost.

We had no idea if that hypothesis was correct - that engineers would care. We did a launch HN to find out and got quite a lot of good feedback and growth.

Re scaling, you are spot on - this is definitely hard! We have users doing 5 million events/day on Heroku's cheapest standard tier dyno. We offer paid support for people who need something higher volume still if it doesn't work well out of the box. We're working on supporting databases other than Postgres as people need them.

Re:How we raised $3M for an open source project by malisper on Saturday June 06, 2020 @10:12AM (#23439305)
> We felt the people building the thing should have the context of usage data and this shouldn't only sit in another team - which was a behavior we saw happening quite frequently when we did user interviews early on.

What is preventing the engineers from getting access to that data? Existing product analytics tools usually don't charge by seats, so if they wanted to look at the data, they should be able to get access to Mixpanel, Amplitude, etc.

Compare that to Posthog's pricing of $25/user/month which incentives you to minimize the number of people at your company that have access to Posthog.

> We had no idea if that hypothesis was correct - that engineers would care. We did a launch HN to find out and got quite a lot of good feedback and growth.

Heap also found a lot of success on HN when they launched: https://news.ycombinator.com/item?id=5424206. While there was a lot of excitement early on, the kind of crowd it attracted wasn't a good demographic to sell to long term.

> We have users doing 5 million events/day on Heroku's cheapest standard tier dyno.

The challenge isn't ingesting the data. Especially with an analytics tool where you can turn off synchronous_commit so ingesting an event doesn't require writing to disk. The bigger challenge is around processing queries that span years of data. Queries get proportionality slower the more data you've collected. A query over 12 months of data is going to be 4x slower than a query over 3 months.

Re:How we raised $3M for an open source project by hitekker on Saturday June 06, 2020 @10:57PM (#23444403)
> Heap also found a lot of success on HN when they launched: https://news.ycombinator.com/item?id=5424206. While there was a lot of excitement early on, the kind of crowd it attracted wasn't a good demographic to sell to long term.

I hope posthog's founders heed this crucial implication: the paying customer for analytics are analysts, not engineers.

Analysts want analytics that work. Engineers want analytics to go away. An analyst wants to augment their data analysis, in order to produce more conclusions or support more decisions for the decision makers they report to. An engineer wants to focus on building technically challenging, long-lived systems, not on churning out short-lived weapons in the justification-explanation wars waged between Data Science, Business Intelligence, Product Management and the C-Suite.

To use a market analogy, the engineer-as-a-customer will pay on cost, not value. Analytics are necessary but secondary to their day-to-day work: for most, it's just a one-line telemetry SDK logging call. If an engineer builds a feature doesn't lead to an uptick in a business metric, that engineer can shift to another org with little fuss. The analyst, on the other hand, is tied to proving the performance of their feature or subject-of-analysis. The longer an analyst has to wait for engineering to build a tool to analyze (sell) a feature, the riskier their position is in the company. Analysts like Product Managers or embedded Data Scientists or BI experts, will accordingly pay a value-based price for analytics. They need it for their jobs.

Posthog will likely work for companies where engineers are also analysts. But in larger companies where the roles are more clearly divided, I forsee posthog's current focus leading them astray.

Re:How we raised $3M for an open source project by mritchie712 on Saturday June 06, 2020 @10:53AM (#23439585)
> What is preventing the engineers from getting access to that data?

In addition, should they even be spending a lot time looking at it? There is a reason companies hire PMs and data analysts (which are cheaper than engineers). I'd totally encourage engineers to understand the business, but at a certain point, they should probably get back to engineering.

Re:How we raised $3M for an open source project by billme on Saturday June 06, 2020 @07:42AM (#23438485)
Curious, in the article, you mentioned - “we believe that open source will eat SaaS's lunch in many product categories” — what to you is the core filter for “SaaS vs Open Source” product market fit?
Re:How we raised $3M for an open source project by james_impliu on Saturday June 06, 2020 @07:52AM (#23438528)
I think the closer your product is to something that developers can use the stronger the OS proposition. If there are other value props (privacy or cheaper at scale) that can definitely help though.
Re:How we raised $3M for an open source project by smithmayowa on Saturday June 06, 2020 @03:00AM (#23437366)
Analysis like this is why I love hackernews, thank you.
Re:How we raised $3M for an open source project by mohitmun on Saturday June 06, 2020 @02:39PM (#23441354)
Just wanted to highlight that heap's engineering blog is one of the best I have ever come across. Love all articles by Kamal and Michael malis
Re:How we raised $3M for an open source project by malisper on Saturday June 06, 2020 @02:44PM (#23441389)
I am Michael Malis :)
Re:How we raised $3M for an open source project by fergie on Saturday June 06, 2020 @03:03AM (#23437376)
Stupid question: Can't you do all of this in Google Analytics? What are the advantages of Posthog/Mixpanel/Amplitude/Heap over GA?
Re:How we raised $3M for an open source project by kevsim on Saturday June 06, 2020 @03:13AM (#23437405)
Not a stupid question. These products tend to offer a lot more functionality and flexibility in terms of how you consume your data. Different analysis possibilities, fancy ML models, custom dashboards, etc.

Also more control over how the data is ingested (sampling or not, etc.)

These products feel more suited to teams building rich apps/webapps where as GA seems to have its roots in sites that are more "content-based" (news sites, etc.).

And for many, not sending the data to Google is an advantage in and of itself.

Re:How we raised $3M for an open source project by dotandgtfo on Saturday June 06, 2020 @04:44AM (#23437666)
I'll also add that product analytics software is built more towards measuring depth of engagement: How many times has this user returned to our site? What is the average LTV of this cohort? What is cohort X's n-day retention compared to cohort Y? What features have correlation to those differences? Did this experiment lead to improvements in retention - 30 days later?

Google isn't built to go that deep. Sure you can see that 3% of users have used feature X, but you can't really effectively dig in and see how upstream or downstream actions and events influence each other. Sure you can create some custom segments on a sessions/user level - but that quickly turns complex and unwieldy if you have several segments, cohorts and funnels. Also there are a lot of charts that are just plain better in product analytics tools. Retention charts, funnels and path diagrams are obvious examples.

Re:How we raised $3M for an open source project by labelbias on Saturday June 06, 2020 @05:18AM (#23437802)
At my previous employer we were using GA and it's very flexible. We were able to track all DOM interactions by just configuring the included js. Although, querying that later with tools engineers can use and build is a whole different story.
Re:How we raised $3M for an open source project by malisper on Saturday June 06, 2020 @10:21AM (#23439360)
There's two issues with GA. First the functionality is somewhat bare bones. It does a really good job out of the box, but it's pretty hard to use beyond that. Take the funnel feature. You need to define your funnels ahead of time before you can start analyzing them. Compare this to any of the other product analytics tool where as long as you were collecting the events up front, you can compute any funnel you want.

Second, once you cross the free tier, you will have to pay at least $150k/year. The free tier is 10 million hits per month[0], which is high enough that most businesses never hit it. Other analytics tools often come in at a better price point at that kind of scale.

[0] https://marketingplatform.google.com/about/analytics/terms/u...