fullung

Albert here :) Decision basically came down to the following: we needed to build a 1 million events/sec log processing pipeline, and had only 5 servers (more were on the way, but would take many months to arrive at the data center). So v1 of this pipeline was 5 servers, each running Kafka 0.8, a service to receive events from the edge and put it into Kafka, and a consumer to aggregate the data. To squeeze all of this onto this hardware footprint, I spent about a week looking for a format that optimized for deserialization speed, since we had a few thousand edge servers serializing data, but only 5 deserializing. Capnproto was a good fit :)

Would love to see results from incorporating https://github.com/segmentio/encoding/tree/master/json!

Segment | Engineering | San Francisco, CA & Vancouver, BC | Full-Time | Onsite + Remote for senior engineers

Segment is building the customer data platform for everyone. We transform data and route to over 200 different integrations, adding new ones every day. We're processing billions of events daily and maintain the analytics infrastructure for companies like DigitalOcean, Instacart, New Relic, and Docker. Our goal is to help companies learn from how their users interact with the products to build even better products. We also like to share our work and what we learn, here are some recent examples:

- https://segment.com/blog/the-tools-of-today-arent-the-tools-... - https://segment.com/blog/show-hn-to-series-d/ - https://segment.com/blog/we-test-in-production-you-should-to... - https://segment.com/blog/secure-access-to-100-aws-accounts - https://segment.com/blog/why-status-codes-matter-in-data-del... - https://segment.com/blog/goodbye-microservices - https://segment.com/blog/2018-in-review/ - https://open.segment.com/

We have a modern stack consisting of Go (golang), AWS ECS/Kubernetes, Docker, JavaScript, React, GraphQL, Kafka, and others! If any of this sounds interesting, we'd love to hear from you! Check out our open positions at https://segment.com/engineering#jobs If you have specific questions, feel free to reach out to marta (at) segment.com (but please do apply on the site).

On this topic, shout-out to ykman oath (if you have a YubiKey).

Segment | Engineering | San Francisco, CA | Full-Time | Onsite

Segment is building the analytics routing layer for the world. We transform data into over 150 different services, adding new ones every day. We’re processing billions of events each month, and maintaining the analytics infrastructure for companies like DigitalOcean, Atlassian, New Relic, and Docker.

Our goal is to help companies learn from how their users interact with the products to build even better products. We also like to share our work and what we learn, here are some recent examples:

https://segment.com/blog/the-right-way-to-manage-secrets/

https://segment.com/blog/exactly-once-delivery/

https://segment.com/blog/a-brief-history-of-the-uuid/

https://open.segment.com

If any of this sounds interesting, we’d love to hear from you! Check out some of our open positions at https://segment.com/jobs or email [email protected] with some info about your background and we’ll kick it off from there.

Not rolling our own exactly: running on ECS now and busy evaluating Kubernetes. These are still quite blunt instruments though.

At some point, as a company grows, it's not just about running some containers. You want batch jobs, complex placement constraints, scheduled tasks, etc., etc.

If you haven't watched through https://youtu.be/4OLlKGT7aVQ, I can recommend it. aspyker does a great job of enumerating many of their complex use-cases.

The Titus team at Netflix is building this on top of EC2/Mesos and ECS.

The rest of us still need to do a bit of work to find/adapt/build our own.

I'll happily create a redirect once you build that for us. :)