jdotjdot

What is the legal plan you have on an annual basis? That sounds like something incredibly useful to have, where can you find one of those and how much do they cost? What do they cover?

I should clarify; I don't think that for the general case you have to go all-in on Materialize, but for the case in my comment--where you are effectively using business logic within Materialize views as the "source of truth" of logic across all of both your analytics and your operation--that requires buy-in. Additionally, if I'm _already_ sending all of my data to a database or to my data warehouse, ETLing all of that data to Materialize also is rather burdensome. Just because I technically could run Materialize side by side with something doesn't mean I necessarily want to, especially given the streaming use case requires a lot more maintenance to get right and keep running in production.

I fully agree with you that for many data science cases, you're likely to stick with batching. Where I see Materialize to be the most useful, and where I'd be inspired to use it and transform how we do things, would be the overlap between when Analytics team are writing definitions (e.g., what constitutes an "active user"?) and are typically doing so on the warehouse, but then I want those definitions to be used, up to date, and available everywhere in my stack, including analytics, my operational database, and third-party tools like marketing tools.

Personally, I'm less interested in one-off migrations like you're suggesting. What I really want is to have something like Materialize embedded in my Postgres. (Such a thing should be doable at minimum by running Materialize + Debezium side-by-side with Postgres and then having Postgres interact with Materialize via foreign data wrappers. It would need some fancy tooling to make it simple, but it would work.) In such a scenario, a Postgres + Materialize combo could serve as the "center of the universe" for all the data AND business definitions for the company, and everything else stems from there. Even if we used a big data warehouse in parallel for large ad hoc queries (which I imagine Materialize wouldn't handle well, not being OLAP), I would ETL my data to the warehouse from Materialize--and I'd even be able to include ETLing the data from the materialized views, pre-calculated. If I wanted to send data to third-party tools, I'd use Materialize in conjunction with Hightouch.io to forward the data, including hooking into subscriptions when rows in the materialized views change.

For what I propose, there are some open questions about data persistence, high availability, the speed to materialize an initial view for the first time, and backfilling data, among other things. But I think this is where Materialize has a good chance of fundamentally changing how analytical and operational data are managed, and I think there's a world where data warehouses would go away and you'd just run everything on Postgres + Materialize + S3 (+ Presto or similar for true OLAP queries). I could see myself using Materialize for log management, or log alerting. I'm just as excited to see pieces of it embedded in other pieces of infrastructure as I am to use it as a standalone product.

I have been waiting for this since the moment I first read about Materialize a year or two ago. I think there's still a lot of work to be done, but at heart, if you can pair technology like Materialize with an orchestration system like dbt, you can use dbt to keep your business logic extremely well organized, yet have all of your dependent views up to date all of the time, and use dbt even to use the same analytical layered views both for analytical AND operational purposes.

The biggest issue I see is that it requires you to be all-in on Materialize, and as a warehouse (or as a database for that matter), it's surely not as mature as Snowflake or Postgres.

Are you referring to LayerCI, possibly?

Quite a few of these issues too strike me as Celery problems rather than RabbitMQ. I’ve run into many of these similar issues and in every case it was due to Celery’s implementation not using RabbitMQ properly and was fixed with an internal patch to Celery.

The most blatant example is the countdown tasks. Celery has a very strange implementation of these (meant to be broker agnostic) where it consumes the task from queue, sees in the task custom headers (which is meaningless to RabbitMQ) that it should be delayed and then sits on the task and takes a new task. That results in heavy memory load on your celery client holding all these tasks in memory, and if you have acks_late set, RabbitMQ will be sitting on many tasks that are claimed by a client but not acked and _also_ have to sit in memory. But that is 100% a celery problem, not Rabbit, and we solved it by overriding countdowns to use DLX queues instead so that we could use Rabbit-native features. Not surprisingly, Rabbit performs a lot better when you’re using native built-in features.

Adding to the pile of people agreeing. SQLite works okay for analytics for now, and I would be super-interested in something embeddable more tailor-made for analytics, but don't understand yet in what specific ways DuckDB is an improvement.

We've been using it for years and love it. Our engineering team went from begrudgingly sometimes doing what they're supposed to in Pivotal to actually _loving_ their project tracker. Never seen anything else like it.

What company was this? I’d love to see a talk about how this worked

We use StackOverflow for Teams with a small team (<15 developers) and it’s been great. While I’m sure our revenue alone won’t make it profitable, I think it’s a product that can work with teams of all sizes. Don’t knock it till you’ve tried it.

This looks exciting, but as other commenters noted, you need to provide more detail about the underlying data and ML strategy. There are many companies peddling both questionable "AI" and questionable "sure-to-work" training and fitness practices, and you sit in the intersection of both. Even if what you offer is real and truly works (and I hope it does!), you'll need to show a bit more to get people to trust you and invest their time in following FitnessAI's instructions.

Can you include links to some of this research? That is fascinating.

I have a friend who is an extreme night owl who has long said then when he sleeps matters far more than how much—but in a negative sense. Even if he gets as much as 10 hours of sleep, if he has to go to sleep early in order to wake up early (6-7am), he’ll be miserable. If he goes to sleep at 3am and wakes up at 9:30am, he’ll feel great despite having gotten much less sleep.

I agree here--we've never had any availability problems like this, and using Segment has been delightful.

I think judging a 6-year old startup based on what its product was one year into the company is rather unreasonable. To put this into perspective, 80% of the company's lifespan has occurred after the grandparent's negative experience. Things change dramatically in that period of time.

Segment is one of my favorite products, and I consider it a core part of our stack. It's enabled us to be leaps and bounds ahead of other companies our size as far as analytics tracking, distribution, analysis in our warehouse, and more. We use pretty much every piece of it--even the esoteric options like their embedded pixel API. It's such a core part of my day that I sometimes actually spend time daydreaming about new features I'd add to the product rather than actually doing my job :)

All I need now is for them to add the ability to push my events in real time to my Kafka cluster. (Please add that!)

Congrats! LogDNA is one of my favorite products. We're likely moving to Kubernetes shortly and were actually worried about logging, so this makes my life a lot easier

WayUp is changing the way that early career professionals find the perfect job, starting with college students. We’re a mission-driven organization and you will be absolutely critical to our long-term success. You enjoy working on a team but also get a thrill out of creating something from nothing, and you’re looking forward to being part of the foundation of our NY-based engineering and data teams.

Please see open roles below:

Senior iOS Engineer - https://www.wayup.com/joinus/#544372

Senior Data Scientist - https://www.wayup.com/joinus/#512217

DevOps Engineer - https://www.wayup.com/joinus/#574860

Recommendations Systems Engineer - https://www.wayup.com/joinus/#571492

Head of Product Intelligence - https://www.wayup.com/joinus/#575145

Senior Backend Engineer - https://www.wayup.com/joinus/#120804

Data Scientist - Applied NLP - https://www.wayup.com/joinus/#580829

We're also hiring for various business roles - check them out here: wayup.com/joinus

Thanks for sharing! I really enjoyed looking around the site. What do you use under the hood for searching and faceting, Solr?

This is a phenomenal idea, thanks for doing this.

Maybe we could even band together and get a preferred negotiated rate Groupon-style if enough of us migrate simultaneously.

Yes, but it's limited to version 1.5.2. Given the backwards-incompatible changes in 2.X, that can be a deal-breaker for many people, sadly.

WayUp (formerly Campus Job - YCW15) - https://www.wayup.com/joinus/

WayUp is the largest marketplace for college students to find part-time jobs, internships, and full-time entry level jobs for graduating students. Everyone remembers their first job. And now, we’re helping everyone get their first job. We democratize hiring by enabling all companies, no matter how big a recruiting team you have (or don't), to easily reach amazing students all over the country. You can read more about us at http://press.wayup.com .

We are one of the quickest-growing companies from our batch (we launched a little over a year ago and we're already 32+ people!), and we're growing out our engineering team and looking for great hackers who love getting features in the hands of users. We're absolutely obsessed with our users and making the experience of finding a job better, and you should be too. We raised our Series A and we're looking for all types of roles:

- Senior Engineering Lead (Python, previous leadership experience - help us scale our team and develop great engineering processes)

- Full-stack developer (Python/Django, Javascript)

- Back-end developer (Python/Django, knowledge of or willingness to learn DevOps)

- Front-end developer (AngularJS preferred but not required)

- B2B Product Manager (previous PM experience ideal)

- Multiple engineering internships (iOS, Python)

- Don't see what you're looking for? Reach out anyway!

We're based in New York City and we're tons of fun--we work with college students all day long :)

Come join us! https://www.wayup.com/joinus/

WayUp (formerly Campus Job - YCW15) - https://www.wayup.com/joinus/ (ONSITE)

WayUp is the largest marketplace for college students to find part-time jobs, internships, and full-time entry level jobs for graduating students. Everyone remembers their first job. And now, we’re helping everyone get their first job. We democratize hiring by enabling all companies, no matter how big a recruiting team you have (or don't), to easily reach amazing students all over the country.

We are one of the quickest-growing companies from our batch, and we're growing out our engineering team and looking for great hackers who love getting features in the hands of users. We're absolutely obsessed with our users and making the experience of finding a job better, and you should be too. We're looking for all types of roles, including a senior engineering hire to help us scale:

- Senior Engineering Lead (Python, previous leadership experience - help us scale our team and develop great engineering processes)

- Back-end developer (Python/Django, excited to work with tools like Elasticsearch, Aloglia, Redis, etc.)

- Jr iOS developer

- Designer (UX/Graphic/HTML/CSS)

- Don't see what you're looking for? Reach out anyway!

We're based in New York City and we're tons of fun--we work with college students all day long :)

Come join us! https://www.wayup.com/joinus/