ambrood
ambrood on Hacker News
Congrats again micah! Also Iceberg catalog on R2 without needing to spin up additional services... lets gooooo
> Maybe you mentioned it in your demo and I missed it, but how does this differ pasting the log messages to ChatGPT / Claude / another LLM? Is it mainly that yours can iterate over a large logfile without blowing up the context window?
We do quite a bit of aggregation over the log file, and generate summary stats and choose what bits to stuff in the LLM. Plan to support more platforms than just spark.
> Does it suffer from the same issue as other LLMs, where it will always identify potential optimizations or improvements even if none are truly needed?
Funnily enough, instructing sonnet-3.7 to not suggest unnecessary optimisations seems to have done the trick!
Posted by
ambrood on Wednesday April 09, 2025 @04:42PM
from the show-hn dept.
https://datasre.ai/
Hey HN,Spark event logs run into 100s of MBs and offer a wealth of insight into your workloads but making sense of them has always been quite a bit prohibitive. We’ve recently built a lightweight tool that automatically parses Spark event logs and surfaces targeted insights to help you optimize your data jobs.Whether you’re chasing down a bottleneck or balancing performance vs. cost, the...
thanks for the feedback! the first version had a lot more detailed code but decided to go with linking to our GitHub than copying all the code. Wanted to illustrate the core touch points involved in extending DF.
Posted by
ambrood on Wednesday November 20, 2024 @08:16PM
from the denormalized.io dept.
DataFusion is primarily a batch OLAP system, so we should be able to support hybrid workloads as well. And definitely agree with you re: Polars dev exp. That is something we are aiming for with our forthcoming Python sdk.
> It'd also be important/useful to support Python udfs (think numpy/jax/etc.).
Yep that's our longterm gameplan.
> It'd be very cool if you could collaborate with or even tap into the polars frontend. If you could execute polars logical plans but with a streaming source, that would be huge.
Are there examples of project that do this? I'd be very much interested in looking into this.
we absolutely do, the library itself is designed to be extensible. we are currently working on adding webhooks as one of our sources. are there are any specific connectors/sources you'd be interested in?
while haven't checked out Fluvio yet, we are fans of Arroyo. regarding latter my understanding is that the team is going for a SQL first complete replacement for Flink. Denormalized is meant to be an embeddable engine you can import within your project. Our plan is to focus on the developer experience for users building with Python and Typescript in particular.
thanks @ztratar. would love to hear about your workloads at embra would be very helpful vis-a-vis the direction of our typescript experience. feel free to drop us an email:
[email protected]
this looks so clutch! curious if this was purpose built for the finance industry?
thanks for the encouraging words @ethegwo. Tonbo looks very cool and potentially something we could use for our state backend (currently using RocksDB which we aren't that happy about). Would love to chat about how we can work together. Feel free to reach out to me -
[email protected]
Posted by
ambrood on Thursday August 15, 2024 @01:16PM
from the show-hn dept.
https://github.com/probably-nothing-labs/denormalized
tl;dr we built an embeddable stream processing engine in Rust using apache DataFusion, check us out at https://github.com/probably-nothing-labs/denormalizedHey HN,We’d like to showcase a very early version of our embeddable stream processing engine called Denormalized. The rise of DuckDB has abundantly made it clear that even for many workloads of Terabyte scale, a single node system outshines...
have been playing this for sometime. quite addictive! ty @cpojer
Posted by
ambrood on Tuesday October 31, 2023 @04:34PM
from the show-hn dept.
https://teamdenormalized.substack.com/p/all-i-wanted-was-to-grok-some-real
Hi HN,We are launching Denormalized (www.denormalized.io), a serverless real-time data platform built on Kafka and Pinot.We felt a bit burnt out by the sheer developer toil we faced when building application around the real-time data stack and set out to create a platform to allow small teams to be very productive with realtime data without having to glue together an elaborate system to serve...
Again much of that is how the accounting for stock grants goes (most of it in company’s history) and not say, actual PNL incurred in incentives.
you realise that much of it is stock grants right?
perhaps update your priors at this point. Uber is a profitable company now.
this isn’t the modus operandi. SOP is Uber will give you a call/leave a message.
Also passengers pay you $20 or so for returning an item.
Only situations where they will disable you automatically is when someone complains about safety or violence or something along those lines.
* I worked there from 2016-2020
lmao... figured it was radius reduction or somesuch. i ended up working on Winterfell (the service that managed automated radius reduction) at some point during my time there.
the picture is from like 8 years ago.
even for 2015 (when this was likely taken) Uber Eats would most certainly not have run from a single laptop. Core services like dispatch, matching, pricing etc in any case weren't geography specific. If i had to guess, this was some Ops person running a local driver incentive, payout query + email workflow from their laptop.
Posted by
ambrood on Tuesday February 07, 2023 @07:11PM
from the twitter.com dept.
Have been playing around with this a bit lately and the OS integration is just really well done. Very nifty interface for shoving context from across your apps into ChatGPT.
pre-IPO RSUs usually come with a double trigger. meaning you vest them monthly or quarterly etc but a liquidity event such as an IPO, is needed to actually "vest" them in a taxable sense.
Uber's internal stuff used resolution 9 i think...
I worked at Uber too. Turns out if you work at the right places, it opens doors for you. But yes most conferences have a "request for proposal" system.
your guide literally helped me hack through a networking class i took in college a decade ago. dont work anywhere close to that stack but this brought back memories.
Love the shoutout to Gairos. But man that on-call was a pain in the arse.
Back when i was majoring in ECE at UIUC, we got to play around with Fab labs :). Yeah you'd pretty much need a advanced degree in EE or Physics to be a semi-conductor engineer I'd reckon.
Again its astounding how many Americans dont realise how the GC process really works. Only 15% of the greencards are handed out to employment based categories, rest go for family reunification. That 15% additional has country of birth quotas etc.