OliverGuy
Eventually, I hope we'll see many more specialised small models that run nicely on consumer and prosumer hardware. Like LLMs specific to income tax law for UK residents, or LLMs specific to building power delivery circuits for PD USB C devices. There probably is a point where the niche-ness does become to niche to justify the post training, but again, small models means much less training required.
Kind of begs the question if this even needs new/better data? Could you take the foundation model that's trained on everything, and then run a final fine tuning training run on just the set of foundation training data that you want the model to specialise in? Eg, extract just the training data that is about UK income tax law and then run a few epochos on just that? Any ML people know if this is a thing or if not, why it doesnt work?
I can't get a read only API key for my YouTube, Gmail, Outlook etc etc accounts. Only full admin access via a browser.
I can get r/o creds for things like GitLab and AWS and K8s, which means I'm very happy to let my agents run wild at work because I know for certain they actually can't do any damage, they can read all they want but can only go as far as opening and MR that I can review and merge.
Even for domestic robots people don't care, see just about every cloud connected smart vacuum robot and the numerous data leaks
And for companies, most of them are fine using SaaS and cloud based hosting platforms, so why would it be any different for robotics?
Sure there is the latency argument, so at least something doing very short term decision making needs to run on device or at the edge, but for most task the end to end time is in the single didit seconds to minuites, so the extra 100ms or whatever that you get going out to some model in the cloud isn't a deal breaker.
Don't know if that's ever possible to know though unless you train a model from scratch but remove all bike product photography and adjacent materials from the training data?
Put it in front of a common-ish Python or NPM package and see what it finds, likely going to be a lot more.
Sure it's kinda cool as a gimmick, but it should be like 600-800ms at most if you do want one
They won't ever squash or stretch video though, so this means the original uploader stretched the 4:3 content to 16:9 at some point before upload
Sounds like from the requirement to live migrate you can't really afford planned downtime, so why are you risking unplanned downtime?
Its dbt inspired stream ETL tool (or maybe just the TL?), it currently just has a dev mode that does RabbitMQ to local Parque files while I'm getting the core of it to a place I'm happy with.
It runs SQL models against the incoming messages and outputs the results to one or more output tables. Has a local WAL so you can tune it to have sensible sized output files (or not, if you need regular updates but at the expense of query perf.)
Planning on adding Protobuf messages, Kafka as a source and S3 and Iceberg tables as sinks this week.
Lightly inspired by a some projects at work where a lot of time and effort was spent doing this and resulted in something not very reusable without a lot of refactor work. Feel like the stream -> data lake pattern should be something that is just SQL + Config, same way dbt is for transformations within a data warehouse.
No plans on adding any cross message joins or aggregations as that would require cross worker communications and I explicitly want to keep the workers stateless (minus the WAL of course)
Would really appreciate any feedback on the core concept, esp. if this is something you'd actually use in prod (if it were finished!) Not sure if there is something that does this already that I don't know about, or if this genuinely fills some sort of hole in the exisitng tooling
TPUv5e with 16 tensor cores for 2 days for the 200M param model.
Claude reckons this is 60 hours on a 8xA100 rig, so very accessibile compared to LLMs for smaller labs
An LLM observability SDK that let's you store pre and post request metadata with every call in as lightweight an SDK as possible.
Stores to S3 in batched JSON files, so can easily plug into existing tooling like DuckDB for analysis.
It's designed to answer questions like; "how do different user tiers of my services rate this two different models and three different systems prompts?". You can capture all the information required to answer this in the SDK and do some queries over the data to get the answers.