leventov

I'm working on Pocketdata - a personal, private AI data plane.

The idea is to take boring components: PostgreSQL, Bifrost (LLM gateway), Open WebUI, LanceDB, Agentgateway (MCP and OpenAPI gateway) and deploy them in Fly.io. One Fly.io "org" per user. The closest equivalent is blaxel.ai, but it caters for AI SaaS startups, not individual customers.

The combination of the fact that Fly secrets are visible only from within the apps, distroless containers, and transparent data encryption for PostgreSQL assures that the service (Pocketdata) provider cannot access their data, only the infrastructure provider (Fly.io) theoretically can, but practically speaking, this gives an extremely high degree of privacy assurance.

The latest update on the project: https://engineeringideas.substack.com/p/tasklet-is-the-o1-mo...

> It seems like a web version of claude code + skills + marketplace, but with an encrypted database, and permissions that let you feel safe using yolo mode. I’m going to try it.

I'm building infra for exactly this thing :) Here I posted about it today: https://engineeringideas.substack.com/p/tasklet-is-the-o1-mo...

Architecture TLDR: In Fly.io, org-per user, manage keys in Fly secrets. A Postgres db with transparent data encryption (TDE), the master key is stored on user's computer in the keychain or in the password manager. Thus the nobody can read the data at rest. All containers are distroless so nobody can ssh onto them. Postgres is backed up via pgBackRest to Wasabi object storage with customer-provided encryption keys that are injected into the containers via Fly secrets.

Apart from the database for record-like things (chats, emails, tables), vectors for larger things (such as web pages) are stored in serverless LanceDB on Wasabi, too.

Also Bifrost (https://github.com/maximhq/bifrost) as LLM gateway and Agentgateway (https://github.com/agentgateway/agentgateway) as MCP and OpenAPI/REST API gateway.

Agents/apps themselves (Open WebUI, Zero Mail, etc.) are separate Fly apps, and have their separate schemas and users in Postgres. They also cannot go to public internet directly (prohibited via https://community.fly.io/t/new-feature-network-policies/1917...), only to Postgres, Bifrost, and Agentgateway.

Postgres, pgBackRest, pgBouncer and another Go sidecar for pg (a la Pocketbase, but with Postgres backend), live in a single container managed by Horust. Bifrost and Agentgateway live in separate containers, but the same Fly machine. This machine might be 1.5gb, 4 vCPUs. LanceDB is on a separate machine because it needs burstable memory and may be infrequently used.

All machines (core, lance, and all individual app/agent machines) are suspendable, so they almost don't cost anything when not in use.

It's not super easy to port a library with domain-specific ML/DS algorithms built around Pandas, such as accepting Pandas dataframes as arguments to its functions and class constructors.

BigQuery Storage Read API claims to support filters and simple projections pushed down to the storage: https://cloud.google.com/bigquery/docs/reference/storage. See also this recent paper: https://research.google/pubs/biglake-bigquerys-evolution-tow...

I've also recently proposed a Table Read protocol that should be a "non-vendor-controlled" equivalent of BigQuery Storage APIs: https://engineeringideas.substack.com/p/table-transfer-proto...

I don't buy this "object storage + Iceberg is all you need for OLAP" hype. If the application has sustained query load, it makes sense to provision servers rather than go pure serverless. If there are already provisioned servers, it makes sense to cache data on them (either in their memory or SSDs) to avoid round-trips to object storage. This is the architecture of the latest breed of OLAP databases such as Databend and Firebolt, as well as the latest iterations of Redshift's and Snowflake's architecture. Also, this is the approach of the newest breed of vector databases, such as Turbopuffer.

For OLAP use cases with real-time data ingestion requirements, object-storage-only approach also leads to write amplification. Therefore, I don't think that architectures like Apache Pinot, Apache Paimon, and Apache Druid are going anywhere.

Another problem with "open table formats" like Iceberg, Hudi, and Delta Lake is their slow innovation speed.

I've recently argued about this at greater length here: https://engineeringideas.substack.com/p/the-future-of-olap-t...

Every once in a while, before falling asleep I enter a "mellow" state in which I can conjure mental images dramatically more vividly than normally. I would say that normally it varies between 10-50% of vividness (also depends on the state of mind, the day, whether I'm under the influence of different chemicals), and in this state of mind it's 90%+ vivid.

And I don't confuse this state of mind with lucid dreaming: I had spontaneous lucid dreams a few times. They were not very vivid, just as my normal dreams (I dream in images, and they are maybe around 50% vivid).

So, there is definitely a way to experience variation of vividness of mental imagery, by a single observer.

Answering my own question about the other frameworks (one also from Google, BTW): https://exa.ai/search?c=news&q=time%20series%20prediction%20...

What are the other four frameworks?

> For one, who wants to do stuff in tensorflow anymore let alone tensorflow-probability.

AutoBNN is a JAX library and has nothing to do technically with TF Probability. It was developed by the TF Probability team.

> DL community prefers pytorch and stats community prefers Stan.

It looks like the JAX ecosystem for stats is growing: NumPyro is based on JAX, PyMC has a JAX backend, https://github.com/blackjax-devs/blackjax has effective samplers, there is https://github.com/jax-ml/bayeux, and now AutoBNN.

> This one seems theoretically more interesting than some others but practically less useful.

Are there other factors why you think AutoBNN is not practically useful, apart from being based on the wrong foundation (which was a mistaken belief of yours)?

The big question is whether we really want more centralised cloud applications of this sort in the AI future, with its potentially exacerbated risks of data breaches, identity theft, spam, etc.

Having a "Rama" for local-first, truly distributed (Solid Project style), self-sovereign-identity based apps would be differentially better probably.

Bad analogy. Get in good shape to feel better, have more energy, be more attractive to other people, etc.

There is no point of getting good at arithmetic today, indeed.

I agree people need to continue study to understand the world (if they are to survive at all), but the current school curriculum is a joke.

Ok, then what's the point of defining 'intelligence' out of existence, if neither LMs nor humans are 'intelligent'?

It might be that humanity becoming non-chauvinistic about their value is the necessary thing for survival.

You can't do it in your basement as of 2023. Very important qualification. It's entirely plausible that continuous evolution of ML architectures will lead to general AI which anyone can start on their phone and computer and learn online from there.

Or so their groupthink tells them.

Bingo. That's why working at all these companies is morally problematic.

Unless software engineers "vote" by going to work elsewhere, this won't change.

Looks for companies with solid ethics, and with founders who have background working at or starting other similar companies.

But not coincidentally, the pay at these companies will be at least multiple times less than at Big Tech.

You have excellently described the job of business analysts and system architects, but this is not the job of 90% of programmers today, including senior-level. Part of this is already done by other people and doesn't require specific programming skills, hence, at the very least, programmers will lose their privileged position. Another part of it is actually too hard for most people who are currently employed as programmers to do on a decent level (such as meaningfully hacking on Linux kernel).

There will be just a short transition period. In 10 years, AI will be writing most of code, and in 20 years - nearly all code. People will do only architecture/business analysis.

No more "exploitation" of labor.

Perhaps people on HN start sensing that successors of Github Copilot will take their programming job. Rightly so.

Personally, I think that in the age of AI programming any notions of code licensing should be abolished. There is no copyright for genes in nature or memes in culture; similarly, these shouldn't be copyright for code.

This is really the same argument as it used to be: help intelligences (used to be only human, now artificial) to find bugs by matching text with code.

Another Northvolt employee here. Besides relocation, we think about allowing remote from European time zones for software development roles.

That's a really long term bet, at least 50 years from now. By that time wealth in the current sense of the word might become meaningless (or might not).