jasim

If tokens are going to get cheaper and faster - then we would stop caring about token efficiency, like how we're all using bloated and slow software, and which keep growing to take up all available hardware improvements.

ERPNext (Frappe) is absolutely brilliant. My only complaint was that it is in Python and is a bit heavy weight (setup etc) for quick prototyping.

They call a table definition - that defines a table at both the SQL level, and at the domain level - a DocType (see: https://docs.frappe.io/framework/user/en/basics/doctypes). When you declare a DocType you immediately get a working table, foreign keys transformed into combo boxes for lookups, forms, and a whole set of conveniences that are common to ERP-like applications.

Before seeing ERPNext I would've thought that such a declarative approach would fail to satisfy the critical remaining 20% of most requirements, but they've built around it in a way that has enabled them to ship a complex ERP system that thousands of businesses of different shapes use. I think it is one of those gems that are not in popular conscience among software engineers, compared to regular frameworks (but very much so in business-ERP adjacent circles), and is worth learning from.

It is so hard to predict the future, of all things. But I think there is another approach which is already tried and tested - frameworks.

I think frameworks would mutate to be agentic - they'll come with skills and documentation targeted to LLM agents, and they'll have far more breadth than anything we've seen, a certain lack of restraint, and will sacrifice composability and orthogonality because they are no longer useful for agents.

The framework I'm building - plug: https://sapporta.com - brings in a set of UI choices (datagrids for every table) and a set of API choices (everything exposed to agents) which is not typical to classical frameworks. If you're building database applications, it contains most of the things that a typical NoCode tool would have, but with complete code and the ability for agents to build upon it.

I think Sapporta is one of the first frameworks with this point of view, but it has only been a few months since Opus 4.8, and I think in the coming months we'll see a lot more vertically integrated horizontal frameworks (the solid base) that can get us malleable software with custom code.

I think this is somewhere in the middle east - Dubai likely - and it is hot there. It is quite predictable that there would be mass opposition with institutional support against turning off air conditioning in a common space, despite having a couple of hours of exclusive rental. That said, it is a good reminder that we have a choice in what we are willing to accept, and can fall back on a BATNA, even if it is not the most ideal outcome.

I've been paying JetBrains in a personal capacity for about 15 years. Almost from the first year of my professional career. I didn't renew this year. I feel so sad and disheartened that a company that produced such brilliant tools have completely lost the plot on AI. They had started work on Fleet, a lightweight VSCode like IDE, before LLMs became big, but abandoned the effort without seeing it through. It could've been a great agentic IDE (all I want is whole file+project view for every conversation, instead of just diff so I can have a better perspective of the codebase). Instead they built multiple confusing tools (might be internal org structure and competition leaking out), and took far too long to adopt SOTA CLI coding agents, and kept charging for it as a separate subscription. I wish they do a whole company rethink on agentic development, but it is going to be tough because the existing enterprise licenses will continue to be the main revenue driver for a few more years, until it all dries up all at once.

We might be moving towards putting more intent - in text - into the code itself, for agents. We should've been doing that for other humans in the past, but it wasn't always encouraged - "code should be self-explanatory, and comments are a sign of poor naming and abstraction", and all that.

These days I run a prompt of this shape to ensure that my code contains as much tacit knowledge as I have -- "I want to make sure all the implicit invariants and structures and concepts described in these plans, are present in the code comments, namings, comments around major types and modules. So even if the original plan file is not present, we should still be able to understand it (Peter Naur - Programming is Theory Building)"

However - I still need the type system as another preserver of invariants and intents. Coding agents still make subtle mistakes and miss very obvious things as the code gets more intricate and conceptually large.

A type error where an expected case is not handled - that is a feedback that can ground it back and force it to think through whether windspeed matters to is_dangerous or not. And the choice of is_dangerous in the article is a bit misleading, because anyone can at first read make the connection from windspeed to is_dangerous. It is a pretty bold variable name and attention grabbing situation.

However codebases are made of far more mundane but widespread and braided sets of tiny invariants that wouldn't catch either a human's or a coding agent's attention without needing a more firmer hand, and so I still do love my type systems despite my best laid comments.

But couple that with a deterministic system - the type checker with exhaustive destructing (and pattern matching, while we are it it) - that would become a fool-proof combination.

I hadn't considered that, you are right. Though I would still prefer TS because the language is all about types; the entire ecosystem is well-typed and the type system is quite powerful (I do enjoy an occasional Omit and Pick). But it is a personal choice; as long as LLMs can generate well-typed Python, then for people who like the language it makes sense. However from the article, I got the sense that they went completely dynamic.

I'm curious about the choice of Python, rather than TypeScript.

I find Ruby a very beautiful language, and Rails is an excellent web framework, but I need typed functions, record types and sum types.

They help not just with correctness, but also as living documentation that lets me understand AI generated code. TypeScript provides discriminated union, but not exhaustive pattern-matching, and its syntax is a bit verbose, but since I'm no longer writing most of the code myself, I can live with it.

However I can't imagine using Python or any other dynamic language going forward. There is likely good reason for you to choose it, and I'm curious to know what that is.

There is no other stack in the last 15 years or so that I look back with the same fond affection as building business applications in Clipper. One important aspect was that software was delivered physically, and I could watch my users using them. If I made a report that helped the operator, or made a workflow easier, their happiness was immediate and nurturing. And there was an immediacy to programming in xBase - if we're building a database backed business application, which was their strong suit, there was very little standing in between us and the problem space.

"There was no exploit. No vulnerability disclosure. No CVE for me to write. The attacker filled out my signup form 942 times, made 942 workspaces, sent 942 batches of about a hundred invitations each, and stopped. They used my tool exactly as designed. The design was just bad enough that the tool was good for phishing."

In my early programming days, working with Clipper, I used to look at Delphi from a distance with awe and a bit of jealousy. There also used to be PowerBuilder and Paradox, as competition to the xBase platforms.

I'd love to hear more about how you're using Delphi and what it excels at, compared to current web and native software stacks.

Everytime someone mentions Clipper (or dBase or FoxPro, or even FoxBase, but Clipper the most), I feel a sene of productive nostalgia, and a constructive anger at the state of technology today. xBase was a beautiful thing - I haven't had as much fun at building software that I've had from the first plink86 till CA-Clipper 5.3's blinker and exospace. Even prolific use of Opus 4.6 doesn't bring the sense of quality and satisfaction that those systems produced.

I'm building a new database tool for the web, a frankenstein of Lotus 1-2-3, dBase, MS-Access, and Claude Code. It is where that anger goes these days.

I'm not sure if this is what you're looking for, but there is a proposal in DataFusion to allow user defined types. https://github.com/apache/datafusion/issues/12644

Parquet files include a field called key_value_metadata in the FileMetadata structure; it sits in the footer of the file. See: https://github.com/apache/parquet-format/blob/master/src/mai...

The technique described in the article, seems to use this key-value pair to store pointers to the additional metadata (in this case a distinct index) embedded in the file. Note that we can embed arbitrary binary data in the Parquet file between each data page. This is perfectly valid since all Parquet readers rely on the exact offsets to the data pages specified in the footer.

This means that DataFusion does not need to specify how the metadata is interpreted. It is already well specified as part of the Parquet file format itself. DataFusion is an independent project -- it is a query execution engine for OLAP / columnar data, which can take in SQL statements, build query plan, optimize them, and execute. It is an embeddable runtime with numerous ways to extend it by the host program. Parquet is a file format supported by DataFusion because it is one of the most popular ways of storing data in a columnar way in object storages like S3.

Note that the readers of Parquet need to be aware of any metadata to exploit it. But if not, nothing changes - as long as we're embedding only supplementary information like indices or bloom filters, a reader can still continue working with the columnar data in Parquet as it used to; it is just that it won't be able to take advantage of the additional metadata.

I think this post is a response to some new file format initiatives, based on the criticism that the Parquet file format is showing its age.

One of the arguments is that there is no standardized way to extend Parquet with new kinds of metadata (like statistical summaries, HyperLogLog etc.)

This post was written by the DataFusion folks, who have shown a clever way to do this without breaking backward compatibility with existing readers.

They have inserted arbitrary data between footer and data pages, which other readers will ignore. But query engines like DataFusion can exploit it. They embed a new index to the .parquet file, and use that to improve query performance.

In this specific instance, they add an index with all the distinct values of a column. Then they extend the DataFusion query engine to exploit that so that queries like `WHERE nation = 'Singapore'` can use that index to figure out whether the value exists in that .parquet file without having to scan the data pages (which is already optimized because there is a min-max filter to avoid scanning the entire dataset).

Also in general this is a really good deep dive into columnar data storage.

Well put. It should always be "Created by <person>" rather than "Processed by LLM". We can already see it with Claude Code - its commit messages contain a "Generated by Claude Code" line, and it guarantees a pandemic of diffused responsibility in software engineering. But I think there is no point in railing against it - market forces, corporate incentives, and tragedy of the commons all together make it an inevitability.

I'm sorry I don't follow. The fact that you use an LLM to classify a transaction does not mean there is no audit trail for the fact. There should also be a manual verifier who's ultimately responsible for the entries, so that we do not abdicate responsibility to black boxes.

Accounting, specifically book-keeping, really plays to the strengths of LLMs - pattern matching within a bounded context.

The primary task in book-keeping is to classify transactions (from expense vouchers, bank transactions, sales and purchase invoices and so on) and slot them into the Chart of Accounts of the business.

LLMs can already do this well without any domain/business specific context. For example - a fuel entry is so obvious that they can match it into a similar sounding account in the CoA.

And for others where human discretion is required, we can add a line of instruction in the prompt, and that classification is permanently encoded. A large chunk of these kind of entries are repetitive in nature, and so each such custom instruction is a long-term automation.

You might have not been speaking about simple book-keeping. If so, I'm curious to learn.

I can think of two instances, where the LLM embracing best practices for human thought leads to better results.

Claude Code breaks down large implementations to simpler TODOs, and produces far better code than single-shot prompts. There is something about problem decomposition that works well no matter whether it is in mathematics, LLMs, or software engineers.

The decomposition also shows a split between planning and execution. Doing them separately somehow provides the LLM more cognitive space to think.

Another example is CHASE-SQL. This is one of the top approaches in Text-to-SQL benchmark in bird-bench. They take a human textual data requirement, and instead of directly asking the LLM to generate a SQL query, they run it through multiple passes: generating portions of the requirement as pseudo-SQL fragments using independent LLM calls, combining them, then using a separate ranking agent to find the best one. Additional agents like a fixer to fix invalid SQL are also used.

What could've been done with a single direct LLM query is instead broken down into multiple stages. What was implicit (find the best query) is made explicit. And from how well it performs, it is clear that articulating fuzzy thoughts and requirements into explicit smaller clearer steps works as well for LLMs as it does for humans.