kaathewise

Polars relies on threading heavily, even when streaming files. And it appears that each thread loads quite a bit of memory. I've encountered OOM issues when incrementally reading Arrow IPC files which had very large batches. Fixed it by setting $POLARS_MAX_THREADS to 1, which amusingly also improved the performance on my very narrow task.

Table--Maker's dilemma [0] says that we probably can't get correct rounding for some functions. That said, you can still calculate every function within 1 ULP by using extended precision (for example the double-double arithmetic, which uses two doubles to represent a single fp number) and rounding.

But I don't think anyone does this in practice. Simulations don't need perfect precision, anything involving real data has to deal with measurement errors which are usually much larger than f64 ULP scales, and then there's configurable arbitrary-precision for when it's really needed.

[0]: https://people.eecs.berkeley.edu/~wkahan/LOG10HAF.TXT, which coincidentally is also about the log function

True!

Older numerical libraries such as cephes had to work with heterogeneous non-IEEE floating point implementations: you can still see remnants of VAX tests in the docs, for example. That, too, must've been quite a struggle

Yeah, those operations (+fma) are the only ones for which IEEE 754 mandates correct rounding.

Calculating correct rounding for other functions without hardware support is both challenging to implement and is often really slow. And that's before one gets into special functions. Or, even worse, inverses of special functions. Sometimes you might be lucky when those don't over/underflow around the edges of the domain

StageX does reproducible builds, so they are signed independently and can also be verified locally. I don't think it applies to Astral, but it's useful for packages with a single maintainer or a vulnerable CI, where there is only one point of failure.

But I also think it'd be nice if projects provided a first-party StageX build, like many do with a Dockerfile or a Nix flake.

The first programming language I learned was Java. And for us non-native speakers who didn't know English very well at that point public static void did indeed sound like a magic spell. It was behind both an understanding and a language barriers

Totally. A good chuck of the formats are just TSV files with some metadata in header. Setting aside the drawbacks, this approach is both straightforward and flexible.

I think we're seeing some change in that regard, though. VCF got BCF and SAM and got BAM

One of the problems is that a lot of bioinformatics formats nowadays have to hold so much data that most text editors stop working properly. For example, FASTA splits DNA data into lines of 50-80 characters for readability. But in FASTQ, where the '>' and '+' characters collide with the quality scores, as far as I know, DNA and the quality data are always put into one line each. Trying to find a location in a 10k long line gets very awkward. And I'm sure some people can eyeball Phred scores from ASCII, but I think they are a minority, even among researchers.

Similarly, NEXUS files are also human-readable, but it'd be tough to discern the shape of inlined 200 node Newick trees.

When I was asking people who did actual bioinformatics (well, genomics) what some of their annoyances when working with the bioinf software were, having to do a bunch of busywork on files in-between pipeline steps (compressing/uncompressing, indexing) was one of the complaints mentioned.

I think there's a place in bioinformatics for a unified binary format which can take care of compression, indexing, and metadata. But with that list of requirements it'd have to be binary. Data analysis moved from CSVs and Excel files to Parquet, and I think there's a similar transition waiting to happen here

Yeah, I found `anyhow`'s `Contex` to be a great way of annotating bubbled up errors. The only problem is that using the lazy `with_context` can get somewhat unwieldy. For all the grief people give to Go's `if err != nil` Rust's method chaining can get out of hand too. One particular offender I wrote:

   match operator.propose(py).with_context(|| {
    anyhow!(
   "Operator {} failed while generating a proposal",
   operator.repr(py).unwrap()
  )
   })? {

Which is a combination of `rustfmt` giving up on long lines and also not formatting macros as well as functions

Yeah, I presume so. At least that's what Swiss Tables do. The paper is focused more on the asymptotics rather than the real-world hardware performance, so I can see why they chose not to handle such edge cases

The dereference table allows allocations to fail:

https://arxiv.org/pdf/2501.02305#:~:text=If%20both%20buckets...

(the text fragment doesn't seem to work in a PDF, it's the 12th page, first paragraph)

Ah, I see. Best of luck with that!

There are a few projects where I'd love to see a modern spreadsheet implementation. CryptPad comes to mind. They use OnlyOffice, which is quite featurefull, but takes awhile to load and isn't as responsive.

Yep, I've misunderstood, realized it after seeing mdaniel's comment.

Thanks for making this in the first place! I saw IronCalc in the list of projects supported by NLnet and it grabbed my attention.

By the way, if You don't mind me asking, how'd Tuta end up sponsoring IronCalc? It seems that lately they and Proton have been trying to expand their business away from just email. The fact that Tuta is interested in IronCalc makes me think they want to have an office-like offering.

Right, I've made a mistake! I keep getting surprised by the fact it's possible to simply compile a Rust crate with a WASM target and run it in the browser.

A spreadsheet engine. It's a React app with a Rust backend, but it impressed me how snappy it was[0]. Of course, it's not nearly as feature rich as Google Sheets, not to mention Excel.

[0]: https://app.ironcalc.com/

Contrary to the name, I don't think that it's very good. The whole thing is a canvas via WASM, so scrolling isn't smooth, selection doesn't work, and accessibility is seemingly non-existent.

But I think the technology itself is interesting. While most modern UI toolkits use HTML or React-like components, this uses a set of JSONs, which describe the page.

I haven't self-hosted any, but I enjoy using Flarum [0] forums. They load faster than Discourse ones and feel snappy.

[0]: https://flarum.org/

It's a minor complaint, but I'm also evaluating it for a minor project. I just don't like the fact that I can forget to add a flag once and, oh, now I'm sending telemetry on my personal medical documents.

I was searching for a Meilisearch alternative (which sends out telemetry by default) and found Tantivy. It's more of a search engine builder, but the setup looks pretty simple [0].

[0]: https://github.com/quickwit-oss/tantivy-cli