I feel like no project should be allowed to advertise itself as "fast" unless it presents at least one (1) measurement demonstrating it.

Running subjective benchmark now. Generated 1G text file. Vim and nano after some delay are showing things on screen, ox is still thinking. 5 minutes and counting...

  base64 /dev/urandom | head -c 1000000000 > file.txt

One thing about this particular benchmark is that the file it produces is a little bit unusual even for large text files, because it is only contains one (massive) line. Most huge text files you open in text editors are things like multi-gigabyte logs and database dumps, which are composed of many relatively short lines. Text editors often bake that assumption into their data structures. A more representative test would probably be to use hexdump instead of base64 there.

Though I take the point: if a text editor advertises itself as being "fast", this is the kind of stuff you expect it to handle.

EDIT: ok, so i just tried the hexdump version. Vim and Emacs took a couple of seconds each (Emacs being slightly slower and chunkier when you navigate, Vim handles it without issue and is perfectly repsonsive). less opened it instantly (obviously, it's less!). ox has been going for a few minutes now, nothing happening.

It's not the benchmark's fault, ox is just slow for large files.

> the file it produces is a little bit unusual even for large text files, because it is only contains one (massive) line.

XML and JSON without pretty print. Minimized HTML with embedded resources. While not often gigabyte sized those seem to be enough to get most editors to hang and make editing the files a miserable experience.

Yes, true enough, it does happen occasionally, but it's pretty rare (and usually there are other ways to deal with it: using less or similar tools, or just running your massive single-line JSON thing through jq first). And my point was exactly what you're saying: many text editors choke on files like this, so it's not such a huge indictment of ox that it chokes on it as well.

However, ox clearly performs poorly even when your text files aren't massive single lines, so it doesn't really matter: ox basically can't handle files like this regardless of line distribution.

> and usually there are other ways to deal with it: using less or similar tools, or just running your massive single-line JSON thing through jq first

Older Unixen at least (SVR3-based) had a tool called bfs - big file scanner. Used it some, e.g. when, as a system engineer, I helped IT staff of one of our customers, a university processing exam results of tens of thousands of students from dozens of colleges affiliated to that univ. IIRC, its UI was something like a read-only ed (google "unix bfs command"). You used it to scan really large (for the time) files, e.g. data files (input/output) in large data processing environments, for purposes like checking if the input or output files anecdotally looked okay, no major noticeable garbage in them. Haven't checked if it is present in modern Linuxes. Also don't remember if it could handle large files without newlines. Likely not, if based on ed. Didn't have such files to work on then.

As an emacs user, one of the things I really wish it had was robust support for arbitrary file sizes and shapes.

(It'd be fine if not all features worked on these files.)

Sure, most files are reasonably sized and shaped, but sometimes you do find yourself needing to e.g. dig into that single-line 1GB json file to try to debug something, and it's inefficient to have to think, before opening any file, "is more normal tool sufficient for this, or should I use something else?"

There's no fundamental reason we can't have one tool that does it all!

emacs -q or -Q is often quite fast. It's things like syntax highlighting, word wrapping, etc. which modes add which usually slow it down. Anything trying to run regexes over the buffer isn't going to scale well.

I think long lines are (were? [1] seems to be a new feature) a well known problem in emacs even without a custom config. There have been ways to work around this for awhile I think, but as far as I know emacs -Q isn't one of them.

[1] https://www.gnu.org/software/emacs/manual/html_node/emacs/Lo...

oh that's neat, i didn't know about that! could help.

my ideal editor would never block, and use file-sized based approximations to allow scrolling to arbitrary locations w/o having to actually read the contents of the entire. things like syntax highlighting &c would be either strictly time-limited to not drop frames, or done in the background.

Try M-x find-file-literally to avoid any mode-specific hooks, colorization, etc.

Incorrect. The base64 command wraps at 76 columns by default.

Whether there's a wrapping feature depends on the implementation. coreutils base64, yes. Other implementations, maybe not. macOS's /usr/bin/base64 for instance doesn't have the concept of wrapping. FreeBSD base64, judging from the manpage, doesn't do that either. https://www.freebsd.org/cgi/man.cgi?query=base64

Edit: Oops, posted some misinformation. The mistaken part, reproduced below:

> First, wrapping by default only occurs when writing to a tty. When stdout is redirected to a file, wrapping base64 output usually doesn't make sense, so wrapping doesn't happen.

Correction: coreutils base64 wraps regardless of whether stdout is a tty.

> macOS's /usr/bin/base64 for instance doesn't have the concept of wrapping

Yes it does. "-b N" or "--break=N" inserts a line break every N characters.

Yeah. I know -w doesn't work, and missed -b when glancing at the manpage. (I use coreutils day to day.)

I ran it on macOS, and there I got:

    $ base64 /dev/urandom | head -c 1000000000 | wc
       0       1 1000000000

I haven't tried anywhere else, so it might be different on Linux or GNU or whatever.

Could this be summarized as "algorithm" (or data structure more specifically) beats language implementation performance?

Certainly could!

Summary of replies: This benchmark doesn't count because No true Scotsman.

Most would happily trade an editor that is 100x slower for gigabyte single-line text for one that is 10% faster on gigabyte multi line text.

I rarely open gigabyte single-line files, but when I do, I don't want to switch editor.

Most of the time I only realize that everything is on a single line after opening the file.

A good editor doesn't require me to think about these things.

Sure. I'd not be able to actually do anything useful with a gigabyte single-line file so what I'd want my editor to do is say "Longest line is over 100k characters, truncating display to the first 100k".

issue is that the editor can't tell if it's a multiline file with a first big line or an editor with one big line if it's cut at the first line. But I'd rather have the editor bail at the first 100k line than not opening it.

If it's over 1 GB and single line you're almost always going to exit the editor, run some scripts on it, and open it again when it's correctly formatted.

I would rather open it in editor, strike a "reformat all" shortcut, and have the file right there.

I don't want to chase separate tools that can format different text files, that's what editors are for.

It's still annoying if your editor hangs and you have to find some way to kill it before your computer becomes usable again.

Sure, but this one isn't 10% faster on gigabyte multiline text. Others in this thread are testing wrapped text and Ox still chokes.

99% of files I work with are <1000 lines, so I don't mind. For larger files I will fall back to Vim.

The important thing to realize is that text editor performance is less about the technology used and more about the data structures and i/o.

This is not an excuse for an editor being claimed "fast". Textadept is written in C and is also very fast but has the exact same issue -- choking on huge files when Vim and others are able to handle it.

Emacs opened it in about 4 secs on my 2018 MBP.

I wonder: Couldn't we have some kind of simulated line breaks, which an editor places virtually, so that it can manage very long lines easily and efficiently internally, while writing out the file without these virtual line breaks? Basically only so make the conditions appropriate for the editor's string data structure?

In fact, you basically need to do that anyway to wrap long lines effectively.

Software developers find lazy/partial file loading unnecessary but, it is not and it won't...

Vim is amazing. Try searching for patterns in this enormous file ;) Definitely a hard nut to crack.

Personally i see no reason why this is needed (just use vim), but I'm sure a lot of people will use it just because its written in rust^^

I highly doubt you have to manually edit a GiB file by hand. For read-only, less opens it instantly.

I tweak 1.5GB files by hand on a weekly basis on servers across the globe. Fortunately vim works for my use case.

In that case, you would be better served by some automation tools.

You'd have to know my specific use case to say that.

Unfortunately it's not automatable.

I have had to do this in the past. Admittedly, that was only a few times over the course of 15 years.

Yeah but that's not a realistic usecase for most users/developers. I haven't measured if Ox is faster or slower, but for me startup time and ability to quickly open files under 10MB are more important.

> Yeah but that's not a realistic usecase for most users/developers.

It also looks like vim and emacs don't decide what a realistic use case is, or if you're "most" users/developers. They just open text files for editing.

Probably, but then we don’t use our normal editors when we want to open huge files, we use the fast ones.

I open massive files in a text editor all the time.

I also am regularly limited in opening large text files working with genetic data.

I frequently open minified CSS/JS by accident. IntelliJ's project search window includes a small preview of each result when you hover it, so when you navigate through the results your might pass through a minified file and get your IDE to hang for a couple of minutes :)

But for 10MB range do you even feel the delay for any of text editors? For me it feels instant whatever (terminal based) editor i'm using. And what does it even mean "fast" in this case. They all are definitely faster than me typing.

Is something like Emacs, Vim, or Nano genuinely slow for you?

Even for large files, I've never noticed anything beyond microsecond-type delays with those. Definitely does not feel laggy to me, usually.

It is very realistic when I copy a json dump into an editor and then do a key stroke to format it.

My beloved vim doesn't fair so we'll. VScode does very well on this.

I noticed some years ago that vim can be much slower due to highlighting of particular syntax of huge files. For example it was fine with a 10MiB C file, but super slow with a 10MiB XML file, but fast again with ":syntax off"

Are you doing the formatting via a Vim plugin?

I often find it's much faster to pipe out to a command line program to do formatting on huge files.

e.g.

   :%! jq 

or

   :%! xmllint --format -

Huh, thanks, I didn't know you could put the contents on stdin like that (%!) - just in case you (or anyone) was similarly unaware in reverse, you can also put `%` wherever you want to use the filename in the command.

i.e. I would have done:

    :! jq %

and have probably been guilty of (not knowing `%!` per above):

    :! cat % | ...

I open massive text files semi-regularly. If a text editor can't deal with that it's a big con for me.

I'm slowly putting together a library and series of blog posts for really showing this, so stay tuned (one measurement often isn't enough).

I've always wanted to be able to try and verify performance bounds at the CI stage, and I'm hesitantly optimistic it can be done cleanly (although amortized analysis is extremely important, you can actually play "guess the C++ std container from the graph" if you plot the data right)

The input is actually very slow. It's own configuration file (about 300 lines) took over a minute to paste.

It runs in the terminal. I think most people assumes that terminal === fast.

Direct to X can be much faster, as there is no middle man. And can have better keyboard (and mouse) support which in terminals is pretty poor. This is why vi(m) is best for terminals, because it's keyboard shortcuts are adapted for that (except typical 1 second delay on escape key).

But it's close to metal

\s

Python compared to raw assembly development is slow as hell in computationally heavy scenarios.

Development speed of applications in Python compared to raw assembly is nearly infinitely faster, both because it’s easier to write and because the talent pool for ASM is near zero in most cities.

But the same can be said for basically all metrics people use to sell products.

Am I the only one who feels _less_ inclined to use a program if they use 'written in rust' as a selling point?

I mean, I very rarely see non-rust projects say "Written in X!" as a selling point for the end user (why would an end user care?), and the campaign to re-write everything in rust is similarly irritating. I've never explored or read up on the rust community, but the sheer insistence that I see jutting into everything from the outside turns me off.

If I see that something is written in Rust, I'm predisposed positively because I know it'll be a single binary I can easily install, it'll be fast and it'll usually be well-designed (Rust and good cli design correlate, for some reason).

Similar for Go for me. I personally like Go and will go out of my way to use tools written in it because I understand the tooling and the language well enough that building from source is painless.

Agree wholeheartedly. Just yesterday I saw what looked like a very useful tool whose install instructions were "npm install -g <thing>" and I felt a little pang of "ugh" but went ahead and did it anyway. Never got the thing running but did get a nice error message that said "Electron failed to install correctly, please delete node_modules/electron and try installing again." I actually said out loud, "why the hell does this tool use electron?!" and gave up.

Same. I suspect the correlation is because of the crate ecosystem and cargo making it easy to use a good TUI framework.

I think it's mostly because the people who write Rust now are the same kinds of people who care about the cli.

That's a lot more compelling to me than the safety argument I usually see. Maybe I'll look into it more at some point in the future

Check out ripgrep, bat, exa, fd; or just google top rust cli's etc. They are so, so good.

I’ve recently started syncing these single binaries along with zsh dotfiles using them to all the machines I use with syncthing and have been extremely happy with the result. Just had to set PATH=$HOME/Sync/$(uname)/$(uname -m):$PATH and haven’t been without a tool since.

https://hn.algolia.com/?q=written+in (this is by top score, you can also see this by date)

Lots of stuff where X != Rust.

> (why would an end user care?)

They probably don't. But, on GitHub, where programmers go, and on a forum like this, with lots of developers, this means something to many folks here.

Non-rust programmer here, written in X does not tell me anything about the app.

I mean, it tells you that if you wanted to contribute, you'd also have to learn Rust before being able to do so.

Also, beyond that, note my "many"; sure, it doesn't mean anything to you. It feels weird to get upset about something that supposedly has no meaning, though. When I see something that's "written in X" and I don't care about X, I just move on with my day. If I post about how much I don't care about X, it really seems like I do actually care about X, otherwise, I wouldn't take the time to complain about it.

(The OP of this thread said that they did care, negatively. And you didn't explicitly say you don't care either. But this attitude is common and is also elsewhere in this thread so, let me just finally jump off this soapbox real quick...)

I agree with the other poster that it doesn't add anything to the title. I'm not "upset" about it, this is simply a side discussion about what goes in a title. I wanted to add my thoughts and now I'm done.

Surely you can think of one thing that it tells you

It seems cyclical. There was a time when it was "written in Go" that was on the bingo card. I remember this being an original hype point for docker, for instance.

I do agree that I'd rather see the technology choice not highlighted so much, though. But it really is a proxy for some things that matter to me: if it is in Rust, I expect it to be a single binary I can easily install and which starts up quickly and performs well (as opposed to, say, javascript, python, ruby, or java programs), but is likely to suffer very few security problems (as opposed to, say, a C or C++ program), and has a decent shot at growing an active community, which I may be able to contribute to. A lot of these things apply to other technologies (like the aforementioned Go) as well, but it I do feel like I get some information by knowing the technology.

Maybe the better approach would be to enumerate benefits rather than saying what the technology is, even if the benefits flow from the technology choice.

When I’m looking for a new open source application, for various reasons of experience and preferences about what I work with, I often look for and prefer relatively new projects in compiled, statically typed languages that are reasonably active and mature codebases.

That means the solution often ends up being written Go, Rust or C/C++, to the extent that Go and Rust specifically also turn out to be good signalling that the application may meet my other requirements, partly simply because they are relatively new languages. They also seem to signal better thought out and more “serious” solutions, perhaps because of the barrier to entry they present vs interpreted and scripting languages.

So yeah. Rust or other languages certainly aren’t a requirement but they do sometimes present some useful signalling.

You are definitely not the only one.

Nothing has put me off of Rust more than "Rust people". The same thing happened to me with Python over 20 years ago and I've successfully (and happily) avoided that language for over 20 years because of just how unbelievably, unstoppably "effervescent" Python people were in the late 1990s. If Python were 10% as good as those people said is was, it would be, by a wide margin, mankind's greatest achievement for the next 10 millennia.

I get the exact same vibe from "Rustaceans."

They are just (a bit too much) excited...

Statically checked safety is a big deal.

I mean it can have a crappy UI, and not do what you want it to, but at least you won't have a memory leak.

I think for most applications, written in rust shouldn't be a selling point, but things such a cURL can definitely benefit from it

If it's open source, the audience isn't just end users, it's also programmers. They might be interested in looking at the code, reusing it in some way, or even contributing to the project.

Yeah! I think Rust is mature enough now that this isn't necessary from a novelty perspective. And it's not really an obscure language. "In Haskell" for eg is still warranted because it's obscure and uncommon, but I don't think this fits Rust.

I'd add "alternative to popular software" to that list as well. People love to find alternatives that are supposedly better and not from big corporations.

For an even catchier headline, it should be prefaced with: "How I built a..."

I'd seriously like someone to build a HN scraper that aggregates all the articles prefaced by "How I built a..." and then write an article called "How I built a scraper to aggregate 'How I built a...' articles to assess their HN upvotes." I'd upvote that.

Hah, look at the author's user profile!

It’s missing “Open Source” and “artisnal”. Obviously its’s crap /a

Good catch!

and end-to-end encrypted