Personal plug, but have you tried Jsonrs[0]? I typically get much better performance out of it (especially in lean mode, which seems to be the mode you'd want to use for this benchmark) than Jiffy for large JSON encoding workloads.
It's not quite as high as I was seeing with `jiffy` (3,800 req/sec here vs 4,000+ with jiffy), but I'm not confident that was a totally fair comparison. `jiffy` doesn't integrate as nicely with Phoenix, so I was just calling `:jiffy.encode(...)` in the controller and then doing a `text(...)` response. I need to double-check if `json(...)` is doing more work here.
[0] https://github.com/losvedir/transit-lang-cmp/commit/140d693b...
Jsonrs supports encoding protocols by default, but lets you turn it off for a speed boost if you're okay with more Jiffy-like behavior. Plugging it in as Phoenix's JSON library won't turn off that protocol handling, so you're getting the slower operational mode of Jsonrs (which is fine and probably what you want in most cases for correctness, but will definitely eat a few ops/sec in a benchmark)
Also use a release, it will help some things here, especially on "short" benchmark, and make sure that the JIT is run.
Good call on running as a release! I'll try that next.
https://elixirforum.com/t/an-informal-comparison-of-several-...
----
It's not surprising that Elixir is not good at large json handling and complicated data structure manipulation. This reminds me of https://discord.com/blog/using-rust-to-scale-elixir-for-11-m...
At the same time I see that Elixir gave the most consistent response time. Max response time is less than 6x of median compared to ~x10 (Deno / C#) or x23 (Go / Rust) / x58 (Scala), probably thanks to preemtive scheduling of per-request process in Erlang VM
Regarding GenServer, I'm not so sure about that. I suppose I should benchmark it, but intuitively I expect ETS to be better here. There's more overhead in getting the data, but it allows you to concurrently read it from different processes. A GenServer, meanwhile, could respond to a given message faster, but now you're serializing the (e.g.) 50 concurrent virtual users through a single bottleneck. I could have multiple copies of the data in several GenServers, I suppose, at the expense of much more memory use.
These Elixir numbers seem suspect to me, based on my company's extensive benchmarking on many of these same languages. Elixir is somewhat slow at some tasks, but serving up web requests isn't one of them.
> The "billion dollar mistake" is important to me, and while C# has non-nullability sugar in its typesystem (i.e. with ? after a number of types), the type system wasn't as rigorous as I was maybe hoping.
I'm genuinely curious if you have any examples? Nullability checks may be bolted on, but once you enable them, they should consistently prevent you from dealing with any null values that you haven't explicitly allowed. And other than that, the only hole in the type system that I can think of is array covariance, which was unfortunately common at the time (Java had it also) as a way to skimp on generics, but which doesn't occur often today because using generic collections is much more common.
> At one point I had a bug because I did a stopWatch.Elapsed / 1000 by accident instead of stopWatch.ElapsedTicks / 1000. The former is a TimeSpan struct instead of a long like ElapsedTicks, so intuitively it feels like I shouldn't be able to divide it, though it did a best effort and did something to it, though I'm not quite sure what.
This particular one doesn't have anything to do with the type system per se, it's just the way TimeSpan itself works. I'm not sure why "intuitively it feels like I shouldn't be able to divide it", since there are fairly obvious definitions of arithmetic operations on spans and numbers - if T is 10 seconds, then surely T*2 is 20 seconds, and T/2 is 5 seconds? Which is exactly what TimeSpan does by overloading the corresponding operators:
https://learn.microsoft.com/en-us/dotnet/api/system.timespan...
Note that the type of (TimeSpan/double) is still TimeSpan, so the type system is still enforcing proper use - it wouldn't have let you assign it to an int or a double. But it looks like you were just printing it, which is legal for any type:
Console.WriteLine($"loaded stop_times.txt in {watch.ElapsedMilliseconds} ms");I’m not the one you asked, but there’s a difference between you no longer having to deal with null checks and your program no longer having to deal with null checks.
The CLR will still do the null checks and fail your program fast when needed. Other languages will require you to properly handle them (or at least, make it obvious where you chose to bluntly abort the program when they occur)
I also don’t think C# will prevent you from forgetting to do a if null check in cases where you expect values to sometimes be null.
It should throw a warning in situations where a non-nullable value is not clearly non-null when it’s dereferenced. So you won’t get a warning in the body of a method declaration with a non-billable parameter, for example, but you will get a warning if you attempt to pass null into it.
https://learn.microsoft.com/en-us/dotnet/csharp/nullable-ref...
So yes, "will require you to properly handle them" is exactly what I meant.
That approximates it, and likely gives you most of the benefits, but it isn’t the same as strict enforcement of necessary null checks.
There will be cases where the programmer can prove a value isn’t null, but the compiler can’t, and thus, the programmer can choose to ignore the warning.
That’s similar to how Java tries to prove that a variable is initialized when being read. (https://docs.oracle.com/javase/specs/jls/se9/html/jls-16.htm...).
Java makes the safe choice, though. It refuses to compile such programs, thus guaranteeing that every local variable and blank final field will be initialized before first use.
And if you ignore the warnings, that's a deliberate opt-out; why even bother with nullability checks then?
- enable source generation for JSON serialization
https://learn.microsoft.com/en-us/dotnet/standard/serializat...
- use `TryGetValue` to do single dictionary lookup instead of two.
if (TripsIxByRoute.TryGetValue(route, out var tripIxs))
- specify List's capacity, if it's known when creating it, to avoid resizing operations while adding elements to the List var schedules = new List<StopTimeResponse>(stopTimeIxs.Count);
Since I submitted it, though, I posted it to /r/rust and got a lot of feedback. At the time, rust and dotnet were comparable and at the top of the list, with ~10k req/sec. Now rust is far and away the most performant at ~20k req/sec! I also was able to improve Go's performance 30% or so. Still, I want to let the other communities chip in and see if I can improve them.
I was actually just in the midst of exploring how to improve Elixir's results. I'm finding I can almost double my requests per second from simply switching the JSON encoder from Jason to Jiffy. That sort of surprised me since Jason is the de-facto standard, and I thought was super fast.
[0] https://news.ycombinator.com/item?id=33291604