osaariki

That 200k is a reasonable amount to start withdrawing from a 5M portfolio (exactly the 4% rule from the Trinity study [1]), but you’ll want to adjust it for inflation every year. My favorite tool for planning these strategies is TPAW Planner [2], which visualizes the distribution of withdrawals under various market outcomes. It’ll also suggest a portfolio of stocks and bonds that’ll be safer than just T-bills, which have a high risk of not beating inflation.

1: https://www.bogleheads.org/wiki/Safe_withdrawal_rates

2: https://tpawplanner.com/

TUIs are great for low friction remote work. I do a lot of data processing work on remote VMs with a mix of interactive debugging/eyeballing and bulk jobs. TUIs are a great fit for the sorts of tools I build to support this work. The other UI paradigm I end up reaching for is locally hosted web UIs, as models are really good at one-shotting HTML reports with graphs and tables. Inside VS code those get automatically tunneled to the local machine.

You're exactly right. The llguidance library [1,2] seems to have emerged as the go-to solution for this by virtue of being >10X faster than its competition. It's work from some past colleagues of mine at Microsoft Research based on theory of (regex) derivatives, which we perviously used to ship a novel kind of regex engine for .NET. It's cool work and AFAIK should ensure full adherence to a JSON grammar.

llguidance is used in vLLM, SGLang, internally at OpenAI and elsewhere. At the same time, I also see a non-trivial JSON error rate from Gemini models in large scale synthetic generations, so perhaps Google hasn't seen the "llight" yet and are using something less principled.

1: https://guidance-ai.github.io/llguidance/llg-go-brrr 2: https://github.com/guidance-ai/llguidance

I live half way across the world from my folks so I don’t see them often. I’d love something that gives me a greater sense of presence than a video call can give.

We don’t know that LLMs generating tokens for scenarios involving simulations of conscious don’t already involve such experience. Certainly such threads of consciousness would currently be much less coherent and fleeting than the human experience, but I see no reason to simply ignore the possibility. To whatever degree it is even coherent to talk about the conscious experience of others than yourself (p-zombies and such), I expect that as AIs’ long term coherency improves and AI minds become more tangible to us, people will settle into the same implicit assumption afforded to fellow humans that there is consciousness behind the cognition.

For some interesting context: this paper was a precursor to all the work on synthetic data at Microsoft Research that lead to the Phi series of SLMs. [1] It was an important demonstration of what carefully curated and clean data could do for language models.

1: https://arxiv.org/abs/2412.08905

Edge's Password Monitor feature uses homomorphic encryption to match passwords against a database of leaks without revealing anything about those passwords: https://www.microsoft.com/en-us/research/blog/password-monit... So not the first, but definitely cool to see more adoption!

I'm not familiar with how TypeChat works, but Guidance [1] is another similar project that can actually integrate into the token sampling to enforce formats.

[1]: https://github.com/microsoft/guidance

We have an early tool paper [1] for a previous version of the engine, but that's short and with POSIX semantics, so doesn't include a lot of the interesting stuff. The most relevant bit there is the handling of Unicode.

>This doesn't compose though, right? For example, if you have `sam|samwise`, then you can do that transformation, but if you have `\b(?:sam|samwise)\b` then you can't.

You'd get subsumption when you have something like '(?:sam)?wise|wise' and in fact this kind of subsumption due to a "nullable prefix" is the main one currently detected (because we encountered patterns that motivated it). And you're right that all these rewrites should compose regardless of context so that they can be eagerly applied in the AST constructors.

>> but a larger class of patterns get to stay in the DFA world with derivatives. >Can you say more about this?

Yeah, the easiest example I can point at is from that tool paper [1], where a subsumption-based rewrite for counted repetitions can turn an exponential blow-up into a linear one. Off the top of my head I think a pattern like '.*a[ab]{0,1000}' would have a 2^1000 blow-up when determinized into a DFA but stays linear in size with derivatives. However, that loop subsumption rule isn't valid as-is under PCRE semantics, so it still needs some work to be ported to the .NET engine.

Before we get that PLDI paper out the best resource is probably just the code under [2]. It's fairly well commented, but of course that's no substitute for a good write-up.

[1]: https://www.microsoft.com/en-us/research/uploads/prod/2019/0... [2]: https://github.com/dotnet/runtime/tree/main/src/libraries/Sy...

We built the new engine behind .NET's RegexOptions.NonBacktracking with derivatives. We will have a paper at PLDI this year on the work that went into that.

PCRE semantics was indeed the big thing that required new techniques. Basically, you have to modify the derivation function such that the derivatives model what paths through the pattern a backtracking engine would consider before stopping at the current position.

The big thing derivatives buys you is the ability to apply rewrite rules lazily during the construction. For example, when we can prove that a regex R subsumes a regex T, then an alternation R|T can be rewritten to just R, since T is already included in it. These kinds of rewrites often result in the DFA that gets constructed being minimal or close to so. Of course, you do pay somewhat for the machinery to do this, so best-case construction times suffer compared to in traditional NFA+lazy-DFA engines like RE2 or Rust's, but a larger class of patterns get to stay in the DFA world with derivatives.

I hope our work ignites a new interest in regex matching with derivatives. I believe the ability to apply these syntactic rewrites on-the-fly is really powerful and I'd love to see how far folks like you with extensive experience in optimizing regex matchers can take this.

I'd love for someone to do a good quality PyTorch enabled implementation of Sampled AlphaZero/MuZero [1]. RLLib has an AlphaZero, but it doesn't have the parallelized MCTS you really want to have and the "Sampled" part is another twist to it. It does implement a single player variant though, which I needed. This would be amazing for applying MCTS based RL to various hard combinatorial optimization problems. Case in point, AlphaTensor uses their internal implementation of Sampled AlphaZero.

An initial implementation might be doable in 5 hours for someone competent and familiar with RLLib's APIs, but could take much longer to really polish.

[1]: https://arxiv.org/abs/2104.06303

The in-place update work in Koka [1] is super impressive. One of my co-workers, Daan Leijen leads the Koka project and hearing his talks about it have been such a privilege. The work around Koka is really convincing me that functional languages will eventually lead the pack in the effort-to-performance trade-off.

Something that came out of the Koka project that everyone should know about is mimalloc [2]: if your built-in malloc is not doing it for you, this is the alternative allocator you should try first. Mimalloc is tiny and it has consistently great performance in concurrent environments.

[1]: https://koka-lang.github.io/koka/doc/index.html

[2]: https://github.com/microsoft/mimalloc

I've done work on low-latency FHE neural network inferencing [1] and we estimated the FLOPS to be approximately 4 times that of an Intel 8087 floating point coprocessor [2]. This was for a LeNet-5 network on the MNIST dataset with multicore evaluation on a workstation class machine.

My view is that this is already fast enough to support use cases that really need the unique capabilities of FHE. Since this work we've been focused on making FHE more usable with compilers and tooling [3]. Currently most FHE is being programmed like that Intel 8087 was: with the equivalent of assembly by directly calling functions in FHE libraries to perform arithmetic and crypto operations. Imagine having to do register allocation by hand for all of your code. The EVA compiler [4] is meant to be like a "C compiler for FHE", hiding low-level crypto concerns and providing common optimizations.

[1] "CHET: Compiler and Runtime for Homomorphic Evaluation of Tensor Programs": https://arxiv.org/pdf/1810.00845.pdf

[2] https://en.wikipedia.org/wiki/Intel_8087

[3] "EVA: An Encrypted Vector Arithmetic Language and Compiler for Efficient Homomorphic Computation": https://arxiv.org/pdf/1912.11951.pdf

[4] https://github.com/microsoft/EVA

I found this article confusing too. Being on a tech news site I expected it to eventually get to something concrete, perhaps producing chimeric colors [1] for an impactful moment -- something to justify the "hack your brain" line.

Instead the article just seems to say that color design for animation is a bit different.

[1]: https://en.m.wikipedia.org/wiki/Impossible_color

The article touches on DFA based engines and notes that they may not be the fastest in all use cases. To expand on that generally the reason for this would be that the automaton you get from the regular expression has too many states, which can happen for example with very large counters or cases where the state space includes a product of multiple counters due to non-determinism between them (backtracking engines would struggle with the latter case too though).

Something an automata based engine can do (and IIRC re2 does) is that instead of constructing the full DFA, you can construct the much smaller non-deterministic finite automaton (NFA) and perform the product construction on-the-fly while caching states you've seen. However, this only helps if your input doesn't actually visit all those states. A malicious input could target your NFA engine to cause states being constantly kicked out of the cache. The slowdown here is linear though, so you're still better off than with a backtracking engine.

A project I'm working on is the Symbolic Regex Engine (SRM) [1], which uses a symbolic variant of regex derivatives. This technique allows an often-minimal DFA to be constructed lazily. This is similar to NFA based engines, but results in much fewer states thus helping with the cache issue. SRM can also handle some classes of counters natively in how the automaton gets constructed, which can turn a blow-up that would be exponential in a DFA based engine into a linear one.

[1]: https://github.com/AutomataDotNet/srm

I do research on FHE. On a high level querying works by evaluating an equivalence function between the encrypted query key and keys in the database such that the result is 1 if the key matches and 0 otherwise. You can then multiply this with the values and sum all of them together to get just the row that matches.

For write operations you could do a similar trick to select a new version of each value based on the query key matching. You are in effect writing all rows, but using the old value when you don't want to change it.

Now, due to some quirks of FHE you can only do this write operation a limited number of times before you have to perform a more expensive operation called bootstrapping to "refresh" ciphertexts. I find bootstrapping to be the coolest part of homomorphic encryption: you encrypt the secret key itself with homomorphic encryption and then use it to decrypt inside FHE. The result is still encrypted but closer to a freshly encrypted ciphertext.

Exactly. And the magic is that now you can securely compute aggregations of data you've stored without having to download all of your data.

Say you have some field in all of your encrypted, cloud-stored records and want to find the sum of all of them. With traditional encryption you'd either have to trust the cloud with your decryption key or download all of the records for local processing, which could be very slow. With homomorphic encryption you can instead have a cloud service do that computation without any visibility to the actual values and only have to download a single ciphertext containing the result.

This is not how homomorphic encryptions works. The schemes in use are not deterministic.

More than 2^5 has already been demonstrated with GPU implementations, so they definitely do mean 10^5.

CPU is currently a rather natural baseline as the various GPU implementations are still somewhat preliminary. I'm not quite an expert on the implementation challenges around FHE (I work on compilers targeting FHE), but my understanding is that the access patterns of the core operation, the number theoretic transform (NTT, a DFT in a finite field), are not particularly GPU friendly. Dedicated hardware is expected to comparatively very well here.

Homomorphic encryption (HE) is NOT what people would understand as "on-chip cryptography", i.e., secure enclaves. HE is a form of encryption that allows arithmetic operations on encrypted numbers without any access to the decryption key. Secure enclaves on the other hand do decrypt data on-chip and are thus vulnerable to side channel attacks.

You can use HE to implement things like encrypted database lookups that don't reveal what is being queried (even to the database server) and the security of the encrypted query does not depend on any hardware properties of the server providing the service.

Hi! I'm one of the main researchers on this project. If anyone has any questions about the EVA compiler or homomorphic encryption in general I'd be happy to answer.

Both BGV and CKKS are so called batched schemes, which allows you to pack multiple values into a single ciphertext (typically thousands) and perform operations on all of the values for the cost of a single homomorphic operation. Batching is one of the larger sources of speedup since the discovery of fully homomorphic encryption.

The values in BGV and CKKS are integers and both schemes allow homomorphic integer multiplication and addition (CKKS gives approximate results). TFHE is actually not a batched scheme and moreover operates mostly on the level of single bits and boolean gate operations. It is quite difficult to directly compare TFHE with batched integer schemes.

Which scheme is best will ultimately depend a lot on the requirements of the application. If you're operating on integers and don't need bitwise operations, then an integer scheme is appropriate. Applications that need bitwise logic and can work with small bitwidths will be more suited for TFHE. CKKS is especially good for numeric and machine learning applications, because the approximation it implies can be managed and it allows using faster encryption parameters than BGV/BFV would.

You're quite correct that evaluating the back propagation could be done exactly the same as the forward pass. However, if you want to train for more than a handful of steps you'll have to use an operation called bootstrapping to periodically "refresh" the ciphertexts that encode the model. Bootstrapping is essentially evaluating the decryption circuit of HE using HE itself (with an homomorphically encrypted version of the secret key). The problem is that bootstrapping is much more expensive than the other operations in HE.

People have done very effective training on encrypted data using simpler models, like linear or logistic regression. See for example this work [1] from my colleagues at Microsoft Research.

[1]: https://bmcmedgenomics.biomedcentral.com/articles/10.1186/s1...

As other posters already mentioned, there's no control flow in the HE instruction set. This is actually one of the reasons neural networks are a great fit for HE, as they often have fixed data access patterns and no control flow.

TF Encrypted has focused more on MPC [1]. At least that's what their arXiv paper [2] talks about. It does seem they are also working to integrate Microsoft SEAL for HE.

[1]: https://en.wikipedia.org/wiki/Secure_multi-party_computation [2]: https://arxiv.org/abs/1810.08130

You're right, CryptoNets used a data layout optimized for throughput with a batch size 4096. Since then we've done a lot of work on low latency inference with our CHET compiler [1] and my colleagues with LoLa [2]. It all comes down to the data layouts you use.

[1]: https://www.cs.utexas.edu/~roshan/CHET.pdf [2]: https://arxiv.org/pdf/1812.10659.pdf

This is not how this works. Homomorphic encryption (HE) is best thought of as a software CPU that gives you ways to mutate data (but not read it without the secret key of course). Any computation you run has to be using the instruction set offered by HE: addition, multiplication or rotation. When you do an operation on two ciphertexts, you get a new ciphertext back, but no information on its contents.

Now the way you do machine learning here is by translating your model to use the instructions offered by HE. You've effectively recompiled the model to a new architecture.

If you'd like to read more about machine learning with homomorphic encryption, we published a paper on our CHET compiler [1]. I also talk about this space on a high level in this MSR podcast episode [2].

[1]: https://www.cs.utexas.edu/~roshan/CHET.pdf [2]: https://www.microsoft.com/en-us/research/blog/he-compilers-f...

This is wholly untrue. The guarantees HE gives for security of encrypted data is exactly the same as any traditional symmetric or asymmetric crypto system. HE is indeed malleable by design, but doing computation on homomorphically encrypted data gives no information on the contents of the ciphertexts.

>This is more like anonymization (effectively a one-way hash) than encryption. If you encrypt 2 different values with the same algorithm and key, you will get the same ciphertext which reveals information about the original value (i.e., that they are the same).

This is a misunderstanding. Homomorphic encryption schemes are randomized encryptions schemes, which means even if you encrypt the same value twice with the same encryption key you get two different ciphertexts (both look like "random data"). They just have the property that they will decrypt to the same value. This is possible because the space of ciphertexts is much larger than the space of plaintexts.