leod

Happy to see people working on vector search in Rust. Keep it up!

As far as HNSW implementations go, this one appears to be almost entirely unfinished. Node insertion logic is missing (https://github.com/swapneel/hnsw-rust/blob/b8ef946bd76112250...) and so is the base layer beam search.

This is a fair point and also called out in the discussion section. To some degree, this could be mitigated by hotloading shader code (and compiling shaders in debug mode). However, this remains as a fundamental downside of the approach.

Personally, I think that this is a price worth paying!

Awesome work.

Would you be willing to share details about the fine-tuning procedure, such as the initialization, learning rate schedule, batch size, etc.? I'd love to learn more.

Background: I've been playing around with generating image sequences from sliding windows of audio. The idea roughly works, but the model training gets stuck due to the difficulty of the task.

Interesting. They train an image classifier to detect images that were generated by a GAN-trained CNN. I wonder if it could be possible to include this classifier in the training loss, such that the generated images fly under its radar as much as possible. If this makes sense, then I guess the cat-and-mouse game just gained another level. On the other hand, what the classifier is detecting could be a fingerprint of the CNN architecture itself.

(Full disclosure: I have only read the abstract so far.)

Ha! In the model's defense, its training data [1] ends in 2017 -- not sure if hatred for Electron was as prevalent back then.

[1] https://archive.org/details/14566367HackerNewsCommentsAndSto...

Thanks! I actually planned to make results shareable at the start, but, knowing the internet, I did not like the idea of being held responsible for whatever content (say offensive or even illegal things) people would put into the titles.

I've trained a Transformer encoder-decoder model (this was slightly before GPT2 came out) to generate HN comments from titles. There is a demo running at https://hncynic.leod.org

Thank you so much for your comprehensive answer, this helps a lot.

If I understand nshepperd's code correctly, it uses a constant and small learning rate. Do you know if this works better than the learning rate schedule that is usually used for Transformer models (https://www.tensorflow.org/alpha/tutorials/text/transformer_...)?

I haven't heard of gradient checkpointing yet, thank you for the link! Do you know how it compares to gradient accumulation? The latter basically reduces the batch size, but takes the sum of multiple gradients before actually performing an update, thereby having the same effect as the original batch size.

The generated titles are great! You can put them into hncynic (https://github.com/leod/hncynic) to get closer to a fully generated HN experience.

If humans are not limited by the halting problem, it would be great if you could tell me if the following function f halts for all integers n: https://gist.github.com/leod/9b89af30cff21cb925d4522a68c990d...

Thank you!

The model weighs in at 1.2GB with 100M parameters, which is similar to the smallest GPT-2 model.

I wouldn't be suprised if GPT-2 small (+ finetuning on HN data) performed better than what I have trained. Other than hyperparameters, I think there are two main differences: First, I pretrained the model solely on Wikipedia data, while GPT-2 used more general web data. Second, I used an encoder-decoder model, while GPT-2 is a language model. I'm suspecting that the encoder is not very useful for this task.

This is true for some scenarios, like invalidating iterators through deletion (detected at compile-time by the borrow checker), but other scenarios still require runtime checks, right? Consider e.g. array out of bound accesses -- are you aware of approaches that move bounds checks to compile-time? It seems to me that this would be a painstaking process that would require programmers to annotate their code in many places to enable compile-time verification.

This analysis seems to be based on some notion of "Cognitive Complexity", but I can't find its definition in the article. Am I missing something?

True, but that does not show that the approximation can always be learned from examples, does it?

That's certainly an interesting idea, but wouldn't some people just "reply all" for more visibility?

It certainly struck me as an odd analogy. There are plenty of theories that are hard to disprove, why choose one as politically loaded as "Marxism"?

I quite enjoyed The User Illusion (the author has a great way of looking at the world). I think its chapters on consciousness are mostly about what consciousness gets to see. Building up from the idea that there is a very limited amount of information (as in Shannon) going to consciousness per second, plus some other experimental evidence (e.g. Benjamin Libet's experiments), to his thesis that consciousness is essentially a "user illusion".

But if I recall correctly, the book doesn't touch the question of how consciousness can arise to begin with, other than making a passing reference to GEB's strange loops.

I can see some value, for example, in the idea that consciousness (maybe I should rather say subjective perception or awareness) somehow emerges out of a reasonably complex system. But that wouldn't explain what it is in our universe that allows awareness to emerge to begin with.

I think some fun questions to think about are:

* Can we create awareness in a circuit by making it complex enough? What are the requirements?

* If we simulate a universe, does it inherit the ability to create awareness from our universe?

* How the heck do you validate any of these answers?

Well, I'm confused now, consciousness is too hard. Sorry if I sound like gibberish.

EDIT: This isn't meant to be a criticism of the book, which I can only recommend to anyone.

I'm a Vim user as well, and when I tried Colemak for a few months some years back, Vim was what made me go back to QWERTY. It was just too awkward as I would like to be able to work with both layouts.

Creating new bindings didn't work for me at all (especially since I didn't want to lose the ability to use someone else's Vim, or a vanilla Vim). I think the key might be to focus on what shortcut key you're actually pressing instead of where it is on the keyboard. This turned out to be pretty difficult for me with the hjkl keys, which are, as you say, fairly odd.

Interestingly enough though, today I can still write Colemak fluently whenever I accidentally switch to it, and it's a lot of fun! So I might give this whole thing with Colemak a new try, knowing that someone went the same route as I (keeping the default Vim shortcut bindings).

Maybe they believe that there is a higher long-term profit in having well-educated workers.

This is, of course, assuming, that the government is creating this law for the reason of solving crime and not for some special interests / lobbies.

Saying that the modern SPD is leftwing is like saying that the Democrats in the USA are leftwing.