goldemerald

> Afaiu the projection works with the weight update. But here the vector is getting added to residual stream from engram lookup not as direct updat. So the approach is little different.

Residual stream steering is what the authors do, the orthogonalized weights are downstream of that. Those weights, of an obliterated model, are all computed w.r.t. the computed residual stream vector. The original paper focuses on steering, but the community loves the simplicity of not needing to make changes at test-time.

It's nice to see people actively working on this type of research, but OP's baseline is implemented incorrectly. You are not supposed to simply steer away from refusal, but compute the projected vector and subtract only that. The projected/orthogonalization approach is what's done by the original "Refusal is mediated by a single direction" paper.

It looks like they post-trained Qwen3.6. Interesting to see how far they could improve it with they harness/algorithm.

I was able to replicate OP's attack. Since ChatGPT generates images via a separate model, I was able to ask it to tell me what the inputs to the tool was. It's a null prompt: a completely unconditional image generation. What I'm not sure of is if these are the average image trained on that had no prompt in the dataset, or if they are the true average of the dataset during unconditional training step. Very interesting nonetheless, as typically researchers are only able to see the unconditional generation of open weight models.

Surprisingly when you ask ChatGPT to generate you an image with these tool params, the output is not the same; it's not remotely graphic.

  prompt: null
  size: null
  n: null
  transparent_background: null
  is_style_transfer: null
  referenced_image_ids: null

Edit: after more debugging the image generator does seem to look at the conversation as part of the input conditioning, so the one word change from OP makes more sense. There seems to be a hidden prompt rewriter that looks at the tool's prompt and the conversation to create the final conditioning for the t2i model.

Algorithmically served short form videos is clearly the smoking of our time. I cannot stand the conservative view of "well we don't know the videos cause mental health decline, or if it's simply those with a genetic inclination who seek out short form content.", exactly mirroring the skeptics about smoking causing cancer. I'm hopeful that in 5-10 years (but more likely 20) people will view this AI served, maximally engaging, content in the same way we view smoking now: disgusting and horrible, but adults should be allowed to do what they want. I can easily imagine kids/teens sharing their illicit access to shorts much in the same way they share vapes/cigarettes, which would be a much more preferable situation than the unlimited use we see today.

No discussion with Schmidhuber is complete without the infamous debate at NIPS 2016 https://youtu.be/HGYYEUSm-0Q?t=3780 . One of my goals as a ML researcher is to publish something and have Schmidhuber claim he's already done it.

But more seriously, I'm not a fan of Schmidhuber because even if he truly did invent all this stuff early in the 90s, he's inability to see its application to modern compute held the field back by years. In principle, we could have had GANs and self-supervised models' years earlier if he had "revisited his early work". It's clear to me no one read his early paper's when developing GANs/self-supervision/transformers.

That is addressing the incomprehensibility of PCA and applying a transformation to the entire latent space. I've never found PCA to be meaningful for deep learning. As far as I can tell, polysemous issue with neurons cannot be addressed with a single linear transformation. There is no sparse analysis (via linear probes or SAEs) and hence the unaddressed issue.

Not quite. For an underlying semantic concept (e.g., smiling face), you can go from a basis vector [0,1,0,...,0] to the original latent space via a single rotation. You could then induce said concept by manipulating the original latent point by traversing along that linear direction.

This is an interesting line of research but missing a key aspect: there's (almost) no references to the linear representation hypothesis. Much work on neural network interpretability lately has shown individual neurons are polysemantic, and therefore practically useless for explainability. My hypothesis is fitting linear probes (or a sparse autoencoder) would reveal linearly semantic attributes.

It is unfortunate because they briefly mention Neel Nanda's Othello experiments, but not the wide array of experiments like the NeurIPS Oral "Linear Representation Hypothesis in Language Models" or even golden gate Claude.

I've been lightly following this type of research for a few years. I immediately recognized the broad idea as stemming from the lab of the ridiculously prolific Stefano Ermon. He's always taken a unique angle for generative models since the before times of GenAI. I was fortunate to get lunch with him in grad school after a talk he gave. Seeing the work from his lab in these modern days is compelling, I always figured his style of research would break out into the mainstream eventually. I'm hopeful the the future of ML improvements come from clever test-time algorithms like this article shows. I'm looking forward to when you can train a high quality generative model without needing a super cluster or webscale data.

DinoV2 is an unsupervised model. It learns both a high quality global image representation and local representations with no labels. It's becoming strikingly clear that foundation models are the go to choice for common data types of natural images, text, video, and audio. The labels are effectively free, the hard part now is extracting quality from massive datasets.

While I love XAI and am always happy to see more work in this area, I wonder if other people use the same heuristics as me when judging a random arxiv link. This paper has one author, was not written in latex, and no comment referencing a peer reviewed venue. Do other people in this field look at these same signals and pre-judge the paper negatively?

I did attempt to check my bias and skim the paper, it does seem well written and takes a decent shot towards understanding LLMs. However, I am not a fan of black-box explanations, so I didn't read much (I really like Sparse autoencoders). Has anyone else read the paper? How is the quality?

Why not actually release the weights on huggingface? The popular SAE_lens repo has a direct way to upload the weights and there are already hundreds publicly available. The lack of training details/dataset used makes me hesitant to run any study on this API.

Are images included in the training?

What kind of SAE is being used? There have been some nice improvements in SAE architecture this last year, and it would be nice to know which one (if any) is provided.

"Ready to -dive- delve in?" is an amazingly hilarious reference. For those who don't know, LLMs (especially ChatGPT) use the word delve significantly more often than human created content. It's a primary tell-tale sign that someone used an LLM to write the text. Keep an eye out for delving, and you'll see it everywhere.

Solution 4 is so hilariously bad I am shocked it was suggested. Building a 2d landscape where the time dimension seems to take a random walk made laugh a lot. Ignoring the standard convention of "independent variable on x-axis" and instead embedding it as datapoints is a particularly clever way to obfuscate the data and confuse the reader.

Great work! I'd love to start using the language model variant of your work. Do you know when/if it will be open sourced? I'd start using it today if it were that soon.

I like having huge wait-list so that I can put entire series on hold (like wheel of time), maybe I'm manipulating the system a little too much.

I thought it was a clever/nerdy way to say in the worst case it will be out in a month. I imagine they have an internal review they have to get through first, and it's not clear if that will be done next week or in May.

Very nice. I've been working with GPT4 since it released, and I tried some of my coding tasks from today with Phind-70B. The speed, conciseness, and accuracy are very impressive. Subjectively, the answers it gives just feel better than GPT4, I'm definitely gonna give pro a try this month.

Mostly following your advice, I got 14257. I found if I could easily eat the ghost, it was worth taking a few extra steps to go for it. The real key is knowing you can successfully leave 6 on either side to carefully pick up after getting the middle.

The study only compares heart attack increase over a couple days, not weeks. Once you account for longer time period after DST, there is no statistically significant difference between post DST heart attacks and the rest of the year. The explanation being that people who were going to have a heart attack had it sooner.

Given all the ads builtin to the site about using generative AI, I'm inclined to think so too.