mattbit

From my experience, in a majority of real-world LLMs applications, prompt injection is not a primary concern.

The systems that I see most commonly deployed in practice are chatbots that use retrieval-augmented generation. These chatbots are typically very constrained: they can't use the internet, they can't execute tools, and essentially just serve as an interface to non-confidential knowledge bases.

While abuse through prompt injection is possible, its impact is limited. Leaking the prompt is just uninteresting, and hijacking the system to freeload on the LLM could be a thing, but it's easily addressable by rate limiting or other relatively simple techniques.

In many cases, for a company is much more dangerous if their chatbot produces toxic/wrong/inappropriate answers. Think of an e-commerce chatbot that gives false information about refund conditions, or an educational bot that starts exposing children to violent content. These situations can be a hugely problematic from a legal and reputational standpoint.

The fact that some nerd, with some crafty and intricate prompts, intentionally manages to get some weird answer out of the LLM is almost always secondary with respect to the above issues.

However, I think the criticism is legitimate: one reason we are limited to such dumb applications of LLMs is precisely because we have not solved prompt injection, and deploying a more powerful LLM-based system would be too risky. Solving that issue could unlock a lot of the currently unexploited potential of LLMs.

This is how McIntosh & Della Sala put it:

> in the academic literature, it has been suggested that the signature pattern of the DKE (Figure 1A) might be nothing more than a statistical artefact. In a typical study, people’s tendencies to under- or overestimation are analysed as a function of their ability for the task. This involves a ‘double dipping’ into the data because the task performance score is used once to rank people for ability, and then again to determine whether the self-estimate is an under- or over-estimate. This dubious double-dipping makes the analysis prone to a slippery statistical phenomenon called ‘regression to the mean’.

This is not ‘autocorrelation’, it is regression to the mean. I find the article unclear and imprecise. For those interested in a better overview of the Dunning–Kruger effect, I recommend this short article by McIntosh & Della Sala instead:

https://www.bps.org.uk/psychologist/persistent-irony-dunning...

There’s actually two ways of seeing this: the three-domain system and the two-domain system.

The three-domain system divides life in Archaea, Bacteria, and Eukarya. In this system, Archaea and Bacteria can be grouped together as prokaryotes.

In the two-domain system, the division is between Archaea and Bacteria. In this case, eukaryotes are seen as a subgroup of Archaea.

Hope to have cleared up some of the confusion.

Hey, Giskard team member here! I am around to discuss and read your feedback.

I’ve worked in particular on automatic scanning of ML models for bugs and problems, the idea was to systematically scan for general issues and automatically find segments of data on which the model performs worse than average. If you have questions, I am happy to discuss here.

Not exactly, metamorphic testing does not need an oracle. That’s actually the reason of its popularity in ML testing. It works by perturbing the input in a way that will produce a predictable variation of the output (or possibly no variation).

Take for example a credit scoring model: you can reasonably expect that if you increase the liquidity, the credit score should not decrease. In general it is relatively easy to come up with a set of assumptions on the effect of perturbation, which allows evaluating the robustness of a model without knowing the exact ground truth.

Yeah, LightOn has been doing this for years already, it's kind of strange that there was no mention of them in the article. I know them because their offices are close to the research center where I work (in Paris, France). If I'm not wrong they were planning to offer their optical processor as a cloud service too.

> Now what does does a mathematician do? He tries to understand nature and uses mathematics as a language to do that.

I would argue that this is wrong. That's what physicists do, not mathematicians. Mathematics is about abstract ideas, which can live regardless of nature or application. Physics instead is about understanding nature. Most physicists use mathematics to do that, but that's just for practical reasons. They don't always take it for granted, there's a very famous article by Eugene Wigner on this: “The Unreasonable Effectiveness of Mathematics in the Natural Sciences”.

I think it's important to understand this. Sure, computer programming is not math. Physics is not math either. Mathematics is kind of a way of thinking, and mathematical language turns out to be very useful in describing and understanding nature and many other things. Computer programming theory stems out of mathematics, but I agree that everyday programming practice does not strictly require an in-depth math knowledge.

But it depends. One day you wake up and you want to solve a problem: sometimes you need programming, sometimes you need math, sometimes both, or maybe you need some business experience, psychology, whatever. We need different perspectives, I don't think we can compartmentalize these things any more.

“Obtaining structured data from this system is costly and time-consuming due to the need to tell the system you’re not a robot every few searches. It also doesn’t include some key categories of information formerly available via the API, including the primary charge, bond hearing dates, IR number, and the FBI code associated with the type of arrest.”

It seems like they are just trying to steal the name of uBlock. AdBlock CEO says they “do love the name” [0] and are "investing heavily" into the product, but the commits in the repo [1] are just cosmetic changes and rebranding. Oddly enough, the only active committer is anonymous.

I wonder if this will cause legal troubles to the true uBlock Origin.

[0]: https://twitter.com/judemaier/status/1020034358558670848 [1]: https://github.com/uBlock-LLC/uBlock/commits/master