parrt

That definitely makes it seem like it's noticing a great deal of its context window. impressive.

Wow. Cool. I have access to that model and have also seen some impressive context extraction. It also gave a really good summary of a large code base that I dumped in. I saw somebody analyze a huge log file, but we really need something like this needle in a needlestack to help identify when models might be missing something. At the very least, this could give model developers something to analyze their proposed models.

The article shows how much better GPT-4o is at paying attention across its input window compared to GPT-4 Turbo and Claude-3 Sonnet.

We've needed an upgrade to needle in a haystack for a while and this "Needle In A Needlestack" is a good next step! NIAN creates a prompt that includes thousands of limericks and the prompt asks a question about one limerick at a specific location.

BTW, how did you manage all of the throughput to the models and navigate the various throttling strategies for all the models you mentioned?

Looks really cool and useful. Seems like GPT-4o it's a lot better than 4.

Sure thing! Very enjoyable to have people use our work.

:) Yeah, I use my own internal markdown to generate really nice html (with fast latex-derived images for equations) and then full-on latex. (tool is https://github.com/parrt/bookish)

I prefer reading on the web unless I'm offline. The latex its super handy for printing a nice document.

Glad to be of assistance! Yeah, It really annoyed me that this critical information was not listed in any one particular spot.

Thanks! Took me a year to discover the key nut there. L1 vs L2 regularization is not well described I found so I went nuts trying to nail it down.

Also note we recently added 1D and 2D classifier decision boundary plots. See https://github.com/parrt/dtreeviz/blob/master/notebooks/clas...

Thanks. It's morphed over time as we add functionality so it's less clean than before.

That's a good idea. thanks!

Thanks! It took forever to bash my way to victory on that trees. The lib also supports the shallow trees in boosting machines.

Looks like my ANTLR 4 reference book is available, if anybody is interested.

True, but you have to do it one at a time to look at the variables. Also, the exception message doesn't tell you which operator among several is the issue. This just makes it easier. :)

Thanks, Jeremy. :) I didn't go into super huge detail in the article on the implementation part as most readers won't have interest in language nerd details like you and I do.

I wondered about that. It might work just calling my internal pyviz("some python code in string) function from the debugger. It'll execute in context (maybe?)

BTW, here's a link to the twitter thread.

https://twitter.com/the_antlr_guy/status/1313569854918590465

One of the biggest challenges when writing code to implement deep learning networks is getting all of the tensor (matrix and vector) dimensions to line up properly, even when using predefined network layers. This article describes a new library called TensorSensor that clarifies exceptions by augmenting messages and visualizing Python code to indicate the shape of tensor variables. It works with Tensorflow, PyTorch, and Numpy, as well as higher-level libraries like Keras and fastai.

Thankfully pytorch does all of that calculus for us!

Glad to be of service. Yeah, I just couldn't see what RNNs were doing through all of the neural net stuff. Much easier to think of it as just gradually transforming a vector. I also can't see linear algebra w/o visualizing different size boxes :)

Vanilla recurrent neural networks (RNNs) form the basis of more sophisticated models, such as LSTMs and GRUs. There are lots of great articles, books, and videos that describe the functionality, mathematics, and behavior of RNNs so, don't worry, this isn't yet another rehash. (See below for a list of resources.) My goal is to present an explanation that avoids the neural network metaphor, stripping it down to its essence—a series of vector transformations that result in embeddings for variable-length input vectors.

Here's the abstract: Practitioners use feature importance to rank and eliminate weak predictors during model development in an effort to simplify models and improve generality. Unfortunately, they also routinely conflate such feature importance measures with feature impact, the isolated effect of an explanatory variable on the response variable. This can lead to real-world consequences when importance is inappropriately interpreted as impact for business or medical insight purposes. The dominant approach for computing importances is through interrogation of a fitted model, which works well for feature selection, but gives distorted measures of feature impact. The same method applied to the same data set can yield different feature importances, depending on the model, leading us to conclude that impact should be computed directly from the data. While there are nonparametric feature selection algorithms, they typically provide feature rankings, rather than measures of impact or importance. They also typically focus on single-variable associations with the response. In this paper, we give mathematical definitions of feature impact and importance, derived from partial dependence curves, that operate directly on the data. To assess quality, we show that features ranked by these definitions are competitive with existing feature selection techniques using three real data sets for predictive tasks.

Indeed; lagrange multipliers are your friend! My problem initially was just the disconnect between the picture and the soft-constraint lagrange multiplier. Just couldn't figure out how that thresholded like that. :)

haha. :) Yeah, I got stuck for SOOooo long trying to reconcile the standard picture from ESL book with the math. Turns out they don't match! The picture is for a hard constraint whereas the math penalty term just makes bigger coefficients more expensive. Man, that really threw me.

The world certainly doesn't need yet another article on the mechanics of regularized linear models. What's lacking is a simple and intuitive explanation for what exactly is going on during regularization. The goal of this article is to explain how regularization behaves visually, dispelling some myths and answering important questions along the way.