fantispug
Compute is only an issue if you want to train large foundational models from scratch. You can easily train non-neural models on CPU, and fine-tune smaller models (say <4B) on a commodity GPU.
For training data there's lots of datasets online (like Kaggle), but you can build your own and can label things yourself or get a language model to label them for you. Getting good data is the hardest part of ML so this is a good exercise.
Then do some hobby projects: classify whether your photos contain a photo of you or a particular friend/family member, predict the expense category from bank transaction exports, classify your emails into categories, extract book recommendations from Hacker News, ...
Boosted trees over embeddings work surprisingly well too.
When a new model is released they have been able to have a working implementation days later, not months when the model was redundant. And often it's much more usable than the one-off, unmaintained, research code written in a very fragile environment.
BLOOM was a decent fully open and very large model for its time, and a huge coordination effort. It was around the same time as another "joke", OPT, but that gave Meta the knowledge that let them build Llama. Huggingface didn't have the money to compete in that race, but I'd bet it helped them stay at the frontier of AI software.
Google could have owned this space; they had TensorFlow hub, but it was too hard to use and too locked down. Huggingface made it easy to run and train transformers, and got the timing just right.
I'm glad someone is trying to build truly open models (along with the great work at Allen AI), because the likes of Meta and Alibaba are only going to release weights as long as it suits their business.
I guess it lets them better utilise their hardware in quiet times throughout the day. It's interesting they all picked 50% discount.
LLMs, along with image and video generation models, are generators of very dynamic, engaging and personalised content. If Open AI or anyone else wins a monopoly there it could be terrible for Meta's business. Commoditizing it with Llama, and at the same time building internal capability and a community for their LLMs, was solid strategy from Meta.
Each time you embed a document you search for approximate nearest neighbours before adding it, so it is O(N) like MinHash. Vector indexes like HNSW and PQ have better performance/quality tradeoffs than SimHash LSH which is the analogue of MinHash for cosine distance.
The quality depends on what you mean by near duplicate and the embedding model you use. Current models work well, and if you have labelled data you can fine tune them to be better.
The main drawback is the additional cost of embedding all the documents, especially for longer documents. But this cost has dropped really quickly with smaller models, better optimisations, and faster hardware.
What might be missing is in-context learning, prompt engineering, novel forms of attention, RLHF, and LoRA (though it covers adaptors), but this is still changing rapidly and the details may be irrelevant in another year. If you have a look at a recent course like Stanford CS224N 2023 there's a lot of overlap.
In general it won't solve all your problems, but it's helpful for automating simple things like this (but you still need to test edge cases). With this prompt in Python (which I'm more familiar with):
from dataclasses import dataclass
from typing import TypeVar
T = TypeVar('T')
@dataclass
class MaxDict:
items: dict[T, float]
max_value_item: T
def add_item(
def add_item(self, item: T, value: float):
if value > self.items[self.max_value_item]:
self.max_value_item = item
self.items[item] = value
// Print the fibonnaci sequence, except the number 5.
int main() {
int a = 0;
int b = 1;
int c = 0;
while (c < 100) {
c = a + b;
a = b;
b = c;
if (c != 5) {
cout << c << endl;
}
}
return 0;In both cases I think this is a real opportunity; we can let students get more quickly to bigger problems and systems thinking by leveraging these tools. It requires professors to start thinking innovatively about how to teach and assess these subjects.
String theory tries to do the same thing with other forces; so it's not clear whether there's such a transformation. Forces are technically simpler than embedding extra dimensions that there are no evidence for.
I think it would be interesting to see how quickly you could get from there to a useful system, say one where you could download the USB driver from github and compile it. But I definitely don't know enough to do that.
[1] https://www.cl.cam.ac.uk/projects/raspberrypi/tutorials/os/
For tabular data (which is probably most relevant in Pharma, and probably the best place to start) Introduction to Statistical Learning by Hastie et al and Max Kuhn's Applied Predictive modelling cover a lot of the classical techniques.
For univariate time series forecasting "Forecasting Principles and Practice" is great.
For natural language processing foundations Jurafsky's Speech and Language Processing is broadly recommended; for cutting edge natural language processing Stanford's CS224n is great: http://web.stanford.edu/class/cs224n/
But it's ok to start using libraries and fitting models without understanding how they work deeply, and coming back to these books later (just make sure you come back; there's lots of useful ideas in them!) In which case I'd recommend some of the resources the parent doesn't recommend
If you want to go an applied route I'd suggest starting somewhere like Kaggle and looking through the competitions for ones vaguely similar to yours. They've done all the hard work of choosing a challenging but solvable problem, sourcing and splitting the data, and choosing a metric. You then can see what techniques actually work really well, and benchmark different approaches. Academic challenges like Imagenet or Coco are also good for this, but you'll have to work harder to find relevant resources.
Once you've done this a couple of times, you can start framing your own problems, collecting and annotating your own datasets, deploying and maintaining models.
They claim they're making it more accessible with pedagogical examples (and innovations in representing differential vector calculus).
I still prefer index notation. Their notation looks pretty intricate and I think it would be really hard to do long calculations in it. It's also not clear to me how you'd generalise it to non-commutative vector algebras like in quantum mechanics.
I don't have much experience with packrat - but as opposed to pip it's another thing you need to discover and install. And so people don't do it by default when releasing code, and I've had to bisect versions of dependencies to get a working version of code. This can happen in Python too, but is rarer.
I disagree with the "learning curve"; if you've learned other programming languages Python has a pretty simple and familiar core, and Pandas (while the API is an inconsistent mess) is well documented. Base R is quirky compared with modern programming languages, and the API is pretty inconsistent.
I also strongly disagree with the Tidyverse bashing. I'd say it has the shortest learning curve (especially for someone familiar with SQL), and is one of the main reasons I still use R today outside of deep learning - I find it much more friendly to work with than any alternative.
A compatibility interval (at an agreed upon arbitrary level) communicates the magnitude of the difference as well as the uncertainty, which makes it much better for comparing options.
If my medication alleviates symptoms for 94-95% of patients above the current gold standard of 92-93% you could say it's "statistically significantly better", but the marginal improvement may not be worth the investment. Conversely if my medication alleviates symptoms for 50-80% of patients and the gold standard is 45-55% it would at least warrant future research (and if my medication has fewer severe adverse events it might be a better bet overall).
But this is just one small part of the whole picture: ideally we'd have preregistered experiments, experimental data published (or available to researchers where not possible due to confidentiality) and incentives for replication. Maybe this is too much for every field of science, but for ones where a wrong decision could have a severely detrimental impact they would create much more value than moving the P value.
He wrote TeX82 (modern TeX) in WEB, a literate programming language he invented that could be tangled into the Pascal machine code or weaved into literate documentation in ... TeX itself.
In fact he wrote WEB itself in WEB and by hand compiled the output of the tangle program, which he ran on the tangle source to get the same output.
He also wrote the program METAFONT to produce the typefaces, which he used to design the Computer Modern font.
And then he used that to write instruction books for TeX and METAFONT as well as new editions of The Art of Computer Science and others.
TeX is still in regular use in the academic community (ported via Web2C) over 35 years later, despite the changes in technology in that time (with hacks to deal with modern fonts, outputting PDFs instead of DVI, including images and the like).
He taught the yak to shave itself, and it's still clean shaven 35 years later.
Does it matter whether mathematicians think it is handwavy? If any two trained practitioners can get to the same result of a calculation and it matches experiment to appropriate approximations that seems good enough to me.
What's really hard is moving from these first principles to modelling real world phenomena; there are huge problems like protein folding, low temperature superconductors and even just predicting properties of new materials and chemicals.
It's still important for some people to work on fundamental physics, but I suspect there is a lot more opportunity in mere phenomenology.
Python now has type hints and good static checking libraries so you can add as much type checking as your want. Java's type system is atrocious (c.f. Haskell, OCaml); it doesn't check nulls (a common bug), making simple constructs (tuples, unions) often requires a new class file of boilerplate.
The HelloWorldApp is a bad fit for objects. All the functions are static. This confused me for a long time learning Java. You can get a very long way in Python without knowing @staticmethod.
The static nature of Java does make it easy to know where to look for things and introspect code; there tends to be less magic.