So github copilot is pretty neat I entered this code:
result_dataFrame['yearOverYearMin'] = year_dataFrame.groupby('partId')['unitCost'].min()
result_dataFrame['yearOverYearMax'] = year_dataFrame.groupby('partId')['unitCost'].max() result_dataFrame['yearOverYearChange'] = result_dataFrame['yearOverYearMax'] - result_dataFrame['yearOverYearMin']
> result_dataFrame['yearOverYearMax'] = year_dataFrame.groupby('partId')['unitCost'].max()
> And big productivity booster on code entry.
Try reducing the DRY violations and you might have to write less. Almost the entire right side of the first two lines is repeated and third line is only that long because it has to recover the results from the map instead of using local variables.
While tools like boxing gloves reduce the damage your face takes it is usually better to just try and stop hitting yourself first.
There's DRY, then there's premature optimization.
I am unsure about what could possibly be better about them? Twice the chance to misspell one of the string constants? Twice the time to write the code, which in this case seems to be the big blocker? Twice the code to update in case any of it has to change? Twice the complexity per line of code?
In any case, DRY is just an optimization theme. And optimizing for understanding/intent is a higher priority for me. Sometimes they intersect, sometimes not.
It was a bit of an emergency and I was figuring Pandas out at the same time. The un-dry structure is handy for me because I can quickly tweak individual things since as a company we are still figuring out what is needed in the report.
I was writing a parser for URL query strings (?these=things&key=value) and at one point Copilot suggested
return [decodeURIComponent(key), decodeURIComponent(value)];
However it is shockingly good for filling out Python snippets - ie smarter autocomplete when teaching.
Popular libraries like Pandas, Beautiful Soup, Flask are perfect for this.
About 80% time it will fill out the code exactly they way I would want. About 10% time it will be something you want to correct or nudge.
Then about 10% of time it will be a howler or anti-pattern.
Then you simply explain it to students why it is not so great to say insert something at a beginning of a Python list.
Edit: Copilot is also great for filling out comments when teaching
# it can generate Captain Obvious what comments and also some nice why type comments as well
We're only a year into this thing existing.
I'd be VERY surprised if the data from all the interactions with it don't result in a product people care less and less about Microsoft harvesting their data from because of the value produced in exchange.
By 2025, either these models will have hit a wall on diminishing returns and it will take a complete pivot to some other approach to continue to see notable gains, or the products will continue to have improved at compounding rates and access will have become business critical in any industry with a modicum of competition.
Is there a single example in AI, or even technology as a whole, where simply continuing to apply one technique has led to compounding growth? Even Moore's Law, according to none other than Jim Keller[1], is more a consequence of thousands of individual innovations that are each quite distinct from others but that build on each other to create this compounding growth we see. There is no similar curve for AI.
In this case, GPT-3 (released in 2020) uses the same architecture as GPT-2 (released in 2019), expanded to have ~100x more parameters. It's not hard to see that compounding this process will rapidly hit diminishing returns quickly in terms of time, power consumption, cost of hardware, etc. Honestly, if Google, Amazon and Microsoft didn't see increased computational cost as a financial benefit for their cloud services, people might be willing to admit that GPT-3 is a diminishing return itself: for 100x parameters, is GPT-3 over 100x better than GPT-2?
It seems that the big quantum leaps in AI come from new architectures applied in just the right way. CNNs and now transformers (based on multi-head attention) are the ways we've found to scale this thing, but those seem to come around every 25 years or so. Even the classes of problems they solve seem to change discretely and then approach some asymptote.
Copilot will probably improve, but I doubt we will see much compounding. My best guess is that Copilot will steadily improve "arithmetically" as users react to its suggestions, or even that it will just change sporadically in response to this.
My main takeaway from that paper is that a 2x increase in training cost improves performance by 5% (100x by 42%). I only skimmed the paper though.
To me this says that model scaling will not get us very much farther: we can probably do one more 100x but not two.
I talked to someone working on model scaling and they see the same numbers and draw a very different conclusion: my interpretation of their argument is that they view scaling money as easy versus finding new fundamental technical advances.
This paper only discussed transformers. Given the current pace of research, it's not a given that transformers won't be replaced by something else that has better scaling laws.
And indeed, this paper tells you the amount of compute and data you need for a transformer. But Moore's law isn't doing great lately. For the purposes of research,you may be able to train trillion-parameter models - but you will likely not run such a model on your phone.
In order to merit the large cost of both training and running predictions (which don't even fit the entire model in a single GPU for larger models) - models will need to become more parameter efficient than the vanilla transformer. Otherwise it's just too expensive.
> To me this says that model scaling will not get us very much farther: we can probably do one more 100x but not two.
I totally agree with this assessment, and note the absence of a GPT-4 release.
One way to improve upon GPT-3 is almost certainly going to be adding discriminators into the mix:
"GeDi: A Powerful New Method for Controlling Language Models"
https://blog.salesforceairesearch.com/gedi/
And it may be that rather than it being a CNN discriminator such as in that case, it will be a second transformer:
"[2102.07074] TransGAN: Two Pure Transformers Can Make One Strong GAN, and That Can Scale Up"
https://arxiv.org/abs/2102.07074
My point is we may not need to reinvent some new model separate from transformers/GANs and the work so far as long as novel ways of intersecting trained models produces results that can in turn feed intersect with other models.
NVIDIA in particular has had some interesting focus on using AI to generate training data for AI.
"NVIDIA Omniverse Replicator Generates Synthetic Training Data for Robots | NVIDIA Technical Blog"
https://developer.nvidia.com/blog/generating-synthetic-datas...
I think it's very unlikely we need to wait 25 years for a new paradigm.
In fact, I'm willing to wager a bet that whatever new paradigm will arrive will itself be the output of AI simulating different models.
But more anecdotally, the first applied neural network paper in 1989 by LeCunn has pretty much the same format as the GPT paper: a large neural network trained on a large dataset (all relative to the era). https://karpathy.github.io/2022/03/14/lecun1989/
It really just seems that there are a certain number of flops you need before certain capabilities can emerge.
One use case that I reach for relatively often is writing a comment to describe the one line of code I want when I can’t remember the specifics of the API I’m working with. E.g. # reset numpy random seed, and copilot pretty reliably gives me what I want.
I'm not skeptical of the idea at all, however. I see it as the rough analogy of mathematicians using proof assistants. This is a coding assistant.
It's generally true that the vast, vast majority of code one writes is boilerplate or otherwise rote. The core logic and control flow that defines the meaningful bits of any complex program are nested in the middle of tons of error handling, logging, data munging, incidental book-keeping, and other very prosaic tasks.
It will not just be handy to have a friendly intelligence assist in these tasks. It doesn't require much "intelligence" (of the sort required for figuring out what the program should do in the first place and how to architect it).
Once the technology gets firmed up a bit, the productivity multiplier will be palpable and impossible to ignore. Programmers using these tools will complete tasks faster than ones that don't, and the market will shift.
Further extensions of this technology lead to automatic generation of unit tests, automatic refactoring, etc. These all exist as very specialized tools that are explicitly coded, but it's clear to see that adding a bit of ML-driven intuition to them would greatly increase their scope and effectiveness.
Take English strings.en.json, copy to strings.th.json, open side by side, delete English text and watch copilot fill in the thai translation
(Haven't used Copilot yet because it didn't have Emacs support.)
What it does offer for me, in practice, is basically "clever autocomplete". Say I wrote a little bit of code:
Then copilot will suggest that the next line is: That's reasonably smart, and it adapts pretty well to whatever code you are currently writing. It makes writing those little bits of slightly repetitive code less annoying, and I like that – it feels like a useful productivity boost in the same way as regular "dumb" autocomplete is.However I'd say too much of the initial coverage was along the lines of "you can just write the function name and it will fill in the body!". That will work if you want to write `isEven` or `capitalise` or something, which again is quite nice. But I have found that basically any other prompts product a function that either does something completely wrong, or introduces a subtle and non-obvious bug. And this actually feels kind of dangerous – I'd absolutely caught myself a few times from just going "it'll probably be fine" and accepting whatever Copilot has spewed out.
I'll remain sceptical of it as a system that sucks up and regurgitates vast lakes of open-source code and sends my every keystroke to Microsoft, among other things. But it's definitely something I'd pay for a local version of.