Will any of the profits be shared with original authors whose work powers the model?

Which authors? Machine Learning research authors?

I mean the authors who wrote the works which are resold by Microsoft for its own profit without any opt-in or even opt-out, much less compensation.

It doesn't seem to be answered in the article, but if it was and you read it should you have to pay them a fee for the knowledge if it was published openly on the net?

You are confusing two distinct cases.

In the first case, you found/bought a book and read it. No one can or should make you pay for it, unless you stole the book.

In the second case, you found/bought a book then reprinted it infinitely and sold it for profit, ethically you should pay the author and legally you should be in violation of the law.

Even if you made a machine that ingests and recombines books automatically, and you keep that machine locked up and charge people for its use, it is the same scenario: the machine would be absolutely useless without the original books, those books cost people effort and money to produce, yet you pay those people nothing while the machine is basically an infinite money maker for you.

I hope the analogy makes sense.

No.

Now that you have read my answer, you owe me $0.01 because your brain might use this information in the future.

So are you in favour of granting human rights to a machine? If not, your analogy makes zero sense because we are talking about a copyright laundering tool creating derivative works, not a thinking human that presumably we both are.

People's outrage to your valid question is ridiculous. MS and OpenAI will make billions because they scrapped lots and lots of data, but aurhors od those data can't get anything because openai simps will shout. I see this is very american thing to do. Allow corporations to do everything they want, because limitations or just justice and rewarding real authors of data those corporations benefit from is literally communism

Because at the other end of this equation you would have companies like disney holding you at gunpoint for money if you ever spoke about mice.

Oh yeah, because there can be only extremities. Sure.

Ok profits will be shared with all internet users. Send an invoice for $0.0000000000001 for your contributions to the internet corpus.

Made my first million this year myself actually and I probably have many people to credit that I forgot to credit. I can start with Pythagoras, Galileo [insert everyone between], Kernighan, Ritchie. Also the guy who invented pencilin. I'm honestly not sure how these angles arise. Knowledge wants to be free. We are here today because of this fact.

When it comes to spam culture sure. But will we ever be there? "AI art" isn't impressive and will never be. It is impressive in the academic sense. Nothing more.

How’d you make a million dollars in a year dude?

The motivation to produce original knowledge is that it is considered your intellectual property. By suggesting to abolish the notion of intellectual property, are you arguing for some form of communism?

Imagine Google scraping the Internet and not directing you to search results. We’d be with pitchforks the next day. But when OpenAI does it, that’s somehow okay…

Isaac Newton has sadly passed.

Yes, dead people are fine not being paid, so what’s your plan then I fear to ask?

The model is powered by math.

I'm really curious to see if expanding the context length this much will allow GPT to do typical software development tasks on a big codebase. If it can take in a github issue and produce decent code solving a complex issue across many files... will certainly be an interesting time.

My guess is that anything requiring nontrivial business/technical domain knowledge will be fairly safe. Also anything with a visual (or auditory) correlate, like UI work.

Why would you think this? As long as the technical domain knowledge is at least partially published, I don't see them stopping becoming better.

UI stuff just has an input problem. But it is not that hard to think that ChatGPT could place widgets once it can consume images and has a way to move a mouse.

> As long as the technical domain knowledge is at least partially published

Most internal technical and business domain logic of companies isn’t published, though. Every time I asked ChatGPT about topics I had actually worked on over the past decade or two, or that I’m currently working on, it basically drew a blank, because it’s just not the category of topics that are discussed in detail (if at all) on the internet. At best it produced some vague generalisms.

> once it can consume images and has a way to move a mouse.

That’s quite far from ChatGPTs current capabilities, which is strongly tied to processing a linear sequence of tokens. We will certainly improve in that direction as we start combining it with image-processing AIs, but that will take a while.

I wonder if there will be a race to buy defunct companies for access to their now valuable junky tech-debt ridden hairball code, so they can train on it and benchmark on fixing bugs and stuff. With full source control history they could also find bug resolution diffs.

That source code isn’t worth much without the underlying domain knowledge, large parts of which only exist in the employees’ heads, more often than not. Maybe if the code is really, really well documented. ;)

The weights file could be encrypted and require a password before becoming usable.

Check out the announcement. GPT-4 accepts mixed-mode inputs of text and images.

Mouse cursor instructions aren’t a massive leap from the current capabilities, given the rate of progress and recent developments around LLM tool use and the like.

I think what you say goes for most jobs. Why would GPT know much detail about being a machinist or luthier?

Eventually job and role specific information will be fed into these models. I imagine corporations will have GPTs training on all internal communications, technical documentation, and code bases. Theoretically, this should result in a big increase in productivity.

>UI stuff just has an input problem. But it is not that hard to think that ChatGPT could place widgets once it can consume images and has a way to move a mouse.

I remember one of the OpenAI guys on Lex Fridman podcast talking about how one of the early things they tried and failed at was training a model that could use websites, and he alluded to maybe giving it another go once the tech had matured a bit.

I think with GPT-4 being multi-modal, it's potentially a very close to being able to do this with the right architecture wrapped around it. I can imaging an agent using LangChain and feed it a series of screenshots and maybe it feeds you back a series of co-ordinates for where the mouse should go and what action to take (i.e. click). Alternatively, updating the model itself to be able to produce those outputs directly somehow.

Either way, I think that's going to happen.

Yeah, the example given in the OpenAI GPT4 twitter video is someone asking it to write a python script to analyze their monthly finances and it simply just importing dataframes, importing "finances.csv", running a columnar sum for all finances and then displaying the sum and the dataframe. I'm sure it's capable of some deeper software development but it almost always makes radical assumptions and is rarely ever self sufficient (you don't need to look it over and don't need to change the architecture of the code it produced).

How would you have it suggest solutions for multiple files? Has anyone gotten GPT-X to output a valid git patch or something?

You just kind of concatenate the entire codebase into one file, tell the model to do something and output the modified codebase into another file, diff the two and produce a patch automatically.

That codebase=>token stream=>codebase step feels like it could be lossy depending on how you encode things like file paths when concatenating everything, would be interesting to see in practice though!

or you might even be able to feed it individual files with their filenames, then ask it what modifications it would make as a diff for each of the files

Yep, I know that’s been possible since at least GPT-3 davinci

How would you go about it?

I think there's ways but you might have to use pinecone db or something like lang chain to essentially give it a long term memory...

or another option is having one instance or chat order code page and one that basically just has an API index and knows which chat has the related things.

I'd love to get to a point where I can go: Add a cast button to this open source android video app.

I see some FOSS-boosting silver linings in all of this.

>If it can take in a github issue and produce decent code solving a complex issue across many files... will certainly be an interesting time.

Oh snap. I didn't even think about that!

That gives me a fun idea!

I've got a repo that I built and setup CI/CD and setup renovate to automatically upgrade dependencies and merge them when all the tests pass, but of course sometimes there are breaking changes. I don't actively work on this thing and hence it's just got issues sitting there when upgrades fail. It's the perfect testing ground to see if I can leverage it to submit PRs to perform the fixes required for the upgrade to succeed! That'll be hectic if it works.

It can't even do simple sysadmin tasks like fixing a broken installation, or fixing simple configure/make/make install issues.

Also note that image input isn't available to the public yet.

>Image inputs are still a research preview and not publicly available.

$0.12 per 1k completion tokens is high enough that it makes it prohibitively expensive to use the 32k context model. Especially in a chatbot use case with cumulative prompting, which is the best use case for such a large context vs. the default cheaper 8k window.

In contrast, GPT-3.5 text-davinci-003 was $0.02/1k tokens, and let's not get into the ChatGPT API.

> Especially in a chatbot use case with cumulative prompting, which is the best use case for such a large context vs. the default cheaper 8k window.

Depends on what is up with the images and how they translate into tokens. I really have no idea, but could be that 32k tokens (lots of text) translates to only a few images for few-shot prompting.

The paper seems not to mention image tokenization, but I guess it should be possible to infer something about token rate when actually using the API and looking at how one is charged.

Currently, CLIP's largest size is at patch-14 for 336x336 images, which translates to 577 ViT tokens [(336/14)^2+1]. It might end up being token-efficient depending on how it's implemented. (the paper doesn't elaborate)

I disagree that out of all possible use cases for a large context model that a chatbot is really the "best use case".

I would imagine most usecases for the 32k model have much longer prompts than completions, so the $0.06 per prompt token will be the real problem. I can't think of a usecase yet, but that might be because I haven't got a sense of how smart it is.

can't you combine instances of 4k tokens in 3.5 to fake it? having one gpt context per code file, for instance and maybe some sort of index?

I'm not super versed on lang chain but that might be kinda what that solves...

LangChain/context prompting can theoetically allow compression of longer conversation, which will likely be the best business strategy.

> $0.12 per 1k completion tokens is high enough that it makes it prohibitively expensive to use the 32k context model.

this is a lot. I bet there's a quite a bit of profit in there

Gotta pay back M$

> I bet there's a quite a bit of profit in there

Is this profit-seeking pricing or pricing that is meant to induce folks self-selecting out?

Genuine question — I don’t know enough about this area of pricing to have any idea.

And how it works? you can build a context and then ask something in a prompt using it?

Context is how many tokens it can be fed to produce an output. So now you can feed it up to 32k words (tokens).

Token is 0.75 words on average per OpenAI; 32k in tokens would be roughly 24k in words.

https://help.openai.com/en/articles/4936856-what-are-tokens-...

Poe.com(by Quora) have gpt4 now, give it a shot

Seems limited to only one prompt per day. I just tried it.

You can pay $17/month (Apple subscription only for now), for ~unlimited queries to GPT-4 (and Claude+)

I missed that, thanks

> Image inputs are still a research preview and not publicly available.

Will input-images also be tokenized? Multi-modal input is an area of research, but an image could be converted into a text description (?) before being inserted into the input stream.

My understanding is thta the image embedding is included, rather than converting to text.

My understanding is that image embeddings are a rather abstract representation of the image. What about if the image itself contains text, such as street signs etc?

One way to get around context length is to perform embedding and retrieval of your entire corpus. Langchain (https://langchain.readthedocs.io/en/latest/) and Milvus (https://milvus.io) is one of the stacks you can use.

Can you elaborate on how this works?

You run the corpus through the model piecemeal, recording the model's interpretation for each chunk as a vector of floating point numbers. Then when performing a completions request you first query the vectors and include the closest matches as context.

I still doesn't understand how can content length not be limited if you have a conversation composed of several messages each with length nearing the limit of what is allowed. Does it not have to in some way incorporate all the input albeit in one input or multiple inputs?