What exactly is the point of using two models, both far from state of the art and saying, "LLMs are not useful". I mean, what's the thought process here ?

You'll see this in the workplace too. It's very hard to explain to the not-most-productive people that the most productive people on the team are also usually making the most mistakes. They'll always be happy to just sit back and criticize the mistakes. It's the only way they can be better.

Sorry, my comment wasn't LLMs in general, it was this specific pair of LLMs at this specific time.

I've been trying the same prompt since ChatGPT was released, and I keep revisiting it.

They are getting better at writing code, Bard's addition of sourcing information is very important to me.

I do this so I can see if it will help _me_ in what I'm trying to do, and where the failure modes are. Their failure modes are very sneaky and very dangerous for the kind of programming I do.

I would use them to maybe give me a first pass at some code in a language I have zero familiarity with, and only to solve general problems.

I'm thinking throw-away, one-off code where the operational range is known up front and fully middle of the road. I'm thinking reports.

I might try to get it to write something useful in R, I have no knowledge of that.

I can see them as really handy refactoring tools. They take instruction pretty well. They just need to be stable in the code they produce - they keep reintroducing bugs you've instructed them to fix. Changing to an edit stream against code instead of re-generating the code might be a fix?

That has been a bit of my experience with LLMs in general. If you know what you are doing before hand it's not too much of a boost, in fact it can be a bit frustrating to wrestle with it to do what you want when you can do it faster. I can see where it is amazing tooling though. Especially for a new project where you don't really have much of an idea or need a working demo.

But if you don’t know what you are doing, you have to worry about its hallucinations slipping pass you.

Really good point too, though for the most part I like to think if I audit the code well enough it will work? That said the other day I used it for some string manipulation based on powers of 10, and the code looked correct as per my audit of it. However in reality when I ran the code it was really bad. You might say that I my self would produce buggy code the first time around, that is true... but I also find the LLM equally frustrating because for me (and only me, a sample size of 1 calm down people)? I find I can iterate faster over an idea by hand vs using an LLM for a lot of things.

eh, there is always somebody who inevitably frets on about these hallucinations, but I find that most of the people who are super worried about these aren't actually using the LLMs the way that they're intended.

For example, I needed a GitHub action that would automatically check out build and deploy a node project to a VPS. I also needed it to pipe out the secrets into a local environment variable (.env) and copy that over to the VPS as well.

GPT4 got it in one try. Its this kind of DevOps boring mundane crap that GPT excels at eliminating. Even though I had very little familiarity with the syntax for GH action yaml, I was able to read it and reason out that it was correct.

Could I have sifted through the documentation, or waited through 50 stack overflow pages? Sure, but this was a hell of a lot more convenient.

Try using https://www.phind.com on expert mode. You will probably have better results.

Thanks for the tip, phind is much better.

Still has the "won't fit in an int" problem, but it did use byte & 0x77 to get them into ints.

Amazingly, it got the field correct and in the right order!

It did not reference a source, but that seems to be a Bard feature right now.

phind needed two additional prompts to get to correct code.

It needed:

1) write a function in java to decode a rfc 3588 diameter header

2) 32 bit values will not fit into a java int, please fix the code.

3) 24 bit values will not fit into a java int, please fix the code.

I think I'm going to have to come up with a more complicated problem!

Pretty impressively, this is what it produced:

  public class DiameterHeader {
    private int version;
    private long messageLength;
    private int commandFlags;
    private long commandCode;
    private long applicationId;
    private long hopByHopId;
    private long endToEndId;

    public DiameterHeader(byte[] data) {
        if (data == null || data.length != 20) {
            throw new IllegalArgumentException("Invalid Diameter header data");
        }

        version = data[0] & 0xFF;
        messageLength = ((long) data[1] & 0xFF) << 16 | ((long) data[2] & 0xFF) << 8 | ((long) data[3] & 0xFF);
        commandFlags = data[4] & 0xFF;
        commandCode = ((long) data[5] & 0xFF) << 16 | ((long) data[6] & 0xFF) << 8 | ((long) data[7] & 0xFF);
        applicationId = (((long) data[8] & 0xFF) << 24) | (((long) data[9] & 0xFF) << 16) | (((long) data[10] & 0xFF) << 8) | ((long) data[11] & 0xFF);
        hopByHopId = (((long) data[12] & 0xFF) << 24) | (((long) data[13] & 0xFF) << 16) | (((long) data[14] & 0xFF) << 8) | ((long) data[15] & 0xFF);
        endToEndId = (((long) data[16] & 0xFF) << 24) | (((long) data[17] & 0xFF) << 16) | (((long) data[18] & 0xFF) << 8) | ((long) data[19] & 0xFF);
    }

    // Getters for the fields
  }

If you have expert mode on it should reference sources most of the time.

phind seems to be showing the Google search results as "Sources", rather than the actual code used as training data.

Bard links to a github repo with license information.

I was under the impression that the search results were the training data?

What phind does is pulls code from search results and modifies it to suit your needs.

Thanks for the tip. I put in an example that ChatGPT completely failed at and phind eventually got the right answer.

Oddly it went off on a tangent about how the problem can't be solved in Python, which I never mentioned.

But when I explicitly told it I wanted c++ it gave a decent answer.

It did, however, get some nuances seriously wrong.

> Precision loss for large integers: When encoding to JSON, UInt64 and Int64 are encoded as strings of base-10 digits in ASCII, since JavaScript's numeric type only affords 53 bits of precision. This means that large integers might lose precision when converted to JSON.

https://www.phind.com/search?cache=319c7cf5-65e4-4704-a8f3-0...

Try GPT-4.

Have you tried GPT-4? Here's the response I got: https://shareg.pt/Q0ZcWCX

Is it any better? (Your prompt is related to a completely different domain than the one I work in.)

Nifty, thanks!

The code has the standard "you can't fit 24 bit or 32 bit unsigned values in an int" bug. I get the feeling this is a common bug. Probably insufficient testing in the hobby implementations used as training data.

My reading of "int length = first4Bytes & 0xFFFFFF" is that it will truncate values incorrectly. I'm pretty sure that "-1 & 0xFFFFFFF != 0xFFFFFFF". It goes (-1 & 0xFFFFFFF) is a long, which is then downcast back to int, truncating it?

commandCode is wrong, GPT-4 is using 28 bits instead of 24.

I _think_ the command code flag decoding is wrong, but it's been a while since I've done bit fiddling. The spec says Request is bit 0, but the code has it in 0x80 - bit 8?

That's much improved from ChatGPT and Bard. The use of the ByteBuffer avoids many of the problems. I've seen that in some GitHub Diameter decoders.

The RFC is here: https://www.ietf.org/rfc/rfc3588.txt - Page 31 for the header.

You forgot anyone can just run your example in ChatGPT and post a screenshot showing you're talking out of your ass.

The guy who was trying to trash ChatGPT (gpt4) forgot that anyone can just check his example and post a screenshot I think.