hypoxia87

My advice as a PM people leader: build stuff.

Historically, as a PM, you couldn't do anything without an engineering counterpart.

Now, with GenAI tools, new PMs can easily create and deploy their own apps. In doing so, you'll learn the core PM job skills (for example, thinking about user value, defining requirements, creating designs) and get introduced to the core technical work (for example, writing and testing code, deployment).

The latter is very important for PMs to understand and it's now much more accessible than in the past. You can use Cursor and various Figma to Code platforms for code, and ChatGPT and Claude to help with deployment.

When I'm looking to hire PMs, those who have actually built stuff have a leg up over other candidates.

Hope this helps!

Jumping in based on our experience in case it's helpful:

- Evals are very useful and a core part of the best practice for creating LLM apps

- There are already excellent solutions for model and prompt eval (etc.) including Parea from YC and many others

We've actually found the opposite -- Every client project has been based on GPT4 or Gemini, with one exception for a highly sensitive use case based on Llama3.1.

The main reason is that the APIs represent an excellent cost / performance / complexity tradeoff.

Every project has relied primarily on the big models because the small models just aren't as capable in a business context.

We have found that Gpt4o is very fast, when that's necessary (often it's not), and it's also very cheap (gpt4o batch is ~96% cheaper than the original GPT4). And where cost is a concern and reasoning doesn't need to be as good as possible, gpt4o mini has been excellent too.

+1

There are already many prompt/LLM routers available.

We've never found value in them, for similar reasons as mentioned above.

Plus every company's files in OneDrive and SharePoint.

Claude 3 Opus

I assume it's RAG over your chats with heuristically driven questions that are then fed into your system and user prompts.

Notably, this comes after the release of

1) "Custom instructions" in the settings to manually modify your system prompts.

2) Retrieval as part of Assistants API (RAG)

3) The "GPT" builder allowing you to define how the system responds via chats (I assume this uses chats to define system prompts)

Feels like a natural extension of these to me. If someone extracted the new system prompt, I would expect to see a section on this similar to custom instructions.

I think it comes down to prioritizing speed of execution in the race for AGI, especially in the context of two constrained resources:

1. Compute (H100s are almost impossible to get ahold of, especially in the quantities they'd need to train a competitive model)

2. Data (OpenAI has a very rich set of user data from ChatGPT that is necessary for creating a model like GPT-4; a new venture would lack this data -- the cold start problem -- but Microsoft may* have similarly usable data from non-enterprise Bing Chat)

* Someone please feel free to prove me wrong on this - I'm not aware of a MS policy saying they won't use Bing Chat prompts for training, but it could very well exist.

(step 4 is something only some types of agents do)

For another definition that incorporates these elements (explicitly or implicitly), see Llamaindex's high level concepts page [1]:

"Agents: An agent is an automated decision maker powered by an LLM that interacts with the world via a set of tools. Agents can take an arbitrary number of steps to complete a given task, dynamically deciding on the best course of action rather than following pre-determined steps. This gives it additional flexibility to tackle more complex tasks."

[1] https://docs.llamaindex.ai/en/stable/getting_started/concept...

I spend most of my time thinking about LLM-based agents. For me, they're systems that:

1. Have a goal (hard-coded, taken as input, etc)

2. Create a plan (of steps or tasks) to achieve the goal

3. Execute (the steps or tasks in) the plan to achieve the goal (often including tool use)

4. Sometimes they check for success against the goal and/or re-plan based on current status

Love your blog BTW

This is an excellent question! I'd also like to know.