Together with the CEO I've also decided that we do not do this with our own code, it stays on machines we control until someone pays for some artifact we'd like to license.
I'm well aware that many other organisations take a different position and push out basically everything they work on to SaaS LLM:s, in my experience defending it with something about so called productivity and something about some contract clause about the SaaS pinky promising to not straight up take the code. But nothing stops them from running hidden queries against it with their in-house models parallel with providing their main service, and sift out a lot of trade secrets and other goodies from it.
It's also likely these SaaS corporations can benchmark and otherwise profile individual developers, information that would be very valuable to e.g. recruiting agencies.
If you can tell, how is that Copilot performance measured?
I don't know how they measure it, but I assume they have some management dashboard they can view. The requirement is exposed to us in multiple performance objectives with phrases about x% of team making active use of gen ai, use gen ai to increase velocity y%, automate z% of tasks with gen ai, and so on.
To be honest, I didn’t realize that model selection would be such an important point for users. I believed that choosing a high-quality model with strong reasoning capabilities was part of the service’s value proposition.
But lately, more users — including yourself — have been asking for support for other models like Claude Sonnet or LLaMA.
I’m now seriously considering adding an adapter feature. Thank you for your feedback — I really appreciate it.