calum-bird
Show HN: Zero data retention LLMs as a service
Once one provider is cracked, the others fall as well, as these AI companies are all competing viciously for customers. Et voila, ZDR across multiple providers for the small(er) companies out there :)
[1]: https://trelent.com
That way, at inference-time you get the speed of 36B params because you are only "using" 36B params at a time, but the next token might (and frequently does) need a different set of experts than the one before it. If that new set of experts is already loaded (ie you preloaded them into GPU VRAM with the full 132B params), there's no overhead, and you just keep running at 36B speed irrespective of the loaded experts.
You could theoretically load in 36B at a time, but you would be severely bottlenecked by having to reload those 36B params, potentially for every new token! Even on top of the line consumer GPUs that would slow you down to ~seconds per token instead of tokens per second :)
BYOK Encryption Isn't Enough
Show HN: Trelent is your team's fully-encrypted AI assistant
ChatGPT Enterprise and others have content-logging enabled ("abuse monitoring") and don't let you control your data. Your intellectual property is thus at risk. Many of you can't use ChatGPT or similar tools at work because of corporate policy on this basis.We solved that by negotiating with AI providers to disable content logging, then we let you encrypt the persisted chat data in our database...
I am the founder of Trelent. We’re building a useful agent for RPA, with a distinctly structured & hierarchical approach to working with LLMs. We have pilot customers in several industry verticals, have thousands of users, and are venture backed by multiple funds. Our goal is to ultimately enable the next billion software developers.
We’re very early (team of two) and are looking for a founding engineer who can wear several hats when necessary, but has a knack for UX. The focus right now is largely product-side, mainly on the front-end. Our stack is NextJS, Tailwind, and Node. Bonus points if you've worked with LLMs before - we strongly believe that they will change UX (but not through a chatbox alone).
Preference to North America, but fully remote and can hire anywhere for the right candidate. More details & application at https://jobs.trelent.com