News for hackers. Stuff that matters.

calum-bird

Re:Let's talk about AI and end-to-end encryption on Saturday January 18, 2025 @11:38AM (#42749496)
Do you have any resources you can share on managing the enclave’s public key? We found documentation to be sparse.
Re:Claude 3.5 Sonnet on Friday June 21, 2024 @11:01AM (#40750365)
So typically these providers only offer ZDR to "managed" customers, after a lengthy application process. For example, on Azure, "managed" means companies with >$1m, possibly more now, in annual spend. They don't want to waste their time going through this long application process with smaller companies, so we take some of that weight off their shoulders. They get the same revenue at the end of the day, so in many ways it groups smaller companies' LLM spend and sends it straight to their bottom line, and they still get to claim their rolling out AI "responsibly".

Once one provider is cracked, the others fall as well, as these AI companies are all competing viciously for customers. Et voila, ZDR across multiple providers for the small(er) companies out there :)

Re:Claude 3.5 Sonnet on Thursday June 20, 2024 @03:45PM (#40742528)
We (disclosure: founder) do something similar at Trelent[1] but with an emphasis on security. Paid accounts can use OpenAI & Anthropic models, free ones just OpenAI. We have 3.5 sonnet live already. If you want to try it out lmk! Also totally respect building your own open-source :)

[1]: https://trelent.com

For efficiency, 132B.

That way, at inference-time you get the speed of 36B params because you are only "using" 36B params at a time, but the next token might (and frequently does) need a different set of experts than the one before it. If that new set of experts is already loaded (ie you preloaded them into GPU VRAM with the full 132B params), there's no overhead, and you just keep running at 36B speed irrespective of the loaded experts.

You could theoretically load in 36B at a time, but you would be severely bottlenecked by having to reload those 36B params, potentially for every new token! Even on top of the line consumer GPUs that would slow you down to ~seconds per token instead of tokens per second :)

Show HN: Trelent is your team's fully-encrypted AI assistant

https://www.trelent.com

ChatGPT Enterprise and others have content-logging enabled ("abuse monitoring") and don't let you control your data. Your intellectual property is thus at risk. Many of you can't use ChatGPT or similar tools at work because of corporate policy on this basis.We solved that by negotiating with AI providers to disable content logging, then we let you encrypt the persisted chat data in our database...

( Read More... | no comments | on HN )
Re:Ask HN: Who is hiring? (August 2023) on Tuesday August 01, 2023 @11:03AM (#36956892)
Trelent | Founding Engineer | REMOTE (US/Canada preferable) | $100k+

I am the founder of Trelent. We’re building a useful agent for RPA, with a distinctly structured & hierarchical approach to working with LLMs. We have pilot customers in several industry verticals, have thousands of users, and are venture backed by multiple funds. Our goal is to ultimately enable the next billion software developers.

We’re very early (team of two) and are looking for a founding engineer who can wear several hats when necessary, but has a knack for UX. The focus right now is largely product-side, mainly on the front-end. Our stack is NextJS, Tailwind, and Node. Bonus points if you've worked with LLMs before - we strongly believe that they will change UX (but not through a chatbox alone).

Preference to North America, but fully remote and can hire anywhere for the right candidate. More details & application at https://jobs.trelent.com

Re:GitHub Copilot is generally available on Wednesday June 22, 2022 @05:08PM (#31841888)
Copilot does inline completion right now - they implemented it into copilot as a pilot program[0] before OAI went live with that new model IIRC.

[0]: https://openai.com/blog/gpt-3-edit-insert/

Re:Show HN: VS Code Extension to Document Code on Wednesday March 16, 2022 @03:53PM (#30703224)
Haha indeed, NLP is fun!
Re:Show HN: VS Code Extension to Document Code on Monday March 14, 2022 @01:36PM (#30675373)
Our website has a little more @ www.trelent.net. Working on a better way to showcase more examples. Will be inherently bias as we're selecting them (even if we try to take a fair sample of good/bad results), so best would be for you to try it yourself haha
Re:Show HN: VS Code Extension to Document Code on Monday March 14, 2022 @01:34PM (#30675348)
Totally get that. Working on locally-hosted version for enterprise use-case with this in mind. We are either paid in dollars or data, and I'd rather be transparent about that then hide it deep in a Privacy Policy.
Re:Show HN: VS Code Extension to Document Code on Monday March 14, 2022 @01:30PM (#30675309)
Haha I wish! That's just loading the weights for inference :)
Re:Show HN: VS Code Extension to Document Code on Sunday March 13, 2022 @09:02PM (#30667419)
Please note: this does make use of a remote server, so use with caution. Working on locally-hosted version for anyone with ~48GB of VRAM (and change) to spare.