Show HN: NCompass Technologies – yet another AI Inference API, but hear us out

Re:Show HN: NCompass Technologies – yet another AI Inference... by JackC on Monday December 16, 2024 @05:11PM (#42436024)

Random idea -- I think it would be cool for hosts that advertise efficiency to have a dashboard that shows total tokens per watt-hour (or whatever usage:energy metric) graphed over time for each model they host, taking into account as much of their infra as possible.

This would:

- let you boast about your cool proprietary optimizations

- naturally get better over time just from applying public algorithmic improvements

- show up hosts that refuse to do the same

- give you a good incentive to keep on top of your own efficiency and competitiveness over time

- be a good response to users who vaguely know that AI takes "a lot" of energy -- it's actually gotten a lot better, but how much better?

Happy to chat if it would help to have a neutral academic voice involved.

[ Parent | on HN ]

Re:Show HN: NCompass Technologies – yet another AI Inference... by DiederikVink on Monday December 16, 2024 @05:48PM (#42436359)

Thanks for the suggestions! This is definitely something we'll be looking at.

We're currently working on providing a more extensive interface to show users a variety of performance metrics of the models they're running. Having efficiency metrics would be a great addition.

I think additionally an important facet of these tests would be providing clarity on the details of the tests to make them reproducible. I find that sometimes reported stats don't quite translate to real-world experiences. It can feel like results are presented using the workloads that look best on a system, so a standardized/reproducible approach would be best.

We're always keen to chat to as many users/experts/academics/enthusiasts as possible. Please feel free to reach me at [email protected] and we can set up a time to meet!

[ Parent | on HN ]