alanzhuly

Hi! I am from Nexa AI. We just improved Omnivision-968M based on your feedback! Here is a preview in our Hugging Face Space: https://huggingface.co/spaces/NexaAIDev/omnivlm-dpo-demo

The updated GGUF and safetensors will be released after final alignment tweaks. Please feel free to let us know if there's any other feedback!

If we can have this running locally on mobile phone that would be pretty cool. Imagine receiving a work document (for example, product requirement documents), and then this turning it into a podcast to play for me while I am driving. I think my productivity will be through the roof and I don't need to worry about compliance issues.

Hi Everyone!

We built an open-sourced tool to benchmark GGUF models with a single line of code. GitHub Link: https://github.com/NexaAI/nexa-sdk/tree/main/nexa/eval

Motivations:

GGUF quantization is crucial for running models locally on devices, but quantizations can dramatically affect model's performance. It's essential to test models post-quantization (how benchmark comes in clutch). But we noticed a couple of challenges:

1. No easy, fast way to benchmark quantized GGUF models locally or on self-hosted servers.

2. GGUF quantization evaluation results in the existing benchmarks are inconsistent, showing lower scores than the official results from model developers.

Our Solution - We built a tool that:

1. Benchmarks GGUF models with one line of code.

2. Supports multiprocessing and 8 evaluation tasks.

3. In our testing, it's the fastest benchmark for GGUF models available.

Example:

Type below in terminal to benchmark Llama3.2-1B-Instruct Q4_K_M quant on the "ifeval" dataset for general language understanding. It took 80 minutes on a 4090 with 4 workers for multiprocessing.

nexa eval Llama3.2-1B-Instruct:q4_K_M --tasks ifeval --num_workers 4

We started with text models and plan to expand to more on-device models and modalities. Your feedback is welcome! If you find this useful, feel free to let us know on GitHub: https://github.com/NexaAI/nexa-sdk/tree/main/nexa/eval

Thanks for reporting. We are investigating this issue. Could you help submit an issue to our GitHub and provide a screenshot of the terminal (with pip show nexaai)? This could help us reproduce this issue faster. Much appreciated!

Llama3.2 3B feels a lot better than other models with same size (e.g. Gemma2, Phi3.5-mini models).

For anyone looking for a simple way to test Llama3.2 3B locally with UI, Install nexa-sdk(https://github.com/NexaAI/nexa-sdk) and type in terminal:

nexa run llama3.2 --streamlit

Disclaimer: I am from Nexa AI and nexa-sdk is an open-sourced. We'd love your feedback.

For anyone looking for a simple alternative for running local models beyond just text, Nexa AI has built an SDK that supports text, audio (STT, TTS), image generation (e.g., Stable Diffusion), and multimodal models! It also has a model hub to help you easily find local models optimized for your device.

Nexa AI local model hub: https://nexaai.com/ Toolkit: https://github.com/NexaAI/nexa-sdk

It also comes with a built-in local UI to get started with local models easily and OpenAI-compatible API (with JSON schema for function calling and streaming) for starting local development easily.

You can run the Nexa SDK on any device with a Python environment—and GPU acceleration is supported!

Local LLMs, and especially multimodal local models are the future. It is the only way to make AI accessible (cost-efficient) and safe.

I like the latest qwen2.5 (https://nexaai.com/Qwen/Qwen2.5-0.5B-Instruct/gguf-q4_0/read...). It was just released last week. It is one of the best small langauge models right now according to benchmarks. And it is small and fast!

Our models work on both mobile apps and web apps. You can use our model's API, which is flexible and can be integrated with any code.

Currently, to modify the defined functions in the model, you can provide us with a description of your desired functions, and we can customize (fine-tune) a model for you.

However, if your use case is similar to the current functions of our model, you might consider using nested functions to leverage our existing capabilities.

Feel free to reach out for further assistance or more specific customization needs.

Our models can handle diverse user inputs well and have high accuracy in our benchmark results. Feel free to check it out here: https://huggingface.co/NexaAIDev/Octopus-v2/blob/main/androi...

It can also handle high demand thanks to its lightweight architecture. During our test, our API has 100x more rate limiting than GPT-4o API

Our models offer developers faster, cheaper alternative to GPT-4o for implementing function-calling AI agent workflows. Developers can use our model APIs and implement each model's functions to integrate AI agent functionalities into their apps.