nCompass Technologies launches: low-latency deployment of AI models made easy

nCompass Technologies recently launched!

‍
<sup>"Simplified hosting and acceleration of open-source and custom LLMs"

nCompass is an API that requires only one-line-of-code to integrate low latency versions of open-source/custom models into your AI pipeline.</sup>

Founded by Aditya Rajagopal and Diederik Vink
‍

TL;DR

If unpredictable response times and rate limits of OpenAI are causing your tool’s user experience to suffer, nCompass allows you to effortlessly tap into the world of open-source AI models while ensuring that the served models meet your target budget and performance requirements.
‍

The Problem

LLM-based products that use closed-source model providers like OpenAI suffer from slow response times and rate limits.

Open-source models are a great alternative, but hosting a model yourself is a lot of extra work and maintenance which distracts you from your core business.
‍

nCompass' Solution

nCompass provides an API that allows you to integrate accelerated versions of any open-source or custom model of your choice into your AI pipeline. They support OpenAI style chat templates, work with all web frameworks, and have a time-based pricing model that results in a predictable compute cost for users.

How it works

They serve models to users with a simple 3-step process:

1. Select your desired open-source / custom model
2. Provide your performance requirements
3. Set a budget you are not willing to exceed

They set up the deployment that meets these requirements and provide you with a single API Key that you can then use to integrate the model with a single line of code.

The platform supports any model currently hosted on Hugging Face, with some highlights being:

• Mistral-7B : 160ms Time-To-First-Token @ 86 tok/s
• Mixtral-8x7B : 300ms Time-To-First-Token @ 64 tok/s

Demo ⤵️

https://www.youtube.com/watch?v=sdHVji8QGOg

_{Check out their GitHub repository for code examples}

‍

Learn More

_{🌐 Visit ncompass.tech to learn more
‍
🤝 Know anyone that requires accelerated and/or hosted versions of open-source models? Make the intro!
‍
🗓️  Book a demo
‍‍
👥 Follow nCompass Technologies on LinkedIn & X}