​

Introduction to custom benchmarks

1. Send example prompts to a state-of-the-art model to establish a baseline with a Jupyter notebook.
2. Use Narev Cloud to find a model that performs like the baseline with lower cost and better latency.
3. Deploy the optimal model to production.

​

Why not use an LLM leaderboard instead?

• Academic benchmarks are a poor approximation of your app. The leaderboards show scores that LLMs get on standardized tests. The promise is that strong benchmark scores transfer to production, but this often fails. A 3-year-old model can match a current state-of-the-art model despite drastically different benchmark results.
• Academic benchmarks aren’t repeatable. Benchmark results vary depending on who runs them. For example, GPT-4’s score on the MMLU-Pro benchmark ranges from 72.6% to 85.7% depending on the leaderboard.
• People game academic benchmarks. For example, researchers document manipulation of Chatbot Arena in a major study by Singh et al. (2025). Data contamination is rampant (models that use benchmark questions in their training process). Read more about it in our docs on benchmarking.

​

Before you begin

• A Narev account (sign up here) and a Narev API key (get one here)
• Python 3.8+ installed locally or access to Google Colab/Kaggle
• 50-100 example prompts representative of your production use case
• Basic familiarity with Python and Jupyter notebooks
• ~30 minutes to complete the setup and benchmarking

​

Setting up your benchmark

​

Part 1: Create an A/B test endpoint on Narev platform

https://www.narev.ai/api/applications/<YOUR_TEST_ID>/v1

​

Part 2: Open and prepare the environment

# Install required packages
import pandas as pd
import requests
from datasets import load_dataset
from kaggle_secrets import UserSecretsClient # or from google.colab import userdata

# Secrets
user_secrets = UserSecretsClient()
NAREV_API_KEY = user_secrets.get_secret("narev-api-key") # or userdata.get('narev-api-key')
NAREV_BASE_URL = "https://www.narev.ai/api/applications/cmj6nt7g40002i1qymlqvct4z/v1" # TODO: replace with your URL from Part 1

# Variables
SYSTEM_PROMPT="""Solve the following math problem. Only include the number in your answer.
Give your final as a number.
Do not include any reasoning"""

STATE_OF_ART_MODEL="openrouter:google/gemini-3-pro-preview"

dataset = load_dataset("openai/gsm8k", "main")
df_train = pd.DataFrame(dataset['train'])
df_test = pd.DataFrame(dataset['test'])

# take the first 50 prompts from the test
df_test = df_test.head(50)

print(f"First question:\n\n{df_test.iloc[0]['question']}")

def send_to_llm(question: str) -> dict:
    messages = []
    if SYSTEM_PROMPT:
        messages.append({"role": "system", "content": SYSTEM_PROMPT})

    messages.append({"role": "user", "content": question})

    try:
        response = requests.post(
            url=f"{NAREV_BASE_URL}/chat/completions",
            headers={
                "Authorization": f"Bearer {NAREV_API_KEY}",
                "Content-Type": "application/json"
            },
            json={
                "model": STATE_OF_ART_MODEL,
                "messages": messages
            },
            timeout=60
        )

        response.raise_for_status()
        data = response.json()
        return data
    except Exception as e:
        print(e)
        return {}

​

Part 3: Send the requests to Narev

for question in df_test['question']:
    print(f"Processing question: {question}\n")

    response = send_to_llm(question)

    if response.get('choices'):
        print("Success\n\n")
    else:
        print("Failed to send a request\n\n")

​

Optimization in the Narev platform

​

Part 1: Select test Variants

• Gemini 3 Pro Preview - our first model, tagged as state of the art,
• DeepSeek v3.2 Speciale - flagship model from the DeepSeek team
• GPT-5 Nano - recommended by Narev as a competitor performing in the benchmark similarly to the DeepSeek’s model
• Llama 3 70B Instruct - great open source model that I tried in many experiments in the past
• Ministral 3B - tiny, powerful model, from Narev’s recommendations

​

Part 2: Select the Source of Truth model and the Quality Metric

​

Part 3: Run the experiment

​

Analyzing the results

​

Key insights from the benchmark

1. Quality-Cost tradeoff: DeepSeek v3.2 Speciale delivers 98.1% of baseline quality at only 2.7% of the cost—saving $8,552 per million tokens
2. Latency considerations: While cheaper, DeepSeek is 34% slower than baseline. For latency-critical applications, this tradeoff matters
3. Task alignment matters: Smaller models (Llama 3 70B, Ministral 3B) showed poor performance on math reasoning, demonstrating that model capabilities must match task complexity

• Quality - We’re at the optimal spot (this means that there is no variant with a higher quality). Great source of truth. Truly a state of art model.
• Cost - We have one recommendation. It says that IF we were able to sacrifice 1.9% of quality, we COULD save 97% in token cost by choosing DeepSeek v3.2 Speciale
• Latency - For a model with this level of quality, there is no quicker model than that.

​

Deploying to production

1. In the Narev dashboard, set your chosen variant (e.g., DeepSeek v3.2 Speciale) as the production variant
2. Continue using the same endpoint from Part 1: https://www.narev.ai/api/applications/<YOUR_TEST_ID>/v1
3. When you don’t specify a model in your requests, Narev automatically routes to your production variant

# No model specified - automatically uses your production variant
response = requests.post(
    url=f"{NAREV_BASE_URL}/chat/completions",
    headers={"Authorization": f"Bearer {NAREV_API_KEY}"},
    json={
        "messages": [{"role": "user", "content": "What is 25 * 17?"}]
    }
)

​

Resources

• Narev Platform Documentation
• A/B testing best practices
• Data Source Options
• Understanding evaluation metrics
• GSM8K Benchmark Dataset

​

Next steps

• Try with your own prompts and use case
• Experiment with different system prompts to optimize further
• Explore prompt optimization techniques