LLM benchmarking leaderboard that evaluates how well AI models understand Appwrite services. Compare model performance with and without skill file context across 70 questions spanning 7 Appwrite product categories.

Live at arena.appwrite.network

The benchmark tests leading AI models on their knowledge of Appwrite through two modes:

• With skills — Models receive comprehensive Appwrite documentation as context
• Without skills — Models answer based solely on their training data

Questions are split into 57 multiple-choice (auto-scored) and 13 free-form (AI-judged by Claude Sonnet 4.6) across these categories:

• Claude Opus 4.7 — Anthropic
• GPT 5.5 — OpenAI
• Gemini 3.1 Pro (Preview) and Gemini 3.1 Flash Lite (Preview) — Google
• Grok 4.3 — xAI
• DeepSeek V4 Flash — DeepSeek
• Qwen 3.6 Plus — Alibaba
• GLM 5.1 — Zhipu
• MiniMax M2.7 — MiniMax
• Mistral Large 3 2512 — Mistral
• Kimi K2.6 — MoonshotAI

All models are accessed via OpenRouter with temperature set to 0 for deterministic results.

Frontend: React, TanStack Start, Tailwind CSS, Vite, TypeScript

Benchmark: Bun, OpenRouter

• Node.js 18+
• Bun (for benchmark scripts and pre-build step)

npm install
npm run dev

The app runs at http://localhost:3000.

npm run build
npm run preview

npm run lint
npm run format
npm run check

npm run test

The benchmark suite lives in the benchmark/ directory and requires an OpenRouter API key.

cd benchmark
cp .env.example .env
# Fill in your API key in .env

# Run both modes
bun run bench:all

# Or run individually
bun run bench:with-skills
bun run bench:without-skills

Keep in mind, the benchmark only fills missing data in result JSON files, to minimize cost. If you intend to re-run the benchmark on existing results, you should delete the contents of the JSON files first.

Results are saved to src/data/results-with-skills.json and src/data/results-without-skills.json, which the frontend reads at build time.

├── src/                    # Frontend application
│   ├── components/         # React UI components
│   ├── routes/             # File-based routes (TanStack Router)
│   ├── data/               # Static benchmark result JSON files
│   └── lib/                # Types, utilities, site config
├── benchmark/              # Benchmark suite
│   ├── src/
│   │   ├── questions/      # 70 questions across 7 categories
│   │   ├── skills/         # Appwrite documentation for context mode
│   │   ├── runner.ts       # Test execution logic
│   │   ├── judge.ts        # AI judge for free-form answers
│   │   └── config.ts       # Model definitions and settings
│   └── package.json
├── scripts/                # Build-time scripts (GitHub stars fetcher)
└── public/                 # Static assets

MIT — see LICENSE.