Notes on RTX 3090Ti

TL;DR: use syv-ai/qwen38-27b-rtx3090. I'm getting ~100 tok/sec with prefix caching!

I recently got my hands on a (used) RTX 3090Ti and have been playing around with running local models on it. Super fun (though at times, frustrating) learning experience. Some notes.

• As one does these days, I started by asking an agent to find the most capable model that could run on 3090. It settled on qwen3.6 and I asked it to benchmark performance on different inference engines (ollama / llama.cpp, vLLM and sglang). To my surprise, ollama had the best out-of-the-box performance: around 140 tok/sec.
  • Learning: Identical model weights can have wildly varying performance on different inference engines. vLLM and sglang are powerful, but they are not geared towards consumer GPUs. Hardly any of their recipes target the RTX family (ok, a few recipes do support RTX 5090). TBH I didn't want to like ollama (see this) but I couldn't deny ollama was significantly easier to use. It had MTP speculative decoding for 3090, with ~88% draft acceptance. Even after a lot of tweaking and tuning, I couldn't get anywhere close with stock vLLM or sglang.
• But I wanted to see if I could run qwen3.8 (with usable performance). The agent benchmarked ~50 tok/sec on Ollama at 32k context length (number varied quite a bit with the setup). TTFT was much longer, almost 15s. While qwen3.6 felt super snappy, qwen3.8 was just crawling. I couldn't even get qwen3.8 to run on stock vLLM, it kept choking for one reason or another.
  • Learning: devil is in the details. Quantizations, kv-caches, attention heads, batching etc etc – there are a LOT of knobs to play with and while agents are happy to run whatever benchmarks you tell them, your unknown unknowns quickly turn into their blind spots. Lots of time and tokens spent chasing dead ends.
• So I was up and running with qwen3.8 on ollama with a 32k context length. I created a new openclaw agent with this model as the backend to test things out, and very quickly realized 32k context is just too small for openclaw. All sorts of weird failures.
  • Learning: openclaw is a context hog. Be careful what tools and skills get loaded into your agent. As example, on my primary agent, system prompt alone is 10k tokens, and often before you say a word, ~30k tokens could be used up. To be clear, this isn't isolated to openclaw, other harnesses can similarly be bloated.
  • Learning #2: for current agentic workloads, 32k context just doesn't cut it. You need at least 100k, ideally more.
• But even ollama couldn't run qwen3.8 on rtx3090 with context sizes larger than 64k. So I went back to qwen3.6, context length 80k. To test it outside openclaw, I fired up opencode and asked it to build a tetris clone. This was a disaster: tool calls kept failing, file outputs kept getting truncated. Luckily, I decided to try one more harness to be sure, this time Crush. And it crushed it: one shotted a decent Tetris clone, as a single page static HTML.
  • Learning: I ❤️ opencode, but Crush is underrated. Give it a try if you haven't yet.
• I discovered llmfit, what a great tool! I unleashed 0x alpha (via hermes) on it, surely there was some qwen3.8 quantization out there that I could run. After many many tokens (thankfully they were free!), I was told while llmfit reported many qwen3.8 variants that scored "perfect" for my hardware, most were unusable for my use-case (either non-standard metadata for vLLM loaders, or context length caps or other issues). It concluded that I should stick with qwen3.6 on ollama.
  • Learning: llmfit is awesome. But the filters are limited, so treat the "perfect" scores with skepticism. Benchmark against your workloads, then decide!
• Life was good with qwen3.6 for a few days. But I kept yearning for a more capable model that I could run fully locally. I had grand plans to use local AI for all sorts of use-cases especially where private and personal data was involved. Unsloth kept coming up in my research so I thought I'd give it a spin. It happily downloaded qwen3.8 and spun it up – I was overjoyed! Chatting with the model via Unsloth desktop was SUUUUPPPEER slow, I figured it was just qwen3.8. Until my CPUs got very loud and then I realized, Unsloth wasn't using GPU at all – inference was 100% CPU 🤦.
  • Learning: turns out that the llama.cpp bundled with Unsloth didn't recognize my GPU 🤔 I haven't dug deeply, but others have reported similar issues. Overriding that with my system native llama worked, but performance was terrible and it couldn't support large context window. Dead end.
• Finally, some reddit threads + googling landed me at this repo: https://github.com/syv-ai/qwen38-27b-rtx3090. Too good to be true, I thought! I fervently set my agents benchmarking, but alas OpenAI decided to restore the 5h limits just then 😭 Between banked resets and Ox Alpha, I persevered. And what do you know, it worked as advertised! Qwen3.8-27b on RTX 3090, 128k context, > 100 tok/sec. Unbelievable!

Notes on syv-ai/qwen38-27b-rtx3090

• single-user profile fits my use-case, but it supports a batch profile as well
• the repo is under active development (60 commits this week!). YMMV
• FP8 KV cache makes long text windows practical on 3090
• I enabled PREFIX_CACHE, makes sense for openclaw usage. Around 87% hit-rate!
• pushing utilization beyond 0.972 resulted in OOMs in the MTP speculative-decoding path
• Simon reported that Qwen3.8 can wildly overthink, so I've kept thinking to low by default, off for many use-cases.
• Initially this setup was text-only (vllm with --language-model-only flag). The model itself does support vision (understanding, not generation). But vram was tight, so I enabled vision with CPU offload – it has a modest impact on KV cache, and modest latency impact (3% in my tests), but it leaves the GPU intact for the core model weights.

Running a capable model locally is enormously satisfying. My current setup is fast and private, endless possibilities! I have a new found respect for inference engineers and people who work on inference engines.

It is totally bonkers that a 6 year old consumer GPU that was good for PC gaming just a few years ago can run a highly capable model that you can use for code generation, research, analysis, automation and all sorts of other incredible things!