Rendered at 03:18:40 GMT+0000 (Coordinated Universal Time) with Cloudflare Workers.
augment_me 1 days ago [-]
Crazy to not see a word about benchmarking/evaluation of the kernels.
IMO at this time when all performant GPU kernels are generated/hill-climbed, its the largest source of error we see.
Should be kernel run with a hot or cold cache in practice? Should your kernel be bitwise equivalent to a ref? What error margins should your kernel have to ref? How many times are you running the kernel to avoid thermal noise? Are you running on separate GPUs to avoid overfitting the current hardware? Can you cudagraph-compiling your kernel to remove launch overhead? Are you A/B B/A interleaving your kernel/reference runs to avoid running thermal bias?
nomi_uw 10 hours ago [-]
Hmm, this is a good point. I tried to be fairly brief on this part and separated a lot of the profiling to a separate page since I usually use AI for benchmarking these days. I do think there are a lot of principles that humans should learn in practice, even if most of this is done with LLMs.
I'll update the post in the coming week with a section on benchmarking. The current post does have some details on measurement, but I agree that maybe it's not covered in the required depth.
augment_me 6 hours ago [-]
It's a bit of a footgun to do benchmarking with AI models. If you worked with really performant GPU kernels (SOTA, better-than cuBLAS/FlashInfer level), you know that the LLMs reward hack:
When you do this with benchmarking, the same thing happens but on a much deeper, shittier, harder to understand level. It will set up the caches to benefit your kernels, it will set up the timing to benefit your kernels, it will set up the evaluation order to benefit your kernels. I've seen half-cache flushes, which only benefit large shapes.
I wouldn't be surprised if you pointed a language model on your benchmarking with the comment above, and it found something that is reward-hacking your task.
nomi_uw 2 hours ago [-]
Yes, they can reward hack. However, making sure the model knows what your requirements are (e.g., "do not quantize, meet this set of specifications") can mitigate that, and simply looking at the code briefly (or having the LLM describe it back to you) can mitigate it quite easily. Often, such issues are a result of insufficient specificity by the user. If you tell it just to make the model faster, it'll do it and not assume additional requirements.
Same thing applies to benchmarking, a few guardrails goes a long way. As long as you aren't completely vague ("set up benchmarks", vs. "write tests over a variety of common shapes, be careful to control for environmental variables") the models do quite well. I've checked over the code it's written and it's all fairly reasonable. I do mention cache eviction in the article, this generally wasn't an issue and the model did this automatically on the first pass (back in Opus 4.6 days). I think you might not be giving them enough credit here.
I'm not completely familiar with the writers of this article, but I've never had an LLM try to hack this, even sometimes telling me a target was impossible without quantization or approximation:
"Code based analysis already tells a detailed story, but there is an overlooked parameter: "boundaries of the hardware". " - the myainotez post
This project has spanned several models since Opus 4.5. I think even then such details were trivially known by them, thought they required much more handholding than they do now. I'm honestly not sure how the article writers ran into the hacks they did.
edit: I should mention this article is based on my experience developing an RL and inference framework dealing with trainer-inference mismatch. I don't know if hill-climbing a kernel in an isolated set of benchmarks would perform differently. At the end of the article, I mention what my hill-climber looks like:
https://www.nabo.dev/blogs/kernel-primer#section-in-the-age-...
It has to go through numerics and correctness checks, then isolated tests for common shapes (where launch parameters are determined by an autotuner, not the agent), followed by end-to-end throughput tests.
augment_me 1 hours ago [-]
A good harness is great, but if it was that was all, everyone would have SOTA perf and we would be done with GPU kernels.
If you have the tokens, let your harness go at let's say the first bf16 problem, I am curious how far you will get with it (1.73x is the leader and 2.04x is roofline). I think if you manage to get above 1.63x, I would be really interested in learning how you did that.
nomi_uw 15 minutes ago [-]
Seems like the kernel portion of that competition is closed. I can probably compare whatever the serving winner is against JamInfer when it's out (seems like they start that track on the 12th). Main issue being any submission code goes straight to SGLang which I don't think my advisor or university will appreciate spending resources on.
The top of the article shows the e2e times on some math RL tasks, with fp8 weights/activations/KV, comparing against SGLang on two models. They use the same concurrent batch size, quantization scheme, and the final lengths were about the same. I think Qwen3 models are probably very mature compared to Muse by now.
30B-A3B MoE
Jam - 67,241
SGLang - 44,696
+50.4%
8B Dense
Jam - 58,821
SGLang - 41,759
+40.9%
Math RL is pure decode, so kernels are the most major factor here.
Farmadupe 1 days ago [-]
As someone who loves being able to press a button to receive matmuls tuned to the 3090 by his feet, are there rules of thumb to work out how to set thresholds for evals?
Do I want to set a particular threshold on any individual op (or a layer for a given transformer)? e.g "for this test set, 99% of l2norms being within 1e-4 error of the golden reference kernel"
Or do I want to just shortcut all that and go with task based evals? eg "word error no worse than 1% greater than the reference speech-to-text model"?
I'm assuming the latter is the way to go in the first instance?
augment_me 1 days ago [-]
It depends on if you do training or inference and also at what scale.
For your task with the 3090, and I assume inference, I would like you suggest do end-to-end on target task.
Sometimes for example, a normalization layer can cancel out/fix an error from a previous layer, then it makes no sense to spend flops to save the numerics.
This is kind of a wild west in inference serving, you sometimes change exp to tanh, sometimes go full 4bit, sometimes remove full layers because you still perform fine on the end-task.
Numerics and correctness are more important for training kernels and backwards passes
zhengzefeng 16 hours ago [-]
[flagged]
Farmadupe 1 days ago [-]
Thanks! Interestingly I find that coding agents really don't now how to quantize a model to a given loose target. "Make it faster and accurate" is seemingly in distribution, "Make it _even_ faster and worse" is a good way to get Opus 5.5 to say "I tried nothing and I'm all out of ideas"
saagarjha 1 days ago [-]
It’s not crazy at all, the blog post is clearly about human-guided optimization with LLMs rather than hillclimbing with an autonomous agent.
augment_me 1 days ago [-]
All of the above is as-relevant in human-guided optimization.
saagarjha 1 days ago [-]
Yes, but they have nothing to do with GPUs and kernels. Those are general performance standards that would apply to any effort regardless of target.
augment_me 1 days ago [-]
Tell me more about cuda-graphs and async on CPU ;)
saagarjha 13 hours ago [-]
Sure, that might be worth discussing, I agree
kollegekid 1 days ago [-]
The writing here is so clear. Do you use Llms to write?
saagarjha 1 days ago [-]
My guess is that the LLMs helped with the diagrams but the writing is mostly or completely from a human. It’s quite good (and has a few typos :P)
nomi_uw 10 hours ago [-]
I appreciate the compliment, yes this is spot on
pjmlp 18 hours ago [-]
The page design is the standard Claude design, what do you expect?
nomi_uw 10 hours ago [-]
The post is actually mostly human written! I personally am not a fan of Claude's writing and don't even use Claude (my main subscription is Codex).
saagarjha 1 days ago [-]
> On the H200s used for most of this work, the GPU's main memory is HBM, or high-bandwidth memory. Ironically, this is the slowest memory on the chip.
Nit: HBM is off chip, unlike shared memory and registers
nomi_uw 10 hours ago [-]
Will update, thanks for pointing out.
brcmthrowaway 1 days ago [-]
Someone needs to write a book on this
yhjc2692 14 hours ago [-]
Programming Massively Parallel Processors is a classic (written by NVIDIA/CUDA engineers themselves), if you haven’t checked it out already
emigre 18 hours ago [-]
I really liked AI Systems Performance Engineering by Chris Fregly
IMO at this time when all performant GPU kernels are generated/hill-climbed, its the largest source of error we see.
Should be kernel run with a hot or cold cache in practice? Should your kernel be bitwise equivalent to a ref? What error margins should your kernel have to ref? How many times are you running the kernel to avoid thermal noise? Are you running on separate GPUs to avoid overfitting the current hardware? Can you cudagraph-compiling your kernel to remove launch overhead? Are you A/B B/A interleaving your kernel/reference runs to avoid running thermal bias?
I'll update the post in the coming week with a section on benchmarking. The current post does have some details on measurement, but I agree that maybe it's not covered in the required depth.
Primer: https://ornith.ai/defense_kernel_hack.html
In-depth(follow links etc): https://x.com/myainotez/article/2037258418843771349
Parts relevant: https://www.coreauto.com/blog/when-ai-starts-writing-systems...
----
When you do this with benchmarking, the same thing happens but on a much deeper, shittier, harder to understand level. It will set up the caches to benefit your kernels, it will set up the timing to benefit your kernels, it will set up the evaluation order to benefit your kernels. I've seen half-cache flushes, which only benefit large shapes.
I wouldn't be surprised if you pointed a language model on your benchmarking with the comment above, and it found something that is reward-hacking your task.
Same thing applies to benchmarking, a few guardrails goes a long way. As long as you aren't completely vague ("set up benchmarks", vs. "write tests over a variety of common shapes, be careful to control for environmental variables") the models do quite well. I've checked over the code it's written and it's all fairly reasonable. I do mention cache eviction in the article, this generally wasn't an issue and the model did this automatically on the first pass (back in Opus 4.6 days). I think you might not be giving them enough credit here.
I'm not completely familiar with the writers of this article, but I've never had an LLM try to hack this, even sometimes telling me a target was impossible without quantization or approximation:
"Code based analysis already tells a detailed story, but there is an overlooked parameter: "boundaries of the hardware". " - the myainotez post
This project has spanned several models since Opus 4.5. I think even then such details were trivially known by them, thought they required much more handholding than they do now. I'm honestly not sure how the article writers ran into the hacks they did.
edit: I should mention this article is based on my experience developing an RL and inference framework dealing with trainer-inference mismatch. I don't know if hill-climbing a kernel in an isolated set of benchmarks would perform differently. At the end of the article, I mention what my hill-climber looks like: https://www.nabo.dev/blogs/kernel-primer#section-in-the-age-... It has to go through numerics and correctness checks, then isolated tests for common shapes (where launch parameters are determined by an autotuner, not the agent), followed by end-to-end throughput tests.
Look at this challenge:
https://openkernelchallenge.dev/#leaderboard
If you have the tokens, let your harness go at let's say the first bf16 problem, I am curious how far you will get with it (1.73x is the leader and 2.04x is roofline). I think if you manage to get above 1.63x, I would be really interested in learning how you did that.
The top of the article shows the e2e times on some math RL tasks, with fp8 weights/activations/KV, comparing against SGLang on two models. They use the same concurrent batch size, quantization scheme, and the final lengths were about the same. I think Qwen3 models are probably very mature compared to Muse by now.
30B-A3B MoE
Jam - 67,241
SGLang - 44,696
+50.4%
8B Dense
Jam - 58,821
SGLang - 41,759
+40.9%
Math RL is pure decode, so kernels are the most major factor here.
Do I want to set a particular threshold on any individual op (or a layer for a given transformer)? e.g "for this test set, 99% of l2norms being within 1e-4 error of the golden reference kernel"
Or do I want to just shortcut all that and go with task based evals? eg "word error no worse than 1% greater than the reference speech-to-text model"?
I'm assuming the latter is the way to go in the first instance?
For your task with the 3090, and I assume inference, I would like you suggest do end-to-end on target task.
Sometimes for example, a normalization layer can cancel out/fix an error from a previous layer, then it makes no sense to spend flops to save the numerics.
This is kind of a wild west in inference serving, you sometimes change exp to tanh, sometimes go full 4bit, sometimes remove full layers because you still perform fine on the end-task.
Numerics and correctness are more important for training kernels and backwards passes
Nit: HBM is off chip, unlike shared memory and registers