the prefix cache is all important for agentic workloads
in this deepdive we discuss how to preserve it from both the agent client side + the inference engine perspective
coding agents resend the whole conversation on every turn. prefix caching keeps long sessions fast, but only if the harness and the inference engine both do their part.
@zainhas from Together and @parthsareen from @ollama wrote down what each side needs to get right.
I joined @noeri_ai part-time two months ago.
It has been a blast! I’m very impressed by the team that Yejin and Carlos (@guestrin) have built. Everyone is incredibly talented and kind, and it’s been so much fun working together.
I’m especially excited about plural AI for
in a world where agent experience (AX) >> developer experience (DX) you need to make sure agents can use your platform
this is how we use evals to make sure agents can use the Together platform🔥