Metis is the internal intelligence layer I build at Praxis Forward. It draws on what our company knows, from Linear and Notion to our CRM and meetings, so anyone on the team can ask it for help.
We talk to it in Slack, in the open, because we believe in public communication as the default: what stays in a private exchange only helps the people in it. A conversation in a channel is one the rest of the team can read and search later, so people see what Metis can do and how others ask it. Shopify built River on the same idea. It works only in public channels, on the argument that if every interaction with an agent happens in a private window, the only person who learns anything is the person at the keyboard.
In Slack, you mention Metis in a thread and it joins that conversation. From then on it has to decide, message by message, whether it’s being spoken to.
Today, when someone follows up with “can you send the diff?” and doesn’t mention it again, it answers. When two teammates trade “thanks, looks good,” it stays quiet. Metis decides by asking the OpenAI Decisions API how likely it is that a reply is addressed to it, then comparing that probability with a threshold kept in code. The threshold came from an eval.
A natural conversation means knowing when to reply
How an assistant behaves in a conversation decides whether people keep using it. I believe AI only gets used by people when it works as naturally as possible, so it was important to me that interactions with Metis in Slack felt that way. In a thread, that means answering when it’s spoken to and staying out of the way when it isn’t. That is why I spent time on when Metis replies.
After the first mention, every later message in the thread is a candidate for a reply. At first Metis answered every single one, from anyone in the thread, whether or not it needed an answer. That was noisy and annoying. In a thread about our new CRM it kept chiming in, even on a teammate’s offhand remark about dashboards. The quick fix was to force Metis to reply only to @-mentions. The noise stopped, but needing a mention every time felt tedious, and it felt like a band-aid in some ways. Having to remember to @-mention someone in what is meant to feel like a natural conversation is unnatural, so Metis had to make the call about whether it’s being spoken to. Then it occurred to me that a model could make that call: a second, separate call whose only job is to read the new message and decide whether Metis should answer it.
Choosing an OpenAI Decisions API question type
Once a model is making the call, the first constraint is speed. The check runs before every reply, so a noticeable wait would bring back the same unnatural feeling I was trying to remove. That points to a decisions API, an endpoint that takes some text and a list of typed questions and returns typed answers instead of writing a reply. There is no prose to generate and nothing to parse. TypeSafe’s Jev came first, launching in September 2026. TypeSafe describes it as evaluating the questions in parallel instead of producing tokens one at a time. The OpenAI Decisions API followed in October 2026, and it is the one Metis uses. It runs gpt-6-luna and reports answers about ten times faster than the same model through its Responses API. I haven’t measured either myself.
OpenAI’s version has three question types. Each takes some text and a question and answers with probabilities. They differ in the shape of the answer you ask for.
A predicate checks whether a condition is true. You get back one probability from 0 to 1.
A choice picks one option from a fixed set. You get back the pick, a probability for every option, and a confidence value.
A score rates the input against ordered levels. You get back a probability for each level and their weighted average, which can land between levels.
My question is a yes or no, and there are two ways to ask it. A choice with two options, yes and no, returns the model’s pick, which is whichever option it rates higher. That is a cutoff at 50%. A choice also returns each option’s probability, so I could set my own cutoff on those, but a predicate gives me the probability directly. So Metis asks a predicate: how likely is it that this reply is addressed to it?
The call is still binary in the end, but the cutoff is now a number I control. It lives in one constant, ReplyDecision::THRESHOLD, which a spec can test and a code review can argue about.
Metis is built on Ruby on Rails with RubyLLM, which provides a judge primitive: a reusable set of typed questions over application data. Metis’s judge has one probability question, and RubyLLM sends it to OpenAI’s Decisions endpoint as a predicate. Here is app/models/slack_assistant/addressed_judge.rb:
module SlackAssistant
class AddressedJudge < RubyLLM::Judge
model(provider: OpenaiModel::PROVIDER) { OpenaiModel.model_id }
probability :addressed,
"Metis is an AI assistant in a Slack thread. It answered the conversation_with_metis messages. " \
"later_replies were posted after its last answer without mentioning it. " \
"Is reply, which does not mention Metis either, addressed to Metis and waiting for an answer from it?",
{
yes: "The reply asks Metis a question or for more work, or follows up on Metis's last answer",
no: "The reply talks to other people, only acknowledges or thanks, or needs no answer from Metis"
}
# ... sig and typed annotations removed
def self.addressed_probability(input)
OpenaiModel.register!
judge(input).addressed.probability
end
end
endThe yes criterion covers a question, a request for more work, or a follow-up to Metis’s last answer. The no criterion covers thanks, acknowledgements, and conversation between other people, like the “thanks, looks good” exchange from the opening.
The judge sees three things. conversation_with_metis is the latest 10 user and assistant messages from the chat, with tool calls left out. later_replies is up to the latest 10 unanswered replies before this one, which Metis stores as observed turns. reply is the one being judged. Each is cut to 1,000 characters, and context is capped at 10 items. That bounds cost and latency, and stops a long side discussion from being resent with every new reply.
The decision itself lives in app/models/slack_assistant/reply_decision.rb:
# The lowest probability of yes that gets an answer.
THRESHOLD = 0.45
# ... CONTEXT_MESSAGES and MESSAGE_LIMIT constants, sig, initialize, and private helpers removed
def addressed? = probability >= THRESHOLD
def probability
@probability ||= AddressedJudge.addressed_probability(input)
end
def input
{
"conversation_with_metis" => conversation_with_metis,
"later_replies" => later_replies.map { |reply| shorten(reply.prompt_line) },
"reply" => shorten(turn.prompt_line)
}
endThe decision is one line of plain Ruby, probability >= THRESHOLD, so I can test it without a model. Here is the part of spec/models/slack_assistant/reply_decision_spec.rb that pins it:
RSpec.describe SlackAssistant::ReplyDecision do
# ... conversation, reply, and helper setup removed
{
"above the threshold" => [ 0.9, true ],
"at the threshold" => [ described_class::THRESHOLD, true ],
"below the threshold" => [ described_class::THRESHOLD - 0.01, false ]
}.each do |description, (probability, addressed)|
it "treats a reply #{description} as #{addressed ? '' : 'not '}addressed to Metis" do
stub_decisions(probability)
expect(described_class.new(reply).addressed?).to eq(addressed)
end
end
endstub_decisions fakes the Decisions endpoint, so the spec never calls a model. It reads THRESHOLD instead of hard-coding 0.45, so it pins the comparison. The eval owns the number.
What happens around the judgment
The judge will sometimes be wrong, and the two ways of being wrong don’t cost the same, though neither costs much. An answer nobody needed is the noise I started with. A missed answer costs one more message. There is also fast feedback: Metis shows that it is thinking, and when I don’t see that, I know to mention it.
The judgment runs in a background job, not inside Slack’s events request, because Slack expects an answer within seconds and a model call doesn’t fit that budget. When the probability clears the threshold, the turn moves from observed to pending and Metis answers. Below it, the turn stays observed and becomes context for the next mention, which is how a missed answer stays cheap. A failed judgment leaves the turn observed too, so a failure costs a missed answer and nothing worse. Each judgment logs the turn id and the probability, never the text.
Picking the threshold from a RubyLLM eval
To set the threshold I needed replies Metis should answer and replies it shouldn’t, all scored by the same judge. Fortunately we had plenty of existing Slack threads to build a data set from, so I asked Claude to go through them and help me turn them into an eval.
RubyLLM has an evaluation class for running LLM evals like this one. A RubyLLM::Evaluation loads a dataset of cases from app/evals, runs a perform method on each one, and checks the result with assertions. It can also have a model grade answers against written criteria, which I turn off here because the judge already returns a number. Here is app/evals/addressed_judge_evaluation.rb:
class AddressedJudgeEvaluation < RubyLLM::Evaluation
evaluator false
def perform(input) = SlackAssistant::AddressedJudge.addressed_probability(input)
def assertions
probability = output
answered = probability >= SlackAssistant::ReplyDecision::THRESHOLD
assert_equal(expected_output == "reply", answered, "probability #{probability.round(3)}")
end
endperform hands each case’s inputs to the judge and returns the probability. assertions compares the answer at the current threshold with the case’s label, and puts the probability in the failure message, so a miss shows its score. The dataset sits next to the class in addressed_judge_evaluation.yml. It holds 16 cases, 6 where Metis should reply and 10 where it shouldn’t, anonymised from a single thread. Each case has a name, the three inputs the judge sees, and an expected_output of reply or no reply:
cases:
- name: asks_again_after_an_answer
inputs:
conversation_with_metis:
- "<@U_SAM>: are you able to add a company record for Northwind?"
- "Metis: I can't add it directly, but I can format a record for you to enter."
later_replies: []
reply: "<@U_SAM>: can you send me the sign-in link again, the last one expired"
expected_output: reply
- name: colleague_reacts_to_the_news
inputs:
conversation_with_metis:
- "<@U_ALEX>: summarize what shipped in the CRM today"
- "Metis: Today's CRM release adds company pages, a contact picker, and task forms."
later_replies: []
reply: "<@U_JORDAN>: nice one"
expected_output: no replyThose are two of the 16, anonymised and shortened. Every case calls OpenAI, so the evaluation runs only on request with bin/rails "ruby_llm:eval[AddressedJudgeEvaluation]". It prints a line per case and saves the full report as JSON, and EVAL_REPETITIONS=3 runs each case three times.
Claude Code ran the eval in a loop while it built the judge, with gpt-6-luna as the judge model. Every no-reply case scored 0.39 or less. Every reply case but one scored 0.50 or more. Scores were identical across 3 repetitions. The one outlier reply case scored 0.36, so at 0.45 it is missed. That makes the eval 15 of 16.
Any threshold from 0.40 to 0.50 gives the same 15 of 16 on this set. The highest no-reply case scores 0.39 and the lowest reply case besides the miss scores 0.50, so every cutoff between them splits the rest the same way. I set 0.45, the middle of that gap, so a small shift in scores on real replies is less likely to push a case across the line. The failures I’ve seen in use are misses, so I’m not tempted to push it toward 0.50, and 0.40 is still inside the range if Metis needs to answer more.
Claude flagged two of the labels as borderline when it built the eval: the miss at 0.36 and the highest no-reply case at 0.39. Those two set the edges of the gap, so the range is only as firm as those calls.
Running the eval is also how the judge’s sensitivity to small prompt changes showed up. Renaming the later_replies input moved one no-reply case from 0.39 to 0.86. Adding “including about Metis” to the no criterion dropped the eval to 13 of 16. Both changes were reverted. The eval calls OpenAI and runs only on request, so a prompt change goes unmeasured unless someone remembers to run it.
How it works now
In use, Metis answers when it’s spoken to and stays out of the way when it isn’t. The only failures I’ve seen are misses, where it stayed quiet when it likely should have answered. That has happened a few times, and each time the missing “thinking” indicator told me to mention it. I haven’t seen it answer a message nobody needed answered. The goal is to take those misses, add them to the eval, and see what I can tweak.
The same pattern works outside Slack. Whenever a model gates something with a cost, such as sending a message, taking an action, or escalating to a person, ask it for a number and keep the decision in your own code. The decision becomes a line you can unit test, and the threshold becomes an input you can change when the data changes. An eval shows which thresholds work on the cases you have, and the cost of each mistake picks among them. Live traffic then shows whether you picked well.
If a model is about to make a decision that costs something when it’s wrong, I can help you build the eval, the threshold, and the checks around it. See how I help teams ship production AI.
Further Reading
- Harness Engineering: The Four Levers covers the levers I check when an agent goes wrong.
- Autonomous Optimization Loops With Autoresearch is about measuring a change before you keep it.
- Governing AI Agents Without Killing Them covers the boundaries around what an agent may do.
- Agent Loops vs. Workflows: The Boundary That Makes AI Reliable covers where an agent loop should hand off to a workflow.
- My Agentic Coding Workflow is how I plan and build features with agents.