How to collect feedback on an AI chatbot or assistant

Why a thumbs rating alone does not explain a bad answer

A thumbs down tells you that something went wrong, but not what. The answer may be factually wrong, too long, off topic or missing a step. The app may also have failed, with a timeout, a broken stream or a tool call that returned an error. These need different fixes, so the feedback has to carry enough context to tell them apart.

The pattern that works is simple. Every answer gets a thumbs up and a thumbs down. A thumbs down opens a short note field, and the note goes to the REST API with the ids that let someone find the exact exchange. The REST API reference lists every field, and the API returns 201 with the new item id when the report is saved.

Attach thumbs to each answer

Put the buttons next to the answer, not in a global menu. A user who thinks "that was wrong" is looking at the answer, so the feedback should sit right there. Keep the note optional. People will click thumbs down quickly, and a required form stops most of them.

1. Render the thumbs buttons after each assistant message, and store that message's id.
2. On thumbs up, send a praise item with no form.
3. On thumbs down, show a note field with a button labeled to send it.
4. Send the note with the message id and the conversation id, then show a short confirmation.

Here is the browser function that sends either kind of answer feedback. It calls the REST API with the public key, so the chat page's origin must be allowed if you set allowed origins for the product:

const FEEDBACK_URL = "https://escutaproduto.com/api/v1/feedback";

async function sendAnswerFeedback({ kind, note, conversationId, messageId, appError }) {
  const response = await fetch(FEEDBACK_URL, {
    method: "POST",
    headers: { "content-type": "application/json" },
    body: JSON.stringify({
      key: "pk_your_product_key",
      kind: kind,
      message: note && note.length >= 2 ? note : "Thumbs down with no note",
      metadata: {
        conversationId: conversationId,
        messageId: messageId,
        model: "your-model-name",
        promptVersion: "2026-10",
        failure: appError ? "app-error" : "answer-quality",
      },
    }),
  });
  return response.status === 201;
}

A thumbs up can use kind: "praise" with a fixed message such as "Helpful answer", so praise still shows up in the inbox and the 30-day chart. The message is required by the API, so keep the fixed text short and clear.

Which ids belong in metadata?

Metadata is what turns a note into something you can investigate. Send ids that point back into your own system, and keep the content itself out of the report:

Key	What it points to	Why it helps
conversationId	The chat thread	Finds the full exchange in your logs
messageId	The assistant message	Pins the exact answer that was rated
model	The model name you call	Shows whether a model change caused a drop
promptVersion	Your system prompt version	Connects a complaint to a prompt release

Do not paste the whole transcript into metadata. Transcripts are large, often contain personal data, and the metadata limit is 4 KB. The ids let an engineer open the exchange in your own tools, where access rules already apply.

How do you separate model errors from product bugs?

The app knows things the user does not. It knows whether the request timed out, whether the stream stopped midway or whether a tool call failed. Set that fact in metadata when you send the report. The failure key in the example above does this. Use app-error when your code caught a failure, and answer-quality when the request completed and the user simply did not like the answer.

On the team side, sort the work by reading the notes and the failure key together:

• App errors are product bugs. They are fixed in your code, your infrastructure or your streaming layer.
• Answer quality reports are model or prompt problems. They are reviewed against a set of example questions and fixed in the prompt, the retrieval or the model choice.
• A note that mentions the wrong figure or a missing step is answer quality, even when the app did nothing wrong.

Escuta Produto does not classify feedback with AI or tags. The triage is yours, which is why the metadata key matters. Use internal notes in the inbox to record the decision, and export the CSV when you want to count app errors separately from answer complaints.

When should the app ask before it sends?

When your code catches an error, the app can offer a report, but it should not send one without the user's agreement. A background send of every failed request floods the inbox and includes content the user did not choose to share. Show a line such as "This answer stopped early. Send a report to the team?" with a button, then call the same function with appError: true.

Keep the chat's privacy stance in view. Users type personal details into assistants. Do not attach the user's message text to the report unless they wrote the note themselves, and let the note be the only free text that leaves the chat.

What about abuse and very long notes?

Rate limiting applies per IP address and product, at 10 requests per minute. A shared office network can reach that quickly when a team tests the assistant together, so debounce the buttons and disable the send button while a request is in flight. The request body must stay under 16 KB, so cap the note length in the input field. A few hundred characters is plenty for a useful note.

Use the hosted form for longer feedback, such as a detailed write-up of a broken workflow. Link to it from the help page, where a user who wants to explain a problem in full can do so without crowding the chat.

Setting up the assistant inbox in Escuta Produto

Create a product for the assistant and copy its public key into the chat page. Ask one question, rate the answer with a thumbs down and send a note. Check that the item shows the conversation id, the message id and the failure key. Connect the Slack or Discord webhook so the team sees new reports as they arrive.

Workspace admins can also connect an AI client, such as Claude or ChatGPT, to the inbox through the MCP endpoint at https://escutaproduto.com/mcp/inbox. The owner signs in and approves the connection, and the assistant can then list products, read and filter feedback, change status and add internal notes. The notifications guide covers the webhook, and the triage guide describes a routine for sorting the results. For the wider question of when to ask, read how to collect customer feedback, and the SaaS article covers placing the entry point inside a logged-in product.