A customer-feedback workflow often begins with a bounded question: which messages need a closer look? A sentiment classifier can provide one signal for that decision by assigning text to categories such as positive, neutral, or negative.
Small neural networks can be candidates for this work when the task is narrow, the examples are representative, and the output can be evaluated clearly. Choosing that approach requires comparing it with simpler alternatives and with language-model-based classification on the same task.
This evaluation discipline comes from my own 2016 text-classification experiment, written up here.
Separate sentiment from the action you need
Sentiment describes an attitude expressed in text. Intent describes what the sender wants. Urgency describes how quickly a response may be needed. Those signals can disagree.
“I love the product, but the invoice is wrong” contains positive sentiment and a concrete problem. “Thanks for another delay” may express negative sentiment through sarcasm. A routine-looking message can contain an urgent request without emotionally charged wording.
The business should define which distinction matters before choosing the model. If the useful outcome is a queue of unresolved complaints, a generic sentiment score may need to be combined with issue identification and a human review step.
Define labels that people can apply consistently
A training set is a statement of the behavior you want to learn. If reviewers disagree on mixed feedback, short messages, or sarcasm, that disagreement becomes part of the learning problem.
I would start with a written label definition and examine disagreements before treating the labels as ground truth. A message that cannot be classified confidently may deserve an “uncertain” review path. That path is a workflow choice, not necessarily another learned class.
Keep the distinction between examples used for learning, examples used for selecting an approach, and examples reserved for final evaluation. For customer communications, group related messages together so one conversation is not split across those boundaries.
Compare models on the same information
A keyword rule, a conventional classifier, a compact neural network, and an LLM-based approach can all be useful candidates. Give each the same eligible examples and the same target definition. Evaluate quality alongside latency, operating cost, review effort, and data-access requirements.
The representation of text matters, but the important business question is the resulting behavior. A compact model that is fast on familiar messages may struggle with a new product vocabulary. A more flexible model may add cost or require additional controls around the information sent to it.
Fit any learned transformations using the training partition. Test-set information must not influence vocabulary selection, scaling, or model choice. The scikit-learn guidance on data leakage explains this general evaluation failure in detail.
Measure the errors the business will feel
Suppose the target is negative feedback that deserves review. A true positive is a correctly surfaced case. A false positive is a message surfaced unnecessarily. A false negative is a case the system missed.
Precision = TP / (TP + FP) describes how often surfaced cases are actually relevant. Recall = TP / (TP + FN) describes how many relevant cases were surfaced. Both should be reported with their sample counts and the test-set definition. A metric with no eligible denominator should be left undefined rather than invented.
Overall accuracy can conceal an important failure when one category dominates the data. Inspect a confusion matrix and review errors by message type, customer vocabulary, and conversation context. These views help explain what a single aggregate score hides.
A model's output score also needs interpretation. A high score is not automatically a calibrated probability. If scores determine whether a message is reviewed, evaluate the actual review policy and the workload it creates.
Put uncertainty into the workflow
A useful first deployment can prepare a review queue with the source message, proposed category, and a clear route for exceptions. The person reviewing it can correct mistakes before the system takes consequential action.
The workflow should record model identity, inputs used, the prepared result, and the action taken. These records support later error analysis when the business adds a service, changes its terminology, or receives a different mix of requests.
When the workflow changes, use fresh examples to decide whether the model still meets its acceptance criteria. Retraining is one possible response; revising the label definition, representation, or review process may be more useful in a particular case.
Start with one repeatable task
For a business, the deliverable is a process that prepares useful work and exposes exceptions. A sentiment classifier could prioritize incoming feedback. An agent could then assemble relevant customer context or prepare a response draft under agreed permissions.
I start with one process, its tools, success measures, and a clear owner. AI-Rey’s agent services turn that scoped task into a workflow with a record of its work.
Scope, stated plainly. This post is my evaluation workflow for sentiment analysis. My 2016 experiment supplies the hands-on foundation; the examples here explain how I define and score a business task.
Sources and further reading
- My archived 2016 text-classification experiment, the historical work behind this series.
- Brain.js, a library for neural networks in JavaScript.
- Common pitfalls in machine-learning evaluation, including preprocessing and data leakage.