Computer vision on the packaging line: how reliable is AI reject classification really?

What a camera on the line actually sees

A vision system consists of a camera, lighting and a model that compares every passing unit with what it has learned to recognise. As a result, it can only detect what leaves a visible trace at that one point in the line, from that one angle. That is a limitation and, at the same time, a clarification: it lets you determine in advance which defects are realistically within reach.

• Labelling: whether there is a label, whether it is straight, and whether it is the right label for this product and this pack.
• Coding: whether the best-before or lot code has been applied, is complete and legible, and is the code that should be there.
• Closure: whether the cap, lid or seal is in the right position, and whether there is product or dirt in the sealing area.
• Fill and shape: fill level in transparent packaging, dents, tears, deformation or a clear colour deviation.
• Completeness: the number of units in a tray, an outer case or a multipack.

And above all: what a camera does not see

The other list is just as important. Microbiological contamination, a foreign object inside the product, the chemical composition, allergen cross-contamination that leaves no visible trace, the temperature history of a batch, taste and smell: none of that shows up in the image. Vision therefore does not replace metal detection or X-ray inspection, and it is not an alternative to the control measures you have already put in place for those hazards.

There is a subtler limit as well. A defect on the underside of the pack, on a side the camera does not see, or at a moment when the product has already passed the inspection point, simply does not exist for the system. When assessing a quotation, the question of which hazards and defects fall outside the field of view is at least as important as the question of what the system can recognise.

Vision alongside the human line inspector, not in their place

The comparison with an experienced line inspector is not clear-cut, and that is exactly what makes it useful. A human generalises: someone who sees something they have never encountered notices that it is not right and raises the alarm. A human also uses context, such as the fact that there has just been a fault or that a particular batch of film behaves differently. Set against that are well-known weaknesses: judgement differs between employees and between shifts, attention declines as a shift goes on, and at line speed full inspection of every unit is not a realistic task.

A vision system is exactly the reverse. It judges consistently, does not get tired, looks at every unit instead of a sample, and records what it saw. But it only recognises the defect categories it has been trained on, it does not know your context, and it fails silently: it does not report that it does not understand something, it simply passes an unknown defect as good. That last point is the most important difference from a human and the reason the system continues to need oversight.

The sensible division of roles follows naturally from this. The camera does the repetitive, complete check for sharply defined, known defects. The human assesses what is flagged, keeps an eye out for the unknown, and decides and signs. That is the same division that HACCP and FSSC 22000 already require of you, and it is also the division that proves tenable in crop production, in vision sorting after harvest.

Two kinds of error, with very unequal costs

A reject system can be wrong in two ways, and those two errors cost very different things. A false reject is a good unit that is rejected. That costs product waste, lower line utilisation and time for re-inspection. There is also an indirect cost that weighs more heavily than most companies estimate in advance: if the line wrongly rejects too often, operators start turning the sensitivity down or bypassing the system. In practice, a system set too strictly often ends up as a system that is switched off.

A false accept is the reverse: a defective unit that is accepted and goes through to the customer. That costs a complaint, a blocked pallet, in the worst case a recall, and the trust that is lost along with it. The NVWA (Netherlands Food and Consumer Product Safety Authority) publishes safety warnings from companies about unsafe food and consumer products; that is the route by which such a recall becomes public.

Because the costs are so unequal, there is no single correct setting. There is a chosen operating point, and that choice is a management and quality decision, not a technical one. Moreover, the operating point may differ by defect category: a dent in an outer case that slips through is a different matter from a wrong label that slips through. That label carries the allergen information required by Regulation (EU) No 1169/2011, so a wrong label means that legally required information is missing or incorrect. Record who chose which operating point and on what grounds, because that is exactly what an auditor will want to be able to trace later.

Why high accuracy says little about reject classification

In reject classification, accuracy is a misleading number, for a simple arithmetical reason: rejects are rare. If only a fraction of your production is defective, a system that accepts everything already achieves an impressive accuracy percentage, while not stopping a single problem. One composite figure hides exactly the information that matters. So always ask for the two rates separately, and per defect category.

When assessing a vendor, this is the short list of questions that separates a sales figure from a measurement:

• On which dataset was this measured: on your product and your packaging, or on a demo set from the vendor?
• How many units per defect category were in the test, and how many of them belonged to the rare categories that matter most?
• What were the false reject rate and the false accept rate separately, per category, instead of one combined number?
• At which setting was that measured, and how do both rates move if you make that setting stricter or more lenient?
• What does the system do with a defect it has never seen before, and is that visible or does it silently pass as good?
• Was it measured at your line speed, with your packaging material and your lighting, or under laboratory conditions?

How to validate the rate yourself

A reliable rate does not come from a brochure but from a measurement on your own line. That measurement follows a fixed order, and skipping the first step makes all the following steps worthless.

• First set the standard. What counts as a reject, in words and in example images, agreed between quality, production and your customer's specification. Without an agreed standard you measure disagreement, not performance.
• Determine the human ceiling. Have two inspectors assess the same set independently. Where they disagree with each other, a model cannot demonstrably be right either; that spread is your realistic upper limit and, at the same time, your baseline measurement.
• Build an independent validation set. Real production units, spread across shifts, suppliers, packaging batches and lines, with the rare defect categories deliberately over-represented, and strictly separated from the material the model was trained on.
• Run a challenge test. Deliberately feed known defective units in at line speed and record what the system does with them, just as you already periodically verify the metal detector with test pieces. This gives you your false accept rate per category.
• Re-inspect the accepted stream. Periodically take a sample from what the system has accepted and assess it manually. You never find missed defects in the reject bin, only in what was allowed through.
• Report by category in numbers of units, not as a single percentage: how many units were assessed, how many were wrongly rejected, how many were wrongly let through.

Validating once is not validating

A vision system that is right today may have quietly drifted by next month. The causes are rarely spectacular: a new film supplier with a different gloss, a modified label design, a lamp that is ageing, a camera that sits slightly differently after a cleaning round, a recipe change that alters the colour. The model does not report this. It keeps judging with the same certainty, only on the basis of images that no longer resemble what it was trained on.

That is why re-verification belongs in your change management, as with any other control measure. A change in packaging material, label, lighting, camera position or recipe is a reason to verify again before you release.

• Track the reject rate per line and per shift as a control chart. A sudden drop is as suspicious as a spike: a camera that no longer sees anything no longer rejects anything either.
• Keep sampling the accepted stream at a fixed frequency, even when nothing seems to be wrong.
• Record every change to the setting: who adjusted the sensitivity, when and why. A threshold that is quietly turned down during a busy shift is the most common way a validated system stops working.
• Keep the images of the rejects and of a sample of the accepted units, so that a judgement can be reconstructed afterwards and does not exist only as an outcome in a log line.
• Record what happens if the system fails or does not pass verification: stop the line, return to full manual inspection, and block and reassess everything from the last successful verification onwards.

Where this sits in your HACCP plan and under the AI Act

Determine explicitly what status the vision check has in your food safety system: a critical control point, an operational prerequisite programme or a verification step. For a cosmetic check that is a formality; for a label and allergen check it certainly is not. Take the decision in writing, with the reasoning attached, and make sure the records tie in with the traceability and quality data you already keep. An auditor accepts a traceable assessment, not a black box.

Under the EU AI Act, a camera system that sorts on quality is usually not a high-risk system within the meaning of Annex III. In addition, Article 6(1) of Regulation (EU) 2024/1689 sets two conditions side by side, and both must be met. The first: is the AI component a safety component of a product covered by the Union harmonisation legislation listed in Annex I to that Regulation, including the Machinery Regulation (EU) 2023/1230, which applies from 20 January 2027? The second: must that product undergo a third-party conformity assessment under that same legislation? Only if the answer to both questions is yes do you end up on the high-risk track. For many machines the Machinery Regulation allows a self-declaration of conformity, so a yes to the first question alone does not yet settle the classification.

For most packaging lines the answer is no: a camera that only rejects and ejects is normally not a safety component, whereas a camera that stops the machine to protect the operator is a different story. Moreover, the heavy requirements for AI in regulated products will only apply later. Answer the classification question today nonetheless, and in writing, because an auditor will want to see on what grounds you reached your conclusion.

Article 4 already applies, and has done since 2 February 2025. Since 27 July 2026 it has asked for measures that support the AI literacy of employees who work with AI systems. For the operator who adjusts the threshold and the quality officer who assesses the rejects, in practice that comes down to understanding how the model reaches a judgement, where the assumptions and limits lie, and when an outcome must be corrected manually. Finally, note one point that is easily overlooked: if employees' hands or faces appear in the images, the recordings are personal data. Agree what you keep, for how long, and who has access to it.

Where to start in practice

Do not start with a quotation process for the whole factory, but with one defect category on one line, where manual inspection currently takes the most time or carries the greatest risk. That is often the same place where the tight labour market in the food industry is felt most: an inspection step that comes under pressure as soon as staffing falls short. Write out the standard for that one category, with example images, and have quality and production sign it off together.

Then record how things work now: how many units are assessed, by whom, at what frequency, what share is currently rejected, and how many complaints about this type of defect come back from customers. That is your baseline measurement, and without it a vendor trial cannot be assessed fairly. Our free Practice Scan helps you choose that first category and sharpen the baseline measurement, so that an investment in vision rests on your own figures and not on a percentage from a presentation.