What happens to the concept of quality itself when smart models enter the decision cycle?
Listen to this article
Read by Anchor
Before we continue…
In the previous article I reached a simple conviction:Artificial intelligence is a powerful assistant, but it does not bear professional responsibility for the outcome.
Its ability to produce a quick and convincing answer does not absolve us from verification, nor does its recommendation become a decision.
The conclusion was:Let it help you prepare the decision… but do not let it own the decision.
But after a while a more important question emerged:
What if artificial intelligence is no longer just a tool we use to write reports or summarize documents, and instead it chooses for us what we read, what we pay attention to, and what we prioritize?
Here a different story begins.
Artificial intelligence does not merely present information… it reshapes it
When I give a smart model fifty reports and ask it to summarize the main issues, it does not return the fifty reports to me.
It creates for mea shortened version of reality.
It selects some information.
It merges other information.
It determines what it sees as recurring.
And it decides, based on the instructions and context available to it, what deserves to appear in the summary.
Here is a point I believe we do not pause on enough.
We sometimes treat the summary produced by AI as if it were merely a shorter version of the original data.
But it is not always so.
The abbreviation itself is a decision.
When the model condenses one hundred pages into a single page, ninety-nine pages are no longer in front of the decision-maker.
And the question becomes:
What remains?
And what has disappeared?
The model does not “understand” the file the way we understand it
This is also an important point.
When a human reads a report, they bring into the reading their expertise, context, and what they know about the problem, the people, and the project or organization.
Whereas the large language model (LLM) deals with the information made available to it within a specific context.
If the information is not present before it, it will not know it.
And if it is present in a document that was not retrieved or entered correctly, it may behave as if it were missing.
This explains something important:
AI analysis may be very excellent…
for the information they saw
But what about the information they did not see?
This problem appears in all fields.
A doctor uses a system that helps them summarize a patient’s record.
A bank uses a model to analyze client data.
A company uses AI to review thousands of complaints.
Human Resources Management uses it to summarize résumés.
or an engineering team they use to review inspection reports
In all these cases, the quality of the answer depends not only on the model’s intelligence, but also onWhat information has it actually received?
And from here the power of retrieval emerges… and its problem as well.
Many modern AI applications do not rely only on the knowledge inside the model, but use what is called Retrieval-Augmented Generation, RAG.
Simply, the system first queries the document store, extracts the parts it believes are linked to the question, and then passes them to the model to construct the answer.
The idea is very strong.
Instead of the employee searching through hundreds of files, the system can find the parts most relevant to their question within seconds.
But here a new question arises:
What if the system retrieves the wrong document?
Or an older version?
Or five paragraphs that seem more linguistically tied to the question, while the sixth paragraph, the least similar, is actually the most important?
In that case, the model’s answer could be entirely logical based on the retrieved information, but the problem occurred before the model even began to answer.
And this makes me look at AI in a different way.
The problem is not always in the “answer”.
Sometimes the problem isThe information that the model was allowed to build its answer on.
Order is sometimes more dangerous than error.
If the model gave me a clearly wrong piece of information, I could discover it.
But there is a quieter impact.
Imagine a system that analyzes a thousand customer complaints and presents the management with the ten most important problems.
Or analyzes hundreds of non-conformance cases and identifies the most risky suppliers.
or reads medical reports and prioritizes certain cases
It is natural for a person to start with what the system placed at the top of the list.
Here, AI no longer just provides information.
He practicesRanking.
And arrangement affects attention.
Attention influences the decision.
The system may never tell you:
Make this decision
But if it is the one who chooses for you what you see first, what you see second, and what you never see…
It has indeed already started influencing the decision.
Here a different risk emerges: bias toward automation
There is a known human behavior called automation bias.
In simple terms: when a machine or a ready-made recommendation system is presented to us, it becomes easy to give it more weight than it deserves, especially if the system’s results have previously been good.
Initially we review AI recommendations carefully.
After twenty correct results, we begin to trust more.
And after a hundred correct results, the review may shift from:
Is this result correct?
To:
Mostly correct… Let me just see if there is anything clear.
And here a very serious change is happening.
Humans are still present in the process.
And the signature remains human.
But the level of human thinking itself has begun to decline.
In my view, this is one of the most important issues we need to discuss as AI usage expands.
The presence of a human within the process does not necessarily mean that the human still controls it.
And the model itself is not fixed.
There is another point that we rarely consider when we treat artificial intelligence as a work tool.
The model we use today may not be the same as the model we use after several months.
The version may change.
The way of responding may change.
The search or retrieval mechanism may change.
The system's internal instructions may be modified.
We may obtain different results for the same question even though our method has not changed visibly.
Here, AI becomes different from many traditional tools.
We do not always deal with a fixed machine whose behavior can be predicted in the same way.
But with a system that evolves continuously.
Therefore, the more artificial intelligence is integrated into critical processes, the question becomes:
Are we using AI as a tool… or have we begun to rely on a system that changes faster than our procedures?
However, this is exactly its strength.
All these questions do not mean that we should back away from using artificial intelligence.
On the contrary.
I think its strongest uses have not yet fully emerged.
We have vast amounts of knowledge within institutions that we do not benefit from.
Old reports.
Complaints
Correspondence.
Test results.
Lessons learned.
Previous decisions.
Expertise is distributed across employees, files and various systems.
For the first time we have tools that can help us search within this memory, link information together, and discover patterns that are difficult for humans to see amid this volume of data.
And this is not merely an increase in productivity.
This could change the way institutions learn from their past experience.
But here, exactly, we must understand the difference:
AI does not give us the whole reality.
It gives us a reading of reality based on the data it has reached, the way it was retrieved, the instructions given to it, and how it organized the results and presented them to us.
And this is a big difference.
What it means for you
Perhaps the most important question in the next phase is not:
Do we trust artificial intelligence?
In my view, the question is larger than that:
How does artificial intelligence create the image we see before we decide?
What data were they able to access?
What did they exclude?
How do I condense information?
How did they prioritize?
And are we really reviewing his recommendations, or have we, over time, become accustomed to accepting them?
Because AI's real impact will not begin when we let it press the "Approve" button.
It may start much earlier than that.
It starts when they become the one who searches for us.
Summarize it for us.
and arranges for us.
and chooses what deserves our attention
And then perhaps the most dangerous question is not:
Did artificial intelligence make the decision?
Rather:
How much of the decision had already formed before it reached the human?