While building production-ready AI systems for large enterprises, I often see AI engineers, even experienced ones jump directly to Advanced RAG, multi-agent architectures, or complex agentic solutions as soon as they receive a problem statement.
However, what makes an AI engineer truly effective is not building the most sophisticated system possible. It is solving complex business problems using the simplest architecture that is reliable, manageable in production, and cost-efficient.
The more moving parts we introduce into a system, the more complex the architecture becomes. This complexity does not stop at development, it continues into deployment, monitoring, debugging, governance, and long-term maintenance. What initially looks like an innovative solution can quickly become an operational burden.
Over the last few years of building AI solutions in enterprise environments, I realized that many teams ask "What AI technology should we use?" before asking "Do we even need that level of complexity?"
To address this problem, I designed a simple yet practical decision framework consisting of 10 questions. By answering these questions in sequence, you can systematically determine whether your use case requires deterministic code, a prompt-only LLM, RAG, an agent, a multi-agent system, or a fully governed agentic platform.
The goal is simple: build the least complex system that can reliably solve the problem. In most cases, that approach leads to systems that are easier to maintain, cheaper to operate, and more reliable in production.
Full Decision Framework
The diagram above represents the complete decision framework described in this article.
By answering these ten questions sequentially, an AI engineer can determine the most appropriate architecture for a given problem statement, starting from simple deterministic solutions and progressing toward more advanced systems only when necessary.
The framework is designed to help engineers align system design with business requirements rather than technology trends.
Most importantly, it encourages building the simplest architecture that can reliably solve the problem, reducing production complexity, operational overhead, and long-term maintenance costs.
Let's now walk through each question in detail and understand the reasoning behind every decision point in the framework.
Q1 · Does this truly need AI / LLM?
Start with this question. It is very simple, yet extremely powerful.
Sometimes, simple rule-based approaches or statistical methods can solve a problem more efficiently than LLMs. Here, I am not saying that LLMs or AI may fail to solve the problem. What I am saying is that even if a problem can be solved using an LLM, it does not necessarily mean that an LLM is the right solution. The same problem might be solved using a rule-based engine written in Python.
A solution that is simple to manage in production is often better than an LLM-based solution. From a cost perspective as well, rule-based systems are usually much cheaper than LLM-based systems.
So, solving a problem using AI, LLMs, or any other tool is an art. But solving it efficiently is the true craft that AI engineers should possess.
If the answer to this question is Yes, and you genuinely need an AI solution to solve the problem, then we move to the next question.
Q2 · Does it need external / dynamic knowledge?
Let's say the problem statement genuinely requires an AI solution. The next question we should immediately ask is: Do we need any kind of external knowledge to solve this problem?
If we do not need external knowledge, then we can use a simple yet powerful prompt-only approach.
For example, assume we have a collection of documents and we want to classify whether a document contains regulatory information or not. These documents are relatively small in size, with each document containing a maximum of 500 words that can easily fit within the LLM's context window. In this case, a simple prompt template can solve the problem without requiring any external knowledge.
Now let's look at a different example.
Assume the problem statement is to answer customer questions based on the Terms and Conditions of a particular product. These Terms and Conditions are long documents that cannot fit into the LLM's context window. Here, we have a different challenge. The answer to the user's question depends on external domain knowledge contained within those documents. In such cases, we need to move to the next question.
The key takeaway here is simple: always ask whether a prompt-only solution is sufficient, or whether external knowledge is required to solve the problem.
Q3 · Is source knowledge structured or unstructured?
As per our example, the knowledge we need is unstructured. But first, let's see what happens if the knowledge we need is structured data stored in a database.
For example, what if the knowledge source is customer information, and we need to answer questions such as why a customer was charged a late fee or any other fee based on structured data present in a database. I am just making this example up, but in such cases we can use a simple Text-to-SQL approach, where the LLM generates a SQL query and retrieves the required information from the database to generate the response.
Now let's come back to our previous example.
If the problem statement requires Terms and Conditions documents to answer customer questions, then our data is unstructured. In such cases, we need a retrieval mechanism to search relevant information from the documents. This is where approaches such as vector databases or BM25 come into the picture.
Of course, this is required only when the entire document cannot be passed directly into the LLM because of context window limitations. If the document is too large, we need a search capability that can retrieve the most relevant information based on the user's query. To achieve that, we need techniques such as vector search, BM25, or a combination of both.
Since the knowledge we need in our example is unstructured, let's move on to the next question.
Q4 · Does it need multi-step reasoning or planning?
At this point, we know that the solution requires external knowledge and that the knowledge is unstructured in nature. The next natural question to ask is: Do you need multi-step reasoning or planning?
This question helps determine two important things:
- How powerful the LLM needs to be.
- How much complexity you need to introduce into your RAG pipeline.
If the answer is No, then we can design a simple, naive RAG pipeline that retrieves the required information in a single step based on the user's query and generates the response using a lightweight LLM. In many cases, this approach is sufficient and keeps the system simple and manageable.
However, if the answer is Yes, then the system needs reasoning and planning capabilities.
For example, based on the nature of the query, you may first need to retrieve information from a knowledge base and then call a database to fetch customer-specific information before generating the final response. If you look closely, the system is no longer performing a single retrieval step. It now needs to decide what actions to take, in what order, and when to use external tools.
Once planning and tool selection become part of the workflow, the complexity of the system increases. Therefore, if the answer to this question is Yes, we move on to the next question.
Q5 · Does it need to take actions / use tools?
Assume that our system is not taking any actions and is simply answering questions using information from the Terms and Conditions documents. In that case, we need an Advanced RAG system.
By Advanced RAG, I mean techniques such as query expansion, re-ranking models, and other retrieval optimizations that help improve answer quality and retrieval accuracy. While this approach can provide highly accurate results, it is also more complex to build and manage in production compared to a simple Naive RAG pipeline.
Now let's consider a different scenario.
If the system needs to call an external tool, then the answer to this question is Yes. For example, as discussed earlier, if we need to fetch customer information from a database before generating a response, then a tool call is involved.
Once a system needs to interact with external tools, APIs, databases, or other systems, it moves beyond traditional RAG. The LLM must now decide when to use a tool, which tool to use, and how to use the output from that tool to generate the final response.
If the answer to this question is Yes, then we move on to the next question.
Q6 · Are agent actions reversible / low-risk?
This is a very, very important question to ask before allowing an agent to make decisions or take actions on its own.
Are the actions that the agent is taking low-risk and easily reversible?
For example, answering questions, retrieving username information, providing OTP-related guidance, or handling similar low-risk tasks. In such cases, the impact of an incorrect response is relatively small, and we can rely much more on the agent's decision-making capability because of the low-risk nature of the solution.
However, if the agent is making high-risk decisions or performing actions that can cause financial loss, regulatory issues, reputational damage to the organization, or harm to a customer, then we need to be much more careful.
In such scenarios, we should move to the next question, which introduces one of the most important concepts in AI system design: Human-in-the-Loop (HITL).
The higher the risk, the stronger the need for human oversight before an action is executed.
Q7 · Does it need retrieval AND action together?
This question helps us determine whether the system needs only action execution or whether it needs both information retrieval and action execution as part of the same workflow.
If the answer is action only, then we can involve a Human-in-the-Loop (HITL) to review and validate the critical decisions taken by the agent before those actions are executed.
In scenarios where decisions are highly critical and carry significant risk, AI and humans should work in collaboration. The agent can assist by analyzing information, generating recommendations, or proposing actions, while the final decision remains under human supervision.
However, if the answer is both, meaning the system needs to retrieve information and then take actions based on that information, the workflow becomes more complex. The agent must first gather the right context, reason over it, and then decide what action to perform.
In such cases, we naturally move on to the next question.
Q8 · Does it need multiple specialized sub-agents?
This question helps determine whether your problem statement requires a single agent or multiple specialized agents working together.
For example, you may have one agent responsible for understanding fees applied to a customer, another agent focused on payment disputes, and a third agent handling account-related queries. Each agent is specialized in a particular domain and has its own expertise, tools, and responsibilities.
If your use case can be handled by a single agent that performs retrieval, reasoning, and action execution, then you can move forward with an Agentic RAG architecture. In many cases, a single well-designed agent is sufficient and significantly easier to build, monitor, and maintain.
However, if the problem statement requires multiple specialized agents that need to collaborate to solve the task, then the answer to this question is Yes.
In that case, we should move on to the next question.
Q9 · What is the latency tolerance?
This is another crucial question to ask when your solution requires multiple agents or sub-agents to solve the problem efficiently.
At this stage, we have a fairly complex system where multiple LLMs and SLMs are working together, often along with vector databases and external tools. As we continue adding more components to the system, latency becomes a very important consideration.
The reality is simple: the more moving parts we introduce into the architecture, the higher the overall response time becomes.
If your latency tolerance is high and the problem can be solved using batch processing or asynchronous workflows, then a batch-oriented multi-agent approach can work very well. In such cases, waiting a few minutes or even a few hours may be completely acceptable because the nature of the solution is asynchronous.
However, if your problem statement requires real-time responses, then latency becomes a hard constraint. Every additional agent, retrieval step, and tool call adds overhead that can impact the user experience.
If your use case requires a real-time solution, then we should move on to the final question.
Q10 · Governance & data sensitivity check
If the answer to this question is low sensitivity, then we can naturally choose a real-time multi-agent solution with proper fallback mechanisms in place. Here, designing fallbacks is extremely important. No matter how good the agents are, failures will happen, and the system should be able to recover gracefully when they do.
However, if the solution has high sensitivity, then you need the most complex architecture in this framework from both a production maintenance and cost perspective. In such cases, a fully governed real-time multi-agent system with Human-in-the-Loop (HITL) becomes necessary.
At this stage, the system typically includes governance controls, approval workflows, monitoring, audit trails, fallback mechanisms, and human oversight for critical decisions.
This is the most complex solution you can build, and it is also the hardest to maintain in production. In fact, the real challenge begins after deployment. Building the system is often the easier part; operating, monitoring, governing, and continuously improving it in production is where the real work starts.
Conclusion
This framework will guide you toward the right solution based on your problem statement.
Having said that, complexity should come from the requirements of the problem, not from the technology choices we make. The most expensive and difficult systems are often built when the use case is not aligned with the system design.
In such situations, engineers sometimes introduce additional layers of complexity to justify the cost and architecture. While these solutions may show promising results during a Proof of Concept (POC), many of them never make it to production.
On the other hand, when the use case is properly aligned with the system design, the solution naturally becomes more efficient, easier to explain, and easier to operate in production.
The purpose of this framework is not to push you toward RAG, agents, multi-agent systems, or Human-in-the-Loop (HITL) workflows. The purpose is to help you identify the simplest architecture that can reliably solve the problem.
A true AI engineer understands this distinction. Solving a problem with AI is important, but solving it with the right level of complexity is what separates good engineering from great engineering.
The goal is not to reach Question 10. The goal is to stop at the earliest point where the simplest architecture can reliably solve the problem.
If you want to discuss AI Engineering, RAG systems, Agentic AI, or LLM production systems, feel free to connect with me:
LinkedIn: https://www.linkedin.com/in/veer-khot-93177bab/
Website: veerkhot.com
← Back to Articles