In this post we will review how NOT to use AI agents for coding. I am creating this post since I've encountered too many cases of bad code created by AI agents due to people unaware of these issues.
We now have AI agents to due our job, all we do is create a prompt and everything is done for us by the great LLM mind and the tools supplied by the agent.
At least, that' what most managers think, but they are wrong.
An AI agent is a powerful tool that can reduce some tasks duration by order of magnitude. However it can also create an illusion of a great progress while you're actually causing damage in the long run.
Some people are telling: "I am not a software engineer, but I have create this project, and I can add features with Claude code". They are wrong.
Using an AI agent to create a production ready code that can be used and maintained for a long time requires a human supervisor who understand the requirements for such a solution. This might change in the future, but for now it is mandatory. The AI agent create many mistakes in the planning stage, in the architecture domain, and in the implementation phase. You need an experienced software engineer to get a good product.
In case all you want is a simple one time task done, or a proof of concept, the AI agent can do this for you. However anything else must have include experience of building a production solution that an AI agent still does not posses.
Some examples I've encountered are below.
You create a product that processes data path requests, and the AI agent is creating the code using Python.
Really? This could never hold for heavy load. Use a production ready language such as GO...
You run your product on an EC2 instance with multiple processes, but then you discover that some of the processes are terminating due to errors, so you ask AI to handle it, and it created long bash scripts to check the health of the processes and to restart them in case of failure.
Really? Use a kubernetes deployment with liveness and readiness probes...
You use AI agent to create the unit tests for your code, and it creates 10 unit tests each checking another aspect of the feature. You end up with 10000000 (add more zeros at your will) tests each checking a small aspect of the feature, duplicating most of the logic, each using a different approach to test the code.
Really? Add a tests infrastructure and combine similar tests flows into a single test...
You create an new microservice in the kubernetes deployment, reusing the project standards. You test run it locally and everything looks great, but then in production the service causes your entire deployment to be unresponsive. You try checking the logs but they are exploded with log records from the new service and you cannot find the problem.
Really? Choose the correct algorithm to perform your task. Use configurable limits to prevent resource .exhaustion.
Given all this, how should we use an AI agent?
Considering the current LLM maturity we should treat it as both professional and stupid. The risk in using AI agent is that it might take a wrong turn and hit a wall. Then it will build a bug hammer to break the wall. Then it will build a garbage truck to clear the leftovers.
We need to identify these wrong turns as soon as they start. This is done by coding near ~100% of the tasks using the AI agent, but use a small units of work and monitoring the changes. Instead of prompting: "I have a bug in this project that requests longer than 10K are not saved to the database, fix it!" split it to multiple steps. First ask the AI to identify the bug. Then think of the solution yourself (yes, actually use the brain muscle) and instruct the agent of the solution, like "In case the requests are longer than 10K, use a secondary table to save them. Start with the database schema changes only". Notice this cannot be done by a novice software engineer which is not aware of the system implications a solution alternatives. You can consult with the LLM, but again the LLM might provide the book-solution which is over complicated for your needs and maybe does not address the real problem (wrong turn again).
To sum:
AI agent is a great tool. Nowadays I can accomplish a super high throughput of tasks. To achieve this, it is important to understand the advantages and the disadvantages of the LLM.
No comments:
Post a Comment