Why did I tear down and rewrite my own AI Agent?
A rewrite made me re-understand state, context, and system boundaries.
Originally written in Chinese. Translated by AI.

When I first got the Agent's main pipeline running end to end, I thought the hardest part was over.
The model could be called, data could be persisted, vectors could be retrieved, and the workflow could run smoothly to the end. For someone building a complete project with LangGraph for the first time, that was enough to keep me excited for a while.
But once I actually started using it continuously, problems surfaced one after another: even though checkpoints were configured, the model in the next round didn't know what had happened in the previous round; the workflow was split very finely, yet the state was stuffed with intermediate artifacts that didn't need to be saved; quite a bit of information was stored in the database, but the context the model received wasn't necessarily useful.
What was even more painful was that whenever product requirements changed, I always had to first explain why the existing architecture didn't really support this requirement, and then find a way to force it in.
Eventually, I decided to rewrite.
For the sake of discussion, I'll omit the project's specific business below and treat it as a conversational Agent that needs to continuously accumulate context and support multi-turn interaction. This article also doesn't intend to list the tech stack again; I'd rather talk about the places that really got me stuck, and how these problems changed my understanding of Agent development.
1. I treated workflow state as the model's memory
The earliest architecture was a fairly long Pipeline: load business objects, organize basic materials, build the retrieval query, recall information, organize context, and finally call the model.
To make it convenient to pass data between nodes, I put almost everything produced along the way into the Graph State. Query results, retrieval queries, half-organized context, and even SQLAlchemy ORM instances all got stuffed in.
My understanding at the time was simple: isn't State just data shared across nodes? Put it in, and the next node can grab it directly—so convenient.
Then I integrated checkpoints, and the problems arrived.
First was serialization. ORM instances couldn't be handed directly to the checkpoint serialization pipeline at the time like ordinary dictionaries. Second was the scope of persistence: a lot of temporary data that was only useful for the current execution got saved along with the state.
My first fix was to clear the intermediate fields in the last node. Only later did I realize that checkpoints record state during execution, and clearing once at the end cannot eliminate the snapshots already written earlier.
But this wasn't the biggest pitfall yet.
To support continuous follow-up questions, I wanted the second round to directly reuse the context saved in the first round, saving a time-consuming construction process. I configured the same thread_id and called the Graph again, but the model still acted as if it had never seen the previous round.
Looking carefully at the pipeline, the reason was actually quite straightforward: the state was restored, but when I called the model, I only constructed a new HumanMessage. The historical information was lying in the State and was not organized into this request, so naturally the model couldn't see it.
Even more absurdly, in the second round's input, I also passed an empty string to the context field. I originally meant to express "don't recompute this round," but the actual effect was overwriting the existing state with an empty value...
Only after this did I truly separate several things:
| Data | Its responsibility in the system |
|---|---|
| Temporary variables inside a node | Complete the current step, then discard when done |
| Graph State | Execution state passed between nodes and restored as needed |
| Message history | Conversation content that needs to be retained across multi-turn interaction |
| This round's model input | The information the current task actually needs to consume |
Checkpoints can help restore execution state, but which content enters the model input still needs to be designed by me. Conversely, a large State doesn't mean every request will necessarily consume a lot of tokens; only when this content is sent to the model does it become input cost.
This distinction looks very basic, but I really did have to stumble once before understanding it.
I also reconsidered how to divide nodes. Before, I split almost one node per function, and later I went to the other extreme and stuffed the entire pipeline back into one node. Only after continuing did I realize that node boundaries should follow execution needs: steps that need independent observation, branch routing, or parallel scheduling are worth splitting out; transformation logic used only locally is fine as an ordinary function.

2. Sometimes the model processes slowly because I gave it too hard a problem
There was an early input pipeline that needed to recover structured records from screenshots.
I asked the model to identify the speaker, content, and time of each message, and then organize them into strict JSON. To improve accuracy, I also added quite a few restrictions for users: upload screenshots in order, include time, limit the number of images, and so on.
The results were not ideal. In one attempt, two screenshots took about two minutes to process, and there were still confused times, misaligned speakers, and invalid content.
My first reaction, of course, was to keep tuning the prompt. But looking again at the final use, I suddenly realized that I didn't need to recover the original records one by one without missing anything.
What the downstream really needed was: what topic this material revolves around, which information is worth retaining, and in what context they occurred.
So I adjusted the data granularity from "item-by-item original records" to "information organized around a topic." The model no longer had to take on the task of fully reconstructing records, and subsequent retrieval no longer had to first reassemble semantics from a large number of scattered messages.
Of course, this approach also had a cost at the time: I required users to submit materials that belonged to the same topic as much as possible in a single submission, which amounted to handing part of the segmentation work back to users. It reduced processing burden, but it couldn't yet be considered a complete solution to the input experience.
This incident got me into the habit of asking two questions before optimizing the model: what information does the final feature actually need? Is the granularity I'm currently asking the model to output beyond the actual use?
Just because the database can hold raw records doesn't mean the model must reconstruct them item by item; just because the model can output a complex structure doesn't mean the product needs users to wait for it to fill out that structure.
3. The difficulty of context shifted from "saving it" to "how to use it this round"
When I first worked on memory, my attention was mainly on the write side: how to extract materials, how to model data, how to vectorize text.
Later, when I encountered some approaches that organize context with Markdown, I found that they had a lot worth learning in information classification and expression. But among the approaches I tried, some files were required reading every round, while other files were loaded in full on demand.
When there was little material, this was very direct. As the material gradually increased, I began to worry that each time only a small part of the information was needed, yet the entire file had to be loaded first.
So during the refactor, I kept the basic materials and split finer information into independent entries to support later retrieval by question. Here, the database is responsible for managing entry ownership, classification, and time, while the text content is responsible for expressing the semantics the model needs to understand.
But with vector retrieval, the problems didn't automatically disappear.
Early recall mainly relied on semantic distance. Later I gradually added business scope filtering, importance weights, and time decay for some information. Because "very similar to the question" and "suitable for the current answer" are not the same thing.
Especially time.
I once only put a "current time" in the system prompt, and only later realized that it couldn't indicate when the historical messages each occurred. Looking further upstream, even if the database has creation times, if the extracted text loses the year, period, or stage from the original material, the model can still easily mix up similar things that happened at different times.
The later adjustments were divided into two parts: conversation messages explicitly carry timestamps; when extracting information, time anchors that can be confirmed in the material are retained in the body text, and times that cannot be determined are not filled in.
This is more upstream than simply adjusting the ranking formula. If time information is lost during extraction, it is very hard to recover it out of thin air at the retrieval stage.
In addition, I also tried encapsulating knowledge recall as a Tool, letting the model decide when it needs to supplement materials, avoiding mechanically running the full retrieval set every round.
But Tool Call exposed another boundary: the model cannot reliably generate the ID of the current business object. This ID is clearly already in the system runtime context, yet I still asked the model to guess it again.
In the end, I inject parameters already determined in the current context before executing the tool. The model can decide whether retrieval is needed and what to retrieve; as for who the current user is, which object is being operated on, and whether they have permission to access it, these should be determined and verified by the program.
After working through these problems, my understanding of context gradually shifted from "collect as much as possible" to: retain the necessary information for the current task and let it enter the model in an appropriate form.
As for exactly how much each change improved the results, I don't have sufficiently systematic evaluation data, and I don't want to make up a percentage here. What I can confirm is that I began to know which pipeline to investigate along, instead of only staring at the final answer and tuning prompts.
4. Splitting directories doesn't mean real decoupling
In past web projects, I was very used to writing the backend first, then wrapping HTTP interfaces, and finally having the frontend call them. This Agent initially followed a similar structure, with business capabilities placed very close to the web service.
Later, the product mainly interacted through IM, and I also added a CLI for configuration, management, and diagnostic operations. By then, it felt awkward for the business layer to continue depending on the web framework.
So I split out the Service to carry business and data access capabilities, and then had different entry points call it.
That sounds decoupled already, right? Reality punched me again.
Once, when running the CLI's help command, the output turned out to be the web framework's help.
The reason was that the command module imported the Service, and the Service imported the web framework's Request type at the top level. It was originally just a type annotation, but at runtime it brought in the entire framework and affected command-line argument parsing.
The fix wasn't complicated: move pure type dependencies under TYPE_CHECKING, keep the command registration stage as lightweight as possible, and load necessary modules only when actually executing.
But this small incident made me realize that dependency arrows on an architecture diagram ultimately have to be implemented in module imports and initialization behavior. A Service that looks very clean, if importing it brings up the framework, reads arguments, or starts external connections, will still cause problems when called from another entry point.
Later, to simplify user configuration, I tried moving some data capabilities to the remote side. Places that previously took ORM objects had to be changed to consume explicit return structures; database operations scattered in entry points and the Graph also had to continue being consolidated into the Service.
I added a unified dispatch layer: local mode calls the Service, remote mode calls the corresponding HTTP API. Serialization is also handled at this boundary, avoiding enum, time object, and optional parameter conversions scattered everywhere.
For example, a missing query parameter: passing None in a Python function is very natural, but when it reaches HTTP query encoding it may directly error. The fix should be contained at the request boundary while preserving values with actual meaning such as 0 and False; nor should an explicit null in a JSON request body be casually deleted along with it.
These are not complex algorithms, but they determine whether "a function that can be called locally" can become a reliable remote interface.
5. If users configure one less database, the system takes on one more set of responsibilities
Building the CLI and IM entry points was initially to reduce the amount of independent frontend development and let users interact in familiar places.
But when I strung the installation process together, I found that I had only saved the frontend; users still had to prepare a database, model configuration, and bot configuration. Code running successfully and users being willing to use it are still a long way apart.
So I tried to have the server side take over database access, reducing the burden on users to set up their own databases.
The first constraint was clear: convenience cannot justify sending shared database connection credentials to the client. Ordinary data operations can be handled through authenticated APIs, but workflow checkpoints depend on database connections, and they won't automatically gain remote storage capability just because I added HTTP routes.
The approach at the time was to keep two behaviors: local mode continued to use PostgreSQL to persist checkpoints; the experimental shared mode switched to process memory.
The cost was also clear: after a process restart, this part of the short-term memory in shared mode would be lost.
I didn't completely solve this problem; I just accepted a degradation in exchange for a runtime approach that doesn't expose database connections to the client. It also doesn't mean checkpoints can never be remoteized; it's just that the dispatch layer at the time couldn't cover it, and the cost of continuing to abstract had exceeded the goal of that round.
More troublesome problems were still ahead.
Some Services that look like "data synchronization" actually call the model internally. When moving them to the server side, whose model configuration should be used? How should the user's configuration be passed? Who bears the calling cost?
Previously, on a single machine, the database, model configuration, and execution environment were all in the same place, so these questions were easy to ignore. After crossing the network, every one of them has to be answered again.
The same goes for login state. To reduce background tasks being interrupted by token expiration, I once adopted an approach with no expiration time. But the rotation, revocation, and invalidation governance problems it leaves behind still exist, and I can't say authentication is fully designed just because the main pipeline continues to run.
In the end, although some pipelines in shared mode had already been connected, I still hid the public configuration entry. If the ownership of model configuration and the boundary of server-side proxy execution are not handled cleanly, users should not be allowed to treat it as an option that is already ready.
This was also a shift for me at the product level: lowering usage cost means someone has to bear the complexity that was originally handed to users. If users no longer need to configure a database, the server side has to take on the corresponding permissions, configuration, and runtime responsibilities. An "Easy Mode" is far more than just showing a few fewer input boxes.

6. A prompt service outage made me rethink runtime dependencies
There was another lesson that had almost nothing to do with model capability.
To make tuning convenient, I put prompts on an external platform and pulled them in real time at runtime. Modifying prompts didn't require redeploying, and the development experience was indeed very good.
Then the platform service failed.
I had neither kept a local backup nor designed a fallback for read failures, and all pipelines involving prompts immediately went down. The external service recovered after about two days, but those two days were enough to make me realize: I had turned a development convenience tool into a hard runtime dependency.
The lesson this incident gave me was very concrete. Although a prompt is a piece of text, it carries actual execution logic in the system; its version, backup, and availability all need to be taken seriously. Managing it remotely does not make these responsibilities disappear.
Later in the development process, I also began to distill recurring collaboration requirements into Skills: running quality checks after changes, recording changes, syncing Graph documentation, and writing corrections back into the rules.
These procedures cannot replace testing, nor can they guarantee that the Agent executes correctly every time. But at least, when the same kind of omission occurs again, I have a chance to modify the rules that are continuously consumed, rather than just giving another reminder.
Writing to the end
Looking back, this rewrite was not completed all at once on some day when I figured out all the problems and then drew a new architecture diagram.
I first misunderstood checkpoint, then redesigned the state; first pursued fine-grained input, then returned to the semantics that downstream actually needs; first split the business layer and deleted unnecessary entry points, and later added back the HTTP service for the new deployment method. New problems kept emerging, and choices I had made earlier were re-examined.
I no longer agonize over "whether I chose the wrong framework back then." What really made me take a detour was failing to clarify some more fundamental boundaries: what needs to be persisted, what needs to enter the model, which parameters are decided by the program, and what responsibilities get carried along when a capability changes its runtime location.
Now when I start writing a new feature, I spend a little more time than before asking these questions. They may not let me get it right the first time, but they let me write less code that looks finished yet has to be torn down again in a different scenario.
As for this rewrite, the result I am most willing to keep is probably these questions that will continue to influence how I write code later.