I had started to write a series on the principles behind multi-agent organizations and their workflows, but I found that this was leading to a potential trap. When you watch a YouTuber integrate all sets of new tools to create a highly-automated workflow, it is enthralling. These kinds of explorations do bear some fruit; they also make it easier than ever to run very hard in the wrong direction. With multi-agent organizations, for example, there is the trap of falling back to pre-AI engineering instincts to optimize through structure.
In a post-AI landscape, forcing structure can be a waste of time, and a hindrance at worst. For example, I recently built a vector-database-driven semantic search on top of a large ebook corpus for a client. I decided it would be useful to extract the non-domain-specific layers of this semantic search into shared packages/skills to allow reusability.
I spent hours grinding out the semantic search architecture, and extracting it into shared packages will certainly speed up the time taken for agents to turn my prompts into architecture, as well as the time for me to make decisions to steer this effort.
However, I also caught myself wanting to extract more reusable packages from the domain-specific semantic search implementation. For instance, I thought about extracting a reusable package that handles the ingesting of a source and turning it into a set of queries that can be channeled into my search for evaluation and fine-tuning. After the thrill of extracting, this seemed to be valuable. I imposed a source-to-query architecture to reuse.
Herein lies the trap. I would be introducing another shared package to maintain, yet it would impose structure for something that LLMs can handle very easily on their own. Adding structure is existentially satisfying, but if it duplicates what an LLM would do on its own, the reusable layer, ironically, duplicates unnecessarily.
This is the meta principle needed in AI engineering: we need to judge when imposing structures (via shared packages, skills, etc.) adds real value to our agent-driven workflows versus when it good vibes extraction that actually duplicates.
There has always been the potential to create a hasty abstraction. However, in a pre-AI landscape, judgment was needed to associate essential functions in code that repeated across multiple instances. In a post-AI landscape, the judgment extends to when an LLM benefits from a reusable abstraction. This requires knowing what the essential functions of the LLM are, yet this requires evaluation that is often illusive.
Principles must emerge from experience; reusable abstractions must surface from real use cases. Abstractions become hasty when we our ideal diverges from the real. A hasty abstraction is simply an abstraction that doesn’t properly distinguish the essentially-shared vs. domain-specific functions when associating between particular instances.
If the reasoning of LLMs are black boxes, even amidst evolving insights, then the risk for hasty abstraction lies in our working knowledge of the essence of LLMs. If we assume too much of their reasoning, we miss out on the value of an intermediate skill/package extracted from other instances across out projects. If we assume too much, which is the more subtle trap, then we assume our intermediate structures is DRYing our workflows, when it reality it duplicates.
In the exploding landscape of AI, the engineer needs to stay in sync with the essence of LLMs which requires reasoning to lower-level technology than high-level software engineers are used to. AI engineering and AI philosophy go hand-in-hand.