I Thought My AI Was Learning Me, Until I Opened Five Weeks of Logs

For: people who use AI agents, PKM, and vibe-coding  ·  Format: build log  ·  2026-07-07

Image: No matter how much you file away, a drawer that never reopens is the same as empty.

The week Fable came back, and one prompt

Fable does not stay in the subscription for long. It is a brief window where you can use the smartest model. Daniel Miessler said to spend that window like this: point it at the things that need the strongest intelligence and that compound across every future session.

Top of his list was "audit your harness." Check whether your AI setup actually reflects who you are now. As it happened, I had spent months growing a personal knowledge vault and an auto-memory. Pinning rules, capturing lessons, rebuilding indexes. I believed it was running well.

So I fed that window this prompt: "Dig through everything my harness remembers. Look at how it learns across sessions. Find where knowledge dies, where it was captured but never resurfaces, or was never captured at all. Then judge whether it is actually getting smarter about me, or just accumulating."

The word compound stuck with me. I was adding knowledge every session. But was it carrying over to the next session and earning interest, or was it just a bank balance that kept growing? I had to open the logs to know.

The window to use the smartest model opens only briefly. Where do you point that brief moment?

Accumulation was fast, learning ran in a single track

I set loose a set of audit agents to sweep 53 sessions across five weeks, hundreds of knowledge files, the memories, and the automation scripts all at once. Each counted, from a different angle, what actually got reopened. The picture that came back was colder than I expected.

Only three things were really in circulation. The rule file injected automatically every session, five or six blog-writing skills, and two GTM reference docs I touch often. Everything else went nearly untouched.

Of the 215 reference docs, 96% were never opened once in five weeks. 26 of the 28 marketing skills, and the engineering and thinking skills entirely, saw zero. By size, 98% of the active knowledge sat behind a fragile path that opens only "when the AI happens to go find it," and there was not a single mechanism forcing that path.

The accumulation rate was excellent. In June alone, 33 memories grew to add 21. The problem was that it only grew. Nothing retired, nothing disappeared. The balance swelled, but the structure barely paid interest.

Of 215 reference docs, only 9 were ever opened. The stacks are wide, but there is just one trail of footprints.

The disease was not a lack of capture

At first I thought, "should I capture lessons more diligently?" No. The deeper I dug into the logs, the more the real cause turned out to be the opposite: capture was overflowing, and two things were missing.

First, none of the knowledge had state. It was only ever born, never aging, never dying. So a rule I killed in June was still alive in July, butting into every session. New and old, living and overturned, all got the same treatment.

Second, nobody measured whether it was read. The detection tools counted "what did I newly capture," but "was that knowledge actually reused" went unmeasured by any tool. I knew when a file was modified, but nobody knew when it was read. If you do not know reads, you do not know what died. And then you cannot throw anything away.

The two were interlocked. With no measurement, there was no basis to design decay, and with no decay, state stayed mashed into one and just piled up. No amount of extra capture fills these two holes. It only makes the grave bigger.

The one thing that was alive showed the answer

Yet exactly one track was truly learning. The blog pipeline. Over the past few weeks, every time I wrote a post a piece of feedback came back, and each time a rule was born. The orange accent color, mobile paragraph length, no fake first person, no unmeasured ratios.

The interesting part is that when a rule lived only as a document, the same mistake happened again in the next post. The rule existed but did not surface in that moment. The friction repeated, and in the end I built a checker called blog_check.py that automatically catches it before publishing. After that, the mistake stopped.

A formula was hiding here. Keep the rule in a document, and hand enforcement to a machine. A rule that leans on human discipline eventually leaks; only a rule the machine blocks does not. The one pipeline that stayed alive was already proving the design that all the dying rest should follow.

Document rules blow away; the machine gate stays shut. That difference decided whether mistakes recurred.

So I counted the scars first

Before designing anything, I marked four places where the system was actually bleeding. All four had evidence in the logs.

One. A switch built but never turned on. A few days earlier I had written a hook that "rebuilds the index automatically whenever a vault file is edited." It was the newest file in the hook folder. But it was not wired into the settings anywhere. Finished but never connected. The result was that very day's "stale index" warning.

Two. The ghost of a rule that died six days ago. On June 24 I had decided to stop using hand-drawn diagrams, yet an old memory telling me to use them survived with no retirement, quietly injected into every session. A new decision and an old one were negating each other inside the same context.

Three. A detector that false-flags itself. The script that hunts for friction flagged its own retrospective-analysis prompt as a "strong complaint." Like looking in a mirror and being startled by a stranger. The real signal was buried under noise it made itself.

Four. Two loops that cannot see each other. A problem the fast path (rule-ify on friction) fixed in two days was re-proposed three weeks later by the slow safety net (the monthly retrospective) as a "new problem." Because the safety net could not see the fast path's work, it told me to fix what was already fixed. When two memories do not know each other, you get this kind of wasted effort.

Wire resurfacing first

Once the diagnosis was done, the design got simple. Do not make more capture. Instead, lay down the path back to the surface and the thing that measures whether it was read first. There are three cores.

First, the promotion ladder. A lesson does not become a rule in one shot. It starts on the bottom rung, observation (record only); repeat it twice and it becomes an on-demand rule; three times and it becomes a rule injected every session; get it wrong even then and it rises to a gate the machine enforces. The principle is one thing. Recurrence pushes it up. If a rule exists and you still got it wrong, the answer is not reworded text but a surface one level higher.

Second, the supersession protocol. When a new rule overturns an old one, you do not leave the old one lying around. The new file states "what it replaces," and the old file keeps only a retirement mark and a tombstone with its body cleared out. It is removed from the index too. These three moves have to run as one set, or ghosts appear.

Third, the read log. Without it, everything else is a guess. Every time a file opens, it quietly leaves one line of record. Only once this log accumulates can you clear out "a doc not read once in 90 days" with real evidence. That measurement is the prerequisite for decay is not a metaphor, it is the order of execution.

The promotion ladder. When a rule breaks again, you do not reword it, you move it up a rung.

What I actually did in a day

If a design lives only as a document, it becomes an unread grave too. So I put my hands on it the same day.

I wired three enforcement hooks. A hook that rebuilds the index automatically when a file is edited (I finally turned on that unfinished switch), a usage-log hook that records every time knowledge is opened, and a resurfacing router that reads the prompt and automatically pulls the relevant skills and project docs into context. That last one is the core. It swaps the hope that "the AI will go find it" for a machine that shoves it in whenever a keyword matches.

I cut reference docs from 215 to 74. Most were dumps carried over whole from an old note app, finished to-do lists, old brand research. Half of the discoverability problem is solved not by search but by shrinking the denominator. The stacks have to be a scannable size for the index to act like an index. Nothing was deleted. It was all moved to the archive with a recovery manifest left behind.

I cut memories from 33 to 22. I retired 15 that were plain duplicates of vault docs, and rewrote the always-loaded summary into two tiers, core and cluster. That ghost rule got a proper tombstone at this point too.

The important part is that what to delete and what to demote all went through a human approval gate. Once an AI starts pinning or deleting rules on its own, wrong rules get reinforced too. The machine goes as far as proposing candidates with evidence; the decision is the human's.

Three hooks wired, docs from 215 to 74, memories from 33 to 22. Not a day of adding, but a day of connecting and cutting.

Verification left scars too

It would have been a clean story if it ended here, but it did not.

The orchestration that ran the big cleanup died twice. Once from passing an already-executed value instead of a function into a parallel run, once from an input arriving as a string rather than an object. Both times I read the logs, pinned the cause, fixed it, and reattached. Automation is decided less by writing it than by confirming it actually runs.

The one that stung more was this. While verifying whether the new resurfacing router worked in a real session, I judged it "not injected." But when I reopened the raw log, the router had worked fine, and what was wrong was how I had parsed the log. The verification needed verifying.

That passage is a miniature of this whole post. Believing "it is done" and confirming by measurement are different things. Setting a schedule does not mean it runs, writing a rule does not mean it is followed, and glancing at a log does not mean you read it right. So the next completion bar for this system is the moment I watch, with my own eyes, the next session run the hooks automatically and the usage log accumulate on its own. That is not finished yet. Honestly, it is not.

If you are building a knowledge system

This is a personal-vault story, but the same trap waits for anyone who bolts memory onto an AI. A note app, a company wiki, an agent memory. The history part ends here; now hold it up to your own system.

If you keep only one thing, keep this.

Knowledge is not something you stack up, it is something you make resurface. If it is not read, it is the same as never filed.

Sources

• Source of the opening prompt: Daniel Miessler, Prompts to Run When Fable Comes Back (the framing that when top-model access is temporary you should spend it on meta-work that compounds, and the harness-audit item)
• Source of the figures: a hands-on audit of 53 session logs across five weeks from the author's personal knowledge vault (2026-07-07), and the cleanup record

The figures in this post (resurfacing 15% and 4%, docs 215 to 74, memories 33 to 22, and so on) are values measured and tallied directly in the author's system. The improvement effect after cleanup has not been measured yet, and that is stated honestly in the body.