In a previous post, I asked “what skills should we develop to complement AI.”   I discovered – a little painfully – one such skill:  

Learn how to manage your cost of AI. 

Or more broadly:

Learn how to manage the economics of AI-assisted work: deciding what context AI needs, how much autonomy to give it, what should be standardized, and where human judgment is worth inserting.

How I Got There – The Project Management Project

Even in retirement, I have much more I want to do than time to do it. It is a personality feature—or defect—but it is me.   I’ve always used time management hacks to help with that, but they themselves take time to create and administer.   So now I had a solution: AI would design and execute my time management, and I wouldn’t have to do anything beyond the occasional tweak or new idea!   So efficient, in so many ways.

So, I launched a big project with ChatGPT to “let AI manage my work.”    I asked it to add its own insights to the GTD and PARA frameworks that I’m familiar with.   And off it went.  Over a couple of days, it built me an elaborate system coordinating an off-the-shelf app (Todoist) with a substantial file structure. Each project and area of interest had its own control documents and detailed documentation.   So many data fields and files, so much controlled vocabulary, so many ways to evaluate and adjust, all to be done for me by AI in the background.   And it worked.  Wow.

Then I ran out of tokens.  

That hadn’t happened before and, after waiting a few hours for my short term token allowance to replenish on my $20/month plan, I learned a lot more about usage metering and about what had happened.

ChatGPT caps my usage in both a five-hour window and a weekly window (though it will let me buy a quick refresh if I need it).  But I discovered that a few minutes of heavy use by ChatGPT can exhaust my 5 hour cap.   I asked what was going on – ChatGPT explained that our beautiful architecture required repeatedly interpreting and creating complex Word documents, rereading a large architectural description, reconciling Todoist with numerous description and status files, and rereading the context documents I had identified for each project, area, and program.   Well, that does sound like a lot. But before the experience I might have – and indeed did – assume it was a small fraction of what I am paying for.  It is not.

So, what did I learn?

AI will build you a system to do everything you ask and do it very well – even if that is big, complicated, and expensive (in tokens).   Or, more accurately: AI tends to optimize for the request you give it, not for constraints you forgot to state – like how costly its operations are.  If you care about those – and I know I do now – you need to include their consideration as you go along.

There are at least four key costs:

• Model cost/usage: how much AI computation you consume.
• Context cost: how much material the AI has to repeatedly ingest and reason over.
• Complexity cost: how many components, files, interfaces, and dependencies have to stay coordinated.
• Verification cost: how much human effort is required to determine whether the result is actually right.

The key is to optimize the whole problem, the AI costs across all these components and the cost of your own effort, rather than any component in isolation. That needs to be explicit both in your own thinking and in your instructions to AI.

File format is a special case.   Asking AI to repeatedly read, interpret, and write files in complicated or ambiguous formats (like PDFs and Word documents) is costly.   Simple, fast, consistent, easily parsed documents are better.   Well-structured, concise Markdown files (.md) seem to strike a good balance: enough formatting for human readability, but simple for AI to parse and create. They are:

• plain text;
• explicitly structured;
• easy for humans to edit;
• easy for software to parse;
• friendly to version control;
• low formatting overhead.

Indeed, ChatGPT confirms this choice itself – its own Skills system uses a Markdown file SKILL.md as the normal playbook for a skill.

After realizing I was going to need to go to the $200/month plan to pay for our overbuilt (but oh so easy on me) project planning implementation, I backed out to one that is a little more manual  for me but much, much less AI intensive. And once I knew that was what I wanted, I got a lot of good help from ChatGPT in designing it.  I’m using it now, very happy with the results, and staying well within my plan’s usage.

The takeaway: a sufficiently capable AI can make an inefficient process seem painless to the human because the AI absorbs the pain. But the complexity hasn’t disappeared. You’ve merely moved its cost.

Seeing the Sausage Making – Coding an Anki Card Generator

I learned more about working with – and the cost of – leveraging AI in another project: helping me create vocabulary review cards for learning a foreign language.  I like the free, open-source flash card program Anki, but it’s tedious for me to populate new cards.  I wanted not only to take away the grunt work, but also get much better cards, with good examples, grammatical explanations, and readings with native pronunciation.   And ChatGPT is happy – and good at – all of that.   It helped me build a Python-based app that takes a word I want to learn or conjugate, calls ChatGPT through an API to populate and verify the card, generates native-pronunciation audio through another API, assembles everything, and imports the finished card into Anki, where it synchronizes across my devices.

Since this was my first serious app work, rather than simply saying “go do it”, I asked ChatGPT to work with me step-by-step, validating progress at each step.  I told it, in effect, “Don’t be my agent, but rather tell me what you think we should do and let me do it.”

That turned out to be very enlightening.   I could see how the AI-generated software development behaves in practice:  how it goes about creating and validating code,  what works well and less well,   what seems cheap and what seems costly. 

My approach was incremental; I started with a relatively short explanation of what I wanted, worked to get it implemented and changed to suit my preference, then added something new, and then something more. We made solid progress.

But that same incremental process can lead to spaghetti code:  as I would add a feature or change a requirement,  we would add yet another conditional case or helper function,  bolted onto a growing collection.    Each change may be the easiest incremental solution, but repeated often enough, the architecture becomes fragile and inefficient. A human programmer might well fall into the same trap, but the experienced programmer knows to step back and reconsider the architecture. 

For example, my Anki app sends ChatGPT a prompt specifying how to construct each card: the kinds of examples I want, difficulty level, notes, grammatical constraints, and so forth. We started with this as embedded text in our Python program.  But as it grew – as we saw cases where the response wasn’t quite what we wanted –  we ran into bugs with character sequences that Python thinks of as special that were embedded in our text.  So, we added an escaping mechanism that would replace them after ingesting the text.  But over time the text got longer and longer and the escaping mechanism became more and more elaborate and error-prone.   I finally asked why we didn’t separate the prompt out into its own file, edit it in isolation when changing the LLM instructions for a card would make sense, and dispense with the whole escaping mechanism.  That is a better architecture, so we spent some more tokens refactoring to get there.  Without my intervention, I’m not sure that would ever have happened; we might have continued incrementally patching the embedded text indefinitely, leaving us with something buggy, difficult to maintain, and increasingly expensive for AI to work on.

Another phenomenon I saw is the remarkable ability of AI to work around obstacles.  AI will try one thing and, if stymied, come up with a different approach.  In effect, the agent explores alternatives:

• Maybe another package works.
• Maybe I can implement the function myself.
• Maybe there’s another command.
• Maybe I can transform the file first.
• Maybe I can use a different tool.
• Maybe I can do better if I get different permissions.

And, after a few attempts, it is often right.   But the attempts cost AI computation.   And it might well be that I either don’t really need that outcome badly, or I am willing to do something small myself to get to the same outcome.

Persistence is both a superpower and a cost center.

It’s worth telling an agent when you would rather be interrupted than have it repeatedly improvise around a constraint.

There is a bigger lesson here, not just about coding:  it is tempting – and often makes sense – to work iteratively with AI, first giving some general direction, seeing what it comes up with, then providing feedback on improvements.  But pure iteration may not be the right way to work on bigger projects. There are two problems:

• Thinking clearly about what you want up front—even knowing it will change—can reduce both the number and cost of iterations.
• It also helps you be more specific about how you will know you got what you want.   AI can produce extensive and effective tests, especially when asked, but it is testing what it thinks you want. You still need to engage in meta-testing: Are these the right tests? How will you know the system actually delivers what you intended? What, if anything, still requires independent human testing?

That doesn’t mean it’s smart to produce many pages of detailed specifications and tests, but rather that it’s likely worth stepping back and doing some more work before unleashing an AI agent.   In particular:

• Architecture, interfaces, data structures, security assumptions, naming conventions, file locations and acceptance criteria are often worth deciding early.
• Button wording, minor UI layout and features whose desirability you don’t yet understand may be better discovered iteratively.

After developing the basic Anki card creator with me deep in the loop, I decided to put it behind a Windows GUI. This time I switched to agentic mode with Codex and had AI do nearly all the work. I wasn’t involved again until it handed me a bundled executable ready to try. It was astonishing: a great deal of useful code for very little effort on my part.

But wow, did it run through tokens. And by then I understood why.

On Skills

Like Claude, ChatGPT has skills.  After I moved to my much simpler project management system I asked ChatGPT whether some workflows I expected to repeat would make sense as skills (reviewing a project, creating a new one). It thought so.  To elaborate (or possibly repeat) the discussion between Keith and me on skills:

For our purposes, it is useful to think of modern AI systems as having two main layers:

• A large language model (LLM) that is a learned statistical model.  You can think of its training as a kind of lossy compression: enormous amounts of training data are distilled into a much smaller collection of numerical parameters that capture patterns and relationships in those data. It isn’t an archive from which the original training material can simply be reconstructed. But the remarkable part is generalization: the patterns it has learned allow it to make useful predictions and reason about an enormous range of combinations and situations it never explicitly encountered during training.

• An application that sits between the LLM and the user and supplies the environment in which that intelligence operates: context, instructions, memory, tools, files, permissions and mechanisms for taking actions.

So what is a skill? You can think of it as packaging reusable instructions, procedures, and resources so the application can give the LLM the right guidance and context for a recurring kind of work.  The motivation is similar to modularization in conventional software: make a useful procedure consistent, maintainable, and reusable. But instead of specifying a rigid sequence of machine operations, a skill provides a playbook that the LLM can apply with judgment.

Rather than:

“Here is a function. Given these exact inputs, execute these exact instructions.”

we get

“Here is how we do this kind of work. Apply this procedure intelligently to the particulars of this case.”

So, when does it make sense to create a skill?  After all, even creating the skill will itself cost you some tokens.  Here are criteria: a workflow is a good skill candidate when it is:

• Repeated — you’ll do it enough times to justify encoding it.
• Regular — there is a recognizable procedure that should usually be followed.
• Consequentially consistent — you care that important steps aren’t forgotten or drift over time.

Recommendations

For your first serious coding project, consider using AI as a programmer whose work you execute, step by step, asking it to explain its choices. Once you understand the process, switch to an agent that does most of the implementation and testing and hands you a mostly finished product to evaluate.

Periodically ask the AI not to add any new code, but to inspect the accumulated code for duplication, obsolete paths, unnecessary complexity, and opportunities to refactor.

Specify heavily where decisions are expensive to reverse; specify lightly where experimentation is cheap.

Consider using markdown (.md) files where you need documents that both you and AI are likely to read and modify repeatedly.

In ChatGPT’s personalization settings—the instructions that apply broadly across my interactions—I initially thought of adding a request “Let me know if I have just engaged in or seem to be heading towards an activity that is unusually expensive in terms of tokens consumed.”  But, I worked with ChatGPT to make this personalization more specific and ended up with this:

Help me develop an intuitive sense of the cost of different ways of working with AI.

When I ask for work that is likely to consume unusually large amounts of AI usage—especially long agentic workflows, repeated reading or rewriting of large files, extensive iterative coding/debugging, broad searches, or repeated processing of the same context—alert me before proceeding when practical. Briefly explain what is likely to make it expensive and, when there is a substantially cheaper approach, suggest it and explain the tradeoff.

Also, if work we have just completed appears to have consumed unusually large amounts of AI usage, point that out afterward. Briefly explain what aspects of the work were probably responsible and what I could do differently next time, if anything.

Do not interrupt ordinary or moderately sized work to save small amounts of usage. Optimize for good value rather than minimum usage: spending more AI resources is appropriate when it materially saves my time, improves the result, or avoids substantial manual work.  But also, if there is a small amount of manual work that would save a lot of AI resources, please suggest I consider that.  And if you are repeatedly trying to work around some constraint at the cost of AI resources, double-check with me whether I can live without the end result or recommend minor action on my part that will address the constraint.

Distinguish between one-time high usage that is probably worthwhile and workflow designs that create unnecessarily high recurring AI usage. I particularly want to learn to recognize the latter.

Consider turning a workflow into a skill when you expect to repeat it, it follows a reasonably regular procedure, and consistency matters.