This is an AI club, so when we decided our website needed a better design, it seemed only appropriate to ask ChatGPT to redesign it.  That was very helpful, but also made me face what skills I need to hone to work well with AI.

Chat produced a lot of good ideas. And, when I asked, a long and detailed set of implementation instructions. Many of which were wrong.

The errors were usually mundane rather than bizarre. ChatGPT would tell me confidently to “change X in menu Y,” but menu Y might not even exist, or X was not there. My guess is that it was conflating different versions and revisions of WordPress, the open-source software we use for the site.

Bob had given me a useful piece of advice: instead of trying to describe what I was seeing, take a screenshot of the actual screen and give that to the AI. I did this repeatedly. Often, though not always, the screenshot gave ChatGPT enough concrete evidence to correct itself.

The experience made me think about an increasingly important question: what skills should we develop to complement AI?

Perhaps AI will eventually put us all out of work, leaving us to live lives of idle leisure on whatever income Sam Altman and Elon Musk decide is appropriate. Or perhaps we will become the copper tops of The Matrix, supplying energy to our machine overlords. But another possibility is much less dramatic: we humans will adapt what we do to complement the technologies we invent, as we have been doing repeatedly since the Stone Age.

AI may be different in degree, but perhaps not in kind.

So, what is it we should be trying to learn to do?

1. Think of AI as a smart, opinionated friend

I have a friend who is right a surprisingly large fraction of the time, but is sure he is right an even larger fraction of the time. Most of what he tells me is useful and well grounded. But I have learned to be careful, because some things he says confidently turn out to be wrong, incomplete, or overstated in ways that matter.

AI is often like that. Confidence is not evidence.

2. Trade off consequence against cost of verification

Not every AI recommendation deserves the same level of scrutiny.

Some decisions have little consequence—for example, choosing a particular shade of off-white for a website background. Others matter much more—such as installing a WordPress plugin that is no longer maintained and so may surface incompatibilities or, even worse, security vulnerabilities.

Verification also has a cost. Some claims can be checked in seconds. Others may require testing, reading documentation, or cross checking with others.

Putting this into the usual 2×2 matrix format:

	Easy to verify	Hard to verify
Low consequence	Verify casually	Often accept provisionally
High consequence	Always verify	Invest heavily, seek expert help, or find an alternative safer/easier-to-test approach

3. Build verification skills and make them as cheap as possible

The new useful skill is not knowing every answer, but knowing how to test an AI answer quickly and appropriately.

Verification candidates include:

• Force specificity. Always verify the software version, product generation, jurisdiction, or policy date. Ask for exact menu names, version numbers, dates, model numbers, commands, or documentation sections. Vague answers can sound convincing while remaining untestable. Many technical errors are not “hallucinations” so much as stale instructions.
• Show the AI the actual evidence. Upload screenshots, error messages, configuration pages, source text, or data instead of paraphrasing them. This reduces ambiguity and prevents the model from inventing the interface you “must” be seeing.
• Ask for the source of the claim. For anything factual, current, technical, legal, medical, or financial, ask where the answer comes from. Prefer official documentation, primary sources, standards, or original research over secondary summaries.
• Ask it to verify itself. A useful follow-up is: “Check whether this is still true for the current version,” or “What assumptions are you making that could be wrong?” This often exposes version confusion or hidden assumptions.
• Ask for confidence and disconfirming evidence. Try: “What would make this recommendation wrong?” or “What are the strongest reasons not to do this?” This counters the tendency of AI to continue defending its first answer.   Ask for confidence levels, alternatives, or which parts of the answer are least certain. The numerical confidence itself may not be calibrated, but the exercise often reveals weak spots.
• Use sanity checks. Ask whether the answer is plausible before verifying details. Does the magnitude make sense? Is the sequence of steps internally consistent? Would this command obviously destroy data? Would this recommendation contradict something you already know?
• Test cheaply before committing. Use a staging site, sample file, dummy account, test transaction, small dataset, or reversible setting change before applying the recommendation broadly. When several approaches exist, choose the one that is easiest to undo. Backups, checkpoints, snapshots, and export files are forms of verification insurance.
• Cross-check with a second independent source. This can be another AI model, official documentation, a knowledgeable person, or a reliable reference. Independence matters: two systems repeating the same bad source is not much of a check.
• After acting, verify the outcome. Do not assume success because no error appeared. Check the logs, test the result, inspect the output, or confirm the expected state actually changed.

Towards a practice

We are mostly boomers who are no longer the facile adopters of technology that we once were.  But we can be intentional about building AI-complementary skills.  In coming posts, I am going to work to develop (or borrow or point at others’) exercises that will develop the skills described above and more.   With practice comes familiarity, competence, and even habit, things that can serve us well.