How I Trained My AI to Actually Think Like an Analyst

I got curious about how my own brain works.

Not in a philosophical way, in a practical one. I wanted to understand what was actually happening when I dug into a client report and started pulling on threads. What moves was I making? In what order? And could I teach an AI to do the same thing?

So I did something a little meta: I worked with Claude in real time while analyzing an actual client report, narrating my thought process out loud as I went. What came out of it surprised me — not because the AI was impressive, but because the process forced me to articulate something I’d been doing on instinct for years.

Here’s the model I landed on, and how you can use it.

First: Analysis Is Not What Most People Think It Is

Before we get into the how, let’s settle the what.

Analysis is not a dashboard. It’s not a report.

Data alone is not insight. Describing what happened is not analysis. And this is exactly where most AI falls flat.

Ask a model to “analyze this data” and it will describe the data. It’ll surface numbers, use words like “notable” and “significant,” maybe build a chart. But it won’t interrogate the data. It won’t compete narratives against each other. It won’t push back on its own conclusions.

That’s because AI is naturally good at summarizing and weak at questioning. It’s a predictive system, not a skeptical one — and analysis requires skepticism.

The Layers Most People Collapse Together

One of the most common failure modes in analysis, human or AI, is treating assumptions as facts. Before anything else, it helps to be explicit about what layer you’re working on.

An observation is a measurable fact. CTR dropped 22% last month.

An interpretation is a possible explanation. Search intent may no longer match the page.

A conclusion is the best-supported narrative given the available evidence. The page likely needs title and meta repositioning before rankings will convert.

LLMs tend to collapse all three together, which is how you end up with confident-sounding reports that are actually just decorated assumptions. The discipline of keeping these layers separate is something you have to explicitly build into your prompting.

The Analytical Maneuvers

Through my real-time experiment, I landed on a core set of analytical moves. Think of them less as a checklist and more as a toolkit; you reach for different ones depending on what the data is telling you.

Drill — Go deeper into a number that looks interesting. Don’t accept the top-line metric. Ask what’s underneath it.

Reframe — Change the question entirely. Instead of “what are our top keywords?”, ask “which keywords have the most room to move?” Reframing often unlocks more useful answers than drilling alone.

Hypothesize — Form a possible explanation. Keep it explicit and testable.

Test — Pull the data that would confirm or deny the hypothesis. Don’t guess. Go get it.

Challenge — Push back on your own interpretation before anyone else does. This is where most analysis breaks down — people fall in love with their findings and stop interrogating them.

Quantify — Put an actual number on the claim. “A meaningful share of traffic is bots” is not analysis. “31% headline engagement rate vs. 65% in clean-source cohorts implies roughly 35% non-engaged noise” is. Hand-wavy language sounds analytical but commits to nothing testable.

Benchmark — Compare against something meaningful. A number without context isn’t a finding. Is 31.77% engagement low? Compared to what — industry average, last quarter, similar sites? Without the comparison, you don’t actually know.

Predict, then check — This is the most powerful validation move I know. Take your conclusion and derive a downstream prediction from it. If a referral source is truly high-intent, then visitors from that source should also show higher pages-per-session, more contact-page visits, and a higher return rate. Go check those. If they hold, your conclusion gets stronger. If they don’t, something’s off in the story.

Steel-man the opposite — Write the strongest possible case that your conclusion is wrong. Force it. If you can do it persuasively, your original conclusion isn’t done yet.

Determine confidence — Separate what you’ve proven from what’s a likely explanation from what’s still an open hypothesis. Don’t let everything collapse into the same register of certainty.

Reduce noise — Filter out bots, outliers, tracking errors, tiny sample sizes, and one-time anomalies before you build narratives on them. A lot of “insights” are just dirty data with a story attached.

What This Looks Like in Practice

Here’s a concrete walkthrough using organic search data — simple enough to follow, real enough to be useful.

You start by reframing: instead of “what are our best keywords?”, ask “which keywords gained impressions but lost clicks?” or “which keywords changed position most in the last 90 days?” These are more interesting questions — they point to opportunity and vulnerability, not just current performance.

Then you test: pull CTR by keyword, position changes, engagement rate by landing page. Filter out low-volume noise.

You hypothesize: these mid-rank, low-CTR keywords are underperforming because the metadata no longer matches search intent. The ranking is fine; the click signal is broken.

Then, and this is the step most people skip, you steel-man the opposite: maybe CTR dropped because rankings improved into lower-intent queries. You moved from page two to page one, but page one is full of informational results and you’re a transactional page. Go check the SERP. What are the other results showing?

Then you predict: if the metadata mismatch narrative is true, engagement on those pages should also be weak — visitors who do click should be bouncing at a higher rate. Go check that.

Now you have a real finding, not a description of what the data said.

The Four Failure Modes That Kill Good Analysis

When I went back through my work with Claude on that client report, I found four places where we — and by we, I mean I — fell short.

We didn’t benchmark. We called 31.77% engagement “low” without ever asking: low compared to what? Industry average? Last quarter? Similar agency sites? Without a comparison, a number is just a number.

We didn’t triangulate. We looked at one data source the whole time — GA4. The same questions could have been cross-checked against Cloudflare logs, Search Console, or email platform data. When two independent sources agree, confidence goes up. When they disagree, you learn something more valuable than either source alone would tell you.

We didn’t trend. One month is a snapshot, not a pattern. Almost every strategic recommendation worth making requires a pattern — is this improving or deteriorating? Is this a spike or a baseline shift? Single-period data can’t answer those questions, and most decisions require the answer.

We didn’t ask who the report was for. A report for an internal team can stay open and curious. A report for a CEO needs clear conclusions. A report for prospects needs polish and credibility. The audience determines what “done” looks like — and we defaulted to a generic posture that served no one particularly well.

Narratives Compete — That’s the Point

Here’s the thing I keep coming back to: the job of an analyst isn’t to find the insight. It’s to determine which narrative best explains the evidence — which means holding multiple competing explanations at once and eliminating them through testing.

A traffic decline could mean tracking broke. It could mean rankings dropped. It could mean user intent shifted. It could mean bot traffic inflated prior periods and the numbers are normalizing. It could be seasonal. The data alone doesn’t tell you which one. That’s what analysis is for.

The most useful output of a good analysis isn’t a chart. It’s a clear narrative — a defensible story about what’s actually happening and what to do about it. Some of the most common ones I see in marketing data:

Traffic quality is unclear. Volume looks fine but engagement is weak, direct traffic doesn’t convert, and session patterns look robotic. The goal is to separate real visitors from noise.

ROI is a black box. Activity is visible but value isn’t. Conversions are engagement events, not leads or revenue. There’s no source-to-close attribution. You can see what’s happening but not whether it matters.

Acquisition is working, retention isn’t. New users are arriving but not returning. Email lists are stagnant. The funnel has a leak somewhere after the first visit.

One channel is carrying everything. 80% of traffic or conversions come from a single source. The business is one algorithm change or one referral partner pulling out away from a significant problem.

Each of these is a story that leads directly to a different set of questions, tests, and recommendations. That’s what makes analysis worth doing.

A Better Analytical Loop

The conventional model you probably learned looks something like: Question → Prepare → Process → Analyze → Share → Act. That’s not wrong, but it’s too linear for how analysis actually works. Real analysis doubles back on itself constantly.

A more honest version of the loop:

Observe → Reframe → Hypothesize → Test → Quantify → Benchmark → Predict → Steel-man the opposite → Determine confidence → Decide the narrative → Recommend action → Re-test over time.

These don’t always happen in that order. Sometimes you get halfway through and a new reframe sends you back to the beginning. That’s not a failure — that’s the process working.

How to Train Your AI to Do This

The practical application is straightforward. Don’t prompt your model to “analyze this data.” Instead, give it the maneuvers explicitly and tell it which ones to use.

Something like: “Here’s the dataset. Reframe the core question, form two competing hypotheses, test each against the data, quantify your claims with specific numbers, and steel-man the strongest counterargument before reaching a conclusion.”

That’s a different kind of output. It’s slower, more rigorous, and far more useful.

More importantly — do this yourself first. Find a dataset you already understand and walk through each maneuver deliberately. Notice where you reach for vague language instead of a number. Notice where you skip the counterargument because you already like your conclusion. Notice when you’re treating a one-month snapshot as a trend.

That’s the exercise. Not because the AI needs to see you do it, but because the quality of what you get back is almost always a reflection of how precisely you’ve thought through the process yourself.

The model doesn’t need to be smarter. It needs better instructions — and those instructions start with you.