Embrace scepticism and think more critically

Petra says: “Embrace healthy scepticism.

In a time where the SEO industry is drowning in new AI metrics, from AIO attribution to vector index presence, we need to understand that we don't have robust, standardised methods to validate these just yet.

When tools show you things like AI visibility, you should be asking: What's the sample size behind this? What's that number or percentage based on? How are these metrics defined? Is the tool actually using LLM training data, or are they reverse-engineering the attribution models?

Embracing healthy scepticism and using critical thinking isn't a new thing. It's not a new process; we’ve had to use it before, but this situation makes it crucial. Previously, when featured snippets appeared, we had to completely rethink how we evaluated our existing Google Search Console data, because it now meant something different because of the change, and it was in a new context. We’ve had to ask these questions, and not just chase new metrics, but evaluate the data that we are basing decisions on.

Part of the problem now is that people are trying to look at these shiny new metrics as something that they can base decisions on, when they actually might change in a month or two. Everyone's still collecting data, looking at that attribution model, and trying to learn how these LLMs understand your websites, do vector embeddings, understand context, and serve answers – and the models change all the time as well, so that plays a part.

Before you bet your credibility on a shiny new dashboard, you need to ask yourself: What would you actually do differently with this data? Do you have any other way to validate this from something that you already trust and know to be proven?

If you can't answer those questions, you are just chasing the hype instead of thinking critically.”

What's a healthy amount of scepticism to have? How do you know when to stop questioning things?

“It will depend on a case-by-case basis.

I also think we should challenge the idea that data doesn't lie. People like to say that, and it's not true, in my opinion. Data does lie because data only means something when it's interpreted by human beings, in a specific context – and a lot of that can be very easily misinterpreted. Therefore, data often lies.

With data, we need to understand how it was collected and in what context. Can we only make conclusions within that specific context, or is it generalisable?

The key point is to build processes that we can trust. Again, when featured snippets appeared, we had to rethink what Google Search Console data meant. It’s a process of evaluating the data that exists already, which we used to make decisions on, and we trusted because it was proven and provided results.

We can use that as a starting step. How is this changing now? What does this mean? People are seeing top-of-funnel data disappearing. What does that mean for the bottom line? Why is that happening? Are people having a lot of that conversation on LLMs? Is this a problem?

Looking at the data you're familiar with and have already used is a great starting point, and then you can see whether the unproven metrics fit into that

Obviously, be aware of confirmation bias. You're not chasing an AI metric to fit your narrative, but you need to understand the level of robustness that these metrics have and compare them against each other.”

Which metrics do you unquestionably trust, and which do you think should not be trusted right now?

“People are not going to like this answer, but I really don't like the concept of trusting metrics in general. When it comes to metrics, it's not a yes or no question of whether it is trustworthy, but actually a process including two crucial questions; how good is the quality of the data collected and how much does it apply in a certain context. I trust this process, not the data or metric itself.

I always want to understand the context in which the data is presented, and I often question whether data is generalisable, based on this context.

You should always review multiple data points, asking the same question to see whether or not they all point to the same conclusion. This process is crucial and this can make you more confident in your interpretation of data. So, I don't like to use the word trust for that reason; I prefer the word confidence.

You can use Google Search Console to look at your impressions and clicks, and that can tell you a story. However, that can be very misleading unless you’re also looking at analytics, sales data, and a specific product and how that is performing. The more granular you can get with these things, and look at them from different points of view and different tools with different data points, the more confidence you can have.

If you’re asking, ‘How is this new product that we launched performing?’, you can look at all those channels and, if they tell a similar story, your confidence will be higher.”

How does an SEO develop the analytical skills required to be more successful at interpreting data?

“A huge part of analysis is critical thinking and understanding the processes behind those analytical skills. For example, understanding the data pipeline and data collection. How does the data get processed? If you are looking at analytics, what are the triggers that collect the data points for user visits?

Take any of your tools and try to dissect that. Investigate what a certain data point means. If you look at impressions, what does that mean? How does Google collect it? Talk to an analyst in your business and ask, ‘What are the triggers on the website that collect that data point?’ Then, we get into data processing, and you can ask questions about sampling and things like that.

More junior SEOs may not be aware that Google Search Console data is also sampled. If you're dealing with a site that has millions of URLs and you have one Google Search Console property, then the sampling rate might be really high. When you're trying to get really granular data, you might not be getting that very specific data, and you might need to set up additional Google Search Console properties. That’s the second step.

Look at that data lifecycle. Then, based on how it was collected and how it was processed, you can look at how you are analysing it – but when you are analysing it within a context, that context has to match how it was collected.

This is also why, when you set up a campaign and you want to create success measures, you have to define how you're going to collect that data. You want to ask yourself that same question when you’re collecting the data as when you're going to be analysing it. You can't just finish a campaign and then run around trying to collect different data points because they might have been collected in totally different contexts.

A good starting point would be to look at your own website and your own Google Analytics (or whatever platform you are using). If you have analysts in the business, or whoever set up your analytics tracking, talk to those people. Try to understand that full process. That will help you build out some of these skills you need to do a proper investigation.”

What is the ‘explain it to my friend’ test that you use?

“I like to use this test with anything new that I learn. It's a good sense check to see whether you are just chasing new information or you are actually understanding what you are reading and learning about.

Whenever you learn anything new and you're talking to others in the industry, ask yourself: How would I explain this to a friend? We are often really familiar with jargon, and we throw the same sentences around that we've heard and read on LinkedIn 15 times. Does that mean you actually understand it, and what it means for your business?

If you can explain it to a friend, you can explain it to your CEO or any other decision maker or stakeholder.

When we're talking about vector embeddings, vector index presence, or any other words that are being thrown around as if they are new things, are they new? We've been talking about entities and the knowledge graph, with regard to Google and contextual understanding, for a very long time.

Of course, it's in a new context, but when you try to explain it to a friend, you will try to find things that you can connect it with and how things relate. That will help you not just create better use cases in the business, but you will understand it better yourself. You understand the process behind it better, rather than just repeating what you've read a million times.”

How does your ‘So what?’ principle prevent SEOs from getting over-excited by data that doesn’t necessarily mean anything to the business?

“It’s a great principle for ensuring that you're presenting the right things, and trying to get buy-in for the right things as well.

We can get really excited about these new shiny things. If you use a new tool and see that your AI visibility is much higher or lower than you thought, you might feel compelled to present it – either to flag it as a huge risk or as a really good thing that you've done.

However, what does it actually mean? Even if you have ensured that the data is valid and robust enough, and you see a really great number, you have to be able to explain the ‘So what?’ principle: Why should anyone care about this? What does this actually mean to the business? What decision would you make on this? What suggestion are you making? Rather than just focussing on the number.

Within the industry, especially for people who do data and analytics, we tend to focus on creating a nice chart that tells a story, and forget about the ‘Why should anyone care?’ part.”

Why do you believe it’s a strength for SEOs to say, ‘I don't know’, and admit that they’re not completely reliant on the data?

“Becoming confident with saying ‘I don't know’ is a superpower

I'm not suggesting that you don’t look at these new AI metrics. I'm suggesting we have scepticism. A lot of the time, we have to experiment with these things and look at the metrics, and we will be asked questions, so we have to present them.

However, if you can present them in the context of, ‘This is experimental and there are no standard methods to validate anything yet,’ you're showing that you truly understand the stage that this metric is in.

You can say, ‘I don’t know,’ to questions because we are not there yet. Currently, no one knows, and if they do know, you will see it on LinkedIn when the people chasing these metrics publish new results.

You create more trust if people know that you will tell them when you don't know something. You can also say, ‘I don't know, but this is how I will find out,’ or ‘I don't know because this doesn't exist yet, but when it exists, I will hear about it because I follow the right people, the right newsletters, and the right information channels.’

That gives good confidence to other stakeholders in the business that they can trust you because you're not going to BS them.”

Petra, what's the key takeaway from the tip you shared today?

“Reflect on the learnings. Give yourself time to reflect on everything you're learning and follow a process of healthy scepticism.

Use critical thinking. When new information comes to you, have a process and ask some of those questions: Where is this coming from? Has this been validated? Do I have any other points to validate that this new information is relevant and important to me?

Have that process and build that scepticism into your reflections.”

Petra Kis-Herczegh is a Strategic Consultant at Kameleon Journal. Find out more over at KameleonJournal.com.