We are the only sector that does not use publication standards and peer review processes for the studies that we publish. I advocate for a high-quality set of processes because they directly influence our thinking and actions as specialists.
Limited studies and no transparency
Science does not take place in a vacuum. The timeline of science takes us from one hypothesis to another. And evidence-based digital marketing and SEO simply do have a scientific basis. I would like to advocate for a high-quality set of science-publication based processes, because these will directly influence our thinking and acting as SEO and marketing specialists.
Ego makes us want to make everything visible to ourselves, and so we use science to show us what we can make visible in the moment. At the same time, science forces us to recognise the limits of our individual knowledge. On the timeline where science takes place, a free marketplace of ideas and ‘thinking together’, are essential for the realisation of high-quality truth-finding.
The realisation of a high quality ‘thinking together’ through the studies that we publish, is extremely important. However, corporate interests, a lack of standardisation, and a commercialisation of knowledge, threaten our ability to do this. When we limit ourselves to the PR rooms of tech companies and agencies, we run the risk of being unable to reach an expert consensus as specialists.
Something that Google is especially eager to take advantage of, when they want to make it harder for SEOs to artificially manipulate their algorithm system. This throws the unregulated field of SEO and the knowledge of its consultants, back to the Dark Ages. The lack of complete and honest information, relegates SEO specialists to gamblers, scammers and quacks.
Opinion versus protocol
Although specialists also give some advice based on experience, they prefer advised actions to be taken on the basis of evidence. As they are only human, they are quick to assume that ‘guidelines’ written by Google, agencies, and tool makers, are protocols. And we see protocols as instructions that must be based on reliable data. But who provides the data? Google, agencies, and tool makers themselves.
High-value data studies don’t come cheap. Commercial companies will only want to carry them out if there is a return on investment. Without valuable insights into how industry-sponsored surveys are misrepresented, evidence-based marketing and SEO remain an illusion.
Interests and publication bias in the Search industry
The Search industry is dominated by a small number of very large authorities looking to expand their market, rather than looking for ways to help people increase the findability of their websites. Hierarchical power structures and loyalty within the organisations, produce SEOs and marketers who simply do as they are told. And as the organisation’s need for sponsors grows, consultants, publicists, and marketing events, become tools of the Search industry.
If a study has a negative impact on commercial interest, why publish it? Ownership of data and knowledge, leads to publication bias. We therefore never see adverse research results, and no raw data is fully shared. In the digital marketing world, there are a lot of people who are brilliant at statistics, who can analyse the data, but the raw data must first be made available for objective evaluation.
Marketers and SEOs are used to promote the commercial products of the authorities in the sector. Managers are hired and forced to demonstrate their profitability for the organisation, they are not hired because they did good marketing research in the past. For those who have made a name for themselves, and want to move forward, these are the great opportunities that the Search industry offers. In marketing terms, these people are called KOL’s: “Key Opinion Leaders”.
Science based on press releases
KOL’s are selected based on their influence on other marketers and the prestige they bring. They deal with product branding and event attendance. When we hear from them, we have to keep in mind, that no matter how long they’ve been working in the Search industry, they have to provide evidence. We can only have this evidence, if we have the raw data. And they don’t offer that. The “product promoter elite”, has completely lost the independence that they once might have had.
Critics of these ‘thought leaders’, are unlikely to get published on high-authority Search industry related websites, or may even face legal threats. This creates an uneven playing field. To have a career in SEO, you practically need to be publishing. So when industry publishers reject you for being overly critical of their favorite faces, it can pose a huge problem. And it creates a form of suppression and control of science communication in the Search industry by the big companies, including Google.
Standardisation of studies is a step in the right direction
I am not in favor of a regulatory body that fully assesses and directs the SEO sector. Because regulators would likely receive funding from the Search industry itself, and in most cases, even industry regulators, do not get access to raw data. But simple standardisation across the SEO/marketing sectors in the way that we publish studies, is a step in the right direction.
Why don’t we write every study in this way?
Provide the studies with all the necessary sections, and write them in this order: methods, results, conclusion, introduction, summary and title. Methods are almost always completely ignored in study reports. If you’re not showing enough information that readers need to trust how your results came about, why should people trust your research?
The methods should include at least the following: all information about the population of data and the sampling methods used, the procedures and time frame, the analysis plan, any approaches to ensure validity and reliability, and any assumptions and the scope and limitations of the methodology.
Show the results of statistical analysis, confirm that the method is reliable, justify the choice of the methods, and define its limitations. Statistical hypothesis tests can be used to verify each null hypothesis, (assumption), with the results of the study. Each test method used must therefore be stated and relevant parameters must be shown.
In the research conclusion or discussion, you get the chance to sell the data. As part of the study itself, or published separately. Explain what the results mean, whether the methods were successful, how the findings compare to those of other studies, and what the limitations of the study are. Be specific and don’t exaggerate. Because we are free to speculate about an interpretation, but when we do, we have to make sure that we are as accurate as possible. Don’t make things up. Compare your results with other studies, and don’t ignore conflicting studies. Make sure to mention them.
This is what we want to avoid as much as possible, when publishing and speaking around SEO and marketing studies: downplaying negative results, making statements based on personal opinion without substantiation with actual evidence, repeating information from other sections, and over-emphasizing the impact of a study. Of course, it is also important to acknowledge everyone who has helped you with the study, but at the same time you should mention any funding sources.
One step further: combine and share raw data
The constant changes in complexity and volatility of modern algorithmic systems, such as that of Google Search, risks that we as an industry, have to seriously rethink the sample sizes used in our current studies. They are likely too small to measure the differences we need to see in a dynamic algorithm system to identify important modifiable factors, even in the large studies that big tool makers and agencies publish about, and that speakers and specialists use as a basis for their advice. It is quite possible to anonymise and/or ‘white label’ raw data, and/or to take the results from several (smaller) studies to combine them with each other in a different kind of study.
When we allow other technical departments to analyse the studies we publish, we make the data, and therefore the science on the basis of which specialists give their advice and carry out actions, more reliable. We can’t really peer review anything without the complete raw data. Without that, true ‘evidence-based’ marketing and SEO remains an illusion.
When you’re a client with Remco Tensen Consulting, every account, and all your data, is fully accessible to you at all times, and will remain in your ownership, even after you decide to leave.