One thing right away… do not use “ranking for x” as your metric. While ranking and traffic are correlated, assessing an SEO’s work based on “ranking for x” is going to mislead you, waste time, and have large opportunity costs. There are too many variables, such as location, search history, and more.
Here are some major ways I think you can assess their abilities.
Good SEOs can quantify the opportunity costs of not ranking well. Not ranking well has actual costs, but these are rarely on spreadsheets in financial reports. When a competitor ranks well for a buy-phrase, the poorly-optimized client is missing those dollars. Helping a client understand where to spend limited resources to limit these costs and drive revenue is an essential skill.
Good SEOs Focus on Quality and Quantity of Leads 50 visitors ready to buy is better than 500 visitors who didn’t want to be on your site in the first place, so there are times when a lower organic visit count is still SEO success. Assess the readiness to buy element of the leads you get. Lead scoring is essential, as is proper lead attribution.
Good SEOs don’t chase “today’s algorithm.” Good SEOs will forgo immediate spikes in favor of futureproofing your brand’s long term ability to rank and good leaders will give them the time to do so. They also become strongly aware of your searcher’s intent and activity over the buying cycle, helping you focus on the content and target phrases that move the buyer further along.
Good SEOs Help Clients Avoid Mistakes. Good SEOs help clients avoid mistakes – sometimes very expensive ones – that might impact revenue or competitive position. Site migrations are one of these places where traffic can be lost for years if done without a plan. Spending limited resources and time on “us us us” content that cannot acquire links is another.
Good SEOs Enable Reduced Reliance on Pay Per Click. A balance between organic and paid ads is always wise, so as better traffic becomes more-better traffic, you should be able to lower your overall cost-per-lead or cost-per-sale as a business as organic traffic’s lead share increases – especially on unbranded terms.
Good SEOs are Strong Educators and Explainers. Strong SEOs are willing to educate clients, and to make the connections between their business goals and day-to-day activity and long term SEO success. A Good SEO can explain their work at any time to the CEO, or lead meetings with many stakeholders.
Good SEOs know your competitive landscape. Much of SEO is “filling in the gaps” in the search results where competitors are weak and then expanding. They understand the long tail concept and help clients invest their time/resources properly in these precise targets.
Good SEOs know that 60–70% of ranking influence is off page, and will help the client partition their team’s work accordingly.
Good SEOs take part in experiments and try new things, folding useful results into their repository of knowledge they can apply to their clients.
Some of this is explained in my post called “Surviving the SEO Slog” which you might enjoy.