Doing SEO for your site you are facing repetitive routine tasks. Some of them include: checking broken links, checking redirects, checking your HTML markup. It takes a few minutes to manually check source code of several pages. But if the site you are working with includes thousands and thousands of pages, checking becomes a serious issue. The right tools can help you automate repetitive tasks and make your SEO work much easier.
Google Search Console vs SEO Crawler
One option is to use online tools provided by leading search engines: Google Search Console (formerly known as Google Webmaster Tools) and Bing Webmaster. In this case you get complete and reliable information about how your site is featured in search engines. You get information at first hand, right from the search engine, rather than third parties.
But online webmaster tools have few shortcomings. You have to wait a few days before effects of changes you made on website become visible in your search console. Another weak spot is inability to check your site before going live
So you can choose another option: desktop SEO crawler. Using desktop tool you get a couple of advantages: you get check results immediately, you can test your site in the development environment, you can customize information being gathered, you can combine several data sources. This can drastically improve your SEO workflow. For instance, your developer has changed URL structure; you crawl your site and in a few minutes you can see how your SEO is affected by this change; if broken links appeared you can fix them immediately. This way you can avoid many concealed SEO pitfalls before your site is deployed to production.
Eventually, Google Search Console and Bing Webmaster are great SEO data sources, but desktop crawlers are absolutely indispensable in your development-optimisation iterations.
Common crawler SEO tasks
Common SEO tasks you can perform with crawler:
- ✓ check broken internal and outer links
- ✓ check broken images
- ✓ check broken CSS and script files
- ✓ check missed title and description tags
- ✓ check custom HTML markup: Open Graph, Schema
- ✓ check 301 and 302 redirects
- ✓ check if all redirects are appropriate during site migration
- ✓ find duplicate pages
- ✓ create HTML and XML sitemaps
- ✓ and much more
There’s a couple of SEO tools described below. All of them are proved to be robust and reliable. You can choose a tool which suits your needs and preferences.
Xenu’s Link Sleuth
Synopsis: free, simple
Dead simple tool that checks a website for broken links. And makes it perfect. Xenu’s Link Sleuth is fast and utilizes “a simple, no-frills user-interface”. That’s your choice if all you need is to check broken links and generate HTML sitemap.
Netpeak Spider
Synopsis: free, all basic SEO tasks, click and run
Netpeak Spider performs much of basic SEO tasks (list is not exhaustive): crawls website pages, finds errors, harvests title, h1, description tags, finds duplicates. Advanced features include:
- ✓ Page Rank calculation based on internal site links. If you are curious how your link juice is distributed among pages.
- ✓ export to Excel. You can analyze all collected data using all power of your favourite spreadsheet software.
Good point is a simple Settings bar. You don’t have to tick dozens of checkboxes to force the spider to go beyond first page. All you have to do is click Start button.
And all these features are available for free!
Screaming Frog SEO Spider
Synopsis: paid, great tool, must have
Great tool for your everyday SEO needs! It crawls through your website pages in the same way the search engine does. It even supports Google protocol for AJAX sites! So you can make SEO for a website built with Angular, Backbone or any other cool kid framework.
Crawling your site pages Screaming Frog SEO Spider collects everything you need to optimize your SEO performance: title, description, H tags, canonicals etc. Moreover, starting from version 4.0 custom extraction is available. Using Xpath & CSS Path selectors, as well as regular expressions you can extract any data: social tags, hreflang tags, custom Schema markup.
Some of SEO tasks you can perform using Screaming Frog:
- ✓ check missed Google Analytics tags
- ✓ incorrect canonicalisation
- ✓ meta robots errors
- ✓ check multiple redirects path
- ✓ check redirects during site migration
- ✓ much more
Since version 5.0 Screaming Frog can integrate with your Google Analytics and Google Search Console accounts. Now you can combine On-Page SEO data collected during crawling with actual search performance data provided by Search Console and also with audience data available from Google Analytics.
Website Auditor (part of SEO PowerSuite)
Synopsis: expensive, cluttered interface, superfluous features
Website Auditor has a lot of features available. Some of them are useful, but some of them are totally redundant. Some advantages of this tool include tight integration with other parts of SEO PowerSuite package, deep content and keyword analysis.
Custom Spider
If tools described above don’t suit your needs, you can always switch to custom development. Custom spider can collect all data you need observing all your rules. Developing custom spider from scratch requires enormous efforts, so a good point is to consider using web-scraping frameworks available for language of your choice.
