As recently as twenty years ago, we were flirting with new ideas about what a web page should look like. Originally trying to make a page stand out from the crowd, webmasters began to vary from the simple plain-text pages that had been common and started forging ahead into a brave new world of the best colors, animations, and text effects that 1996 had to offer. Picture it with me now — pink and green, and flickering, with an animated gif of a pixelated starburst tiled across the background and some garish midi file autoplaying even before the text has loaded. It was nauseating, profoundly unhelpful, and swiftly abandoned, but it’s been replaced in recent years by another design philosophy with the potential for even greater harm: interactive menu layers.
Now, to be fair, the issue is a little more complex than simple aesthetics. So long as your interactive (usually java) artifacts don’t compromise utility with increased page-load times or lag, an animated menu can be a great way to add a sense of depth and firmness to a webpage, which can seriously enhance user experience. Still, this is one of the rare times when what’s best for user experience might conflict with your SEO needs, unless you take steps to counteract the damage.
Unless you create a workaround, these interactive layers and animations often prevent your site from being fully indexed, and can prevent your content from becoming searchable.
What Is Site Mapping, and Why Is It Still Relevant?
In simplest terms, site mapping is the creation of a single document, like a map or a file hierarchy, which includes a link to every page on your site, and the approximate arrangement of those pages. Think of it like a family tree for your website. A sitemap’s purpose is to show web crawlers how the content on your website is organized so that your content can be more easily visible to search engines. Web crawlers, usually called “spiders” for obvious reasons, are automated scripts with the function of archiving web content. They’re typically deployed by search engines to build up a database of searchable content. They have a number of other uses, but, for our purposes, it’s enough to know that their job is to keep track of your web content so that it can show up in search results.
When it’s phrased that way, treating site mapping as a regular part of SEO seems like a no-brainer, but it isn’t practiced nearly as often as it should be. Now that websites have developed more intuitive interfaces than the glorified file-tree of yesteryear, displaying a sitemap is seen as gauche, or at least archaic. They’re not generally visible to most users anymore, if you don’t know where to look, and there’s a very real possibility that some of the current generation’s business owners have grown up never having seen a sitemap and may simply not realize the important role that sitemaps play in web infrastructure.
Without a proper sitemap, even if your site is perfectly intuitive for a human user (rather than an algorithmic one), many of your pages might be invisible to Google, totally sabotaging any other SEO you might incorporate.
It’s a Spider’s Web; Make Sure Your Pages Get Caught in It
I’m not the first person to point out the pun, that spiders make the web, but it’s an extremely apt description. For easy reference, the “internet” is the infrastructure by which distinct systems can communicate, and the “web” as in “world wide web” is one of the communication protocols which uses the Internet’s infrastructure. The web uses the HTTP protocol, which is just one of several languages the Internet can speak, but which is the one you’ll have had the most experience with. Your browser is a web tool.
Now that that’s out of the way, let’s unpack the pun. The web depends on interconnection, cross-linking, and accessible HTTP addresses. Spiders are employed to literally build the web, in that they follow links, recording the content they find on their way through, and mapping out the structure (defined by hyperlinks) of the web, such that search engines can point a prospective user to a piece of content by knowing where it is in the web. Our tech infrastructure has come a long way in recent years, but it’s still built on an essentially spatial system.
In order to do this, spiders need to follow a consistent pathway of links, like mapping out every possible turn in a maze, in order to generate a complete map.
For SEO, you want to make the spiders’ job as easy as possible. You want to ensure that you have given them a clear path to every single piece of your content, so that nothing gets missed.
Site Mapping and Interactive Menus
When you use an interactive menu, whether it’s in Java, HTML5, Flash (seriously?) or some other system, you’re stopping the spiders in their tracks. They can’t parse that coding, so they don’t bother. They backtrack, and take another path. Imagine your website having three layers:
-The basic layout of your page; the latticework to hang your animation elements on
-The animations and visual artifacts
-The actual content, including menu hyperlinks
When spiders get blocked by the second layer, they never see the third layer. It’s a roadblock. Your content doesn’t get indexed, and it won’t show up in search results.
So, give the spiders a backdoor. In the HTML of every page of your site, in the primary layer, include a link to your sitemap (typically in XML format, but occasionally in HTML, RSS, or any of several other options.) That way, you’re facilitating a clear indexing path, and ensuring that no content gets missed.
You won’t slip through the spider’s web.
Impact on SEO
Site mapping really is one of the most basic forms of SEO. You’re cutting a direct path to your content for spiders to follow, ensuring that no part of your site gets missed, and that every page can show up in search results. Every other SEO tip or practice presumes that your content is at least visible, so this is a pretty clear make-or-break proposition.
By employing a sitemap, you’re essentially introducing your website to Google. The impact on any other SEO you’re employing can’t be overstated. If your pages are hidden, and invisible to spiders, it won’t matter how well optimized they are in any other category.
When Is Site Mapping Most Useful?
If you’ve taken other steps to make sure that your various pages are all clearly and directly linked to each other, spiders should have a pretty easy time indexing your site even without a sitemap. Nevertheless, without getting technical, spiders only follow a finite number of links for a given subdomain, so without a full map you may still end up having some of your content overlooked. There’s no harm in keeping an up-to-date sitemap, even if you think it’s mostly redundant, especially if your site meets one the following criteria:
- Your site is particularly large.
This can mean that some content gets missed, or that indices aren’t refreshed frequently enough to track all your updates and changes
- Your site includes archives or files that don’t interconnect.
This can also sometimes prevent spiders from catching every page, and, as with larger sites, new content may not get archived right away.
- Your site has not yet been linked to, by a number of other sites.
From its inception, Google has depended on interconnectivity to map the web. If your site is not yet enmeshed in that web, a sitemap is a good starting point to get some extra attention from the spiders.
- Your site uses rich media content, or optimizes for annotations like Google News.
Spiders will take your sitemap’s tags and optimized descriptions into account, when inventorying your rich media content. Don’t miss out on the opportunity!
Here’s what Google has to say on the matter:
“Using a sitemap doesn’t guarantee that all the items in your sitemap will be crawled and indexed, as Google processes rely on complex algorithms to schedule crawling. However, in most cases, your site will benefit from having a sitemap, and you’ll never be penalized for having one.”
What Sorts of Content to Include?
Sitemaps should chiefly include your various webpages, but don’t neglect images and video. Spiders will extract metadata about everything straight from your sitemap. For webpages, Google will typically track the date of the most recent update, and the frequency of updates, as well as the priority relative to the rest of your subdomain. Further, Google recommends that you provide the following:
Sitemap Video Entry:
- Running time
- Age appropriateness rating
Sitemap Image Entry:
- Subject matter
- License information
If you aren’t currently using a sitemap of some kind, get one. Don’t hesitate, just make it happen. Every second you go without is one more second that your site remains unoptimized for search. If you do already have one, make certain that you’ve kept it up to date with all your recent content. If it’s been a while, and if your current one will need more than a little tweaking, it might be to your advantage to just scrap the old one and start fresh. That way, you can be sure that nothing gets overlooked and that you’re taking full advantage of all the metadata options available to you.
Be sure that you include a link to your sitemap somewhere on the first layer of every one of your main pages. It’s perfectly fine to link it from every page, but for many sites that would be prohibitive. As a rule of thumb, when you think of your webpage hierarchy, link your sitemap from any page on all but the ultimate layer (e.g. “homepage” and “blog archive” should have links, but not necessarily each individual blog post). Conventionally, your sitemap would have an address like https://www.yourpage.com/sitemap.xml.
Once you’ve done a sitemap overhaul, it’s time to submit a re-indexing request to Search Console and make sure that your new sitemap is reflected in the SERPs.
As with any issue that might be negatively impacting your SEO, your quality score for AdWords, or any other aspect of your digital marketing practice, we can help! We’re the best digital marketing company San Francisco has to offer, and we’re ready to help. Take us up on our offer for a free digital marketing strategy session, and see why we’re the best!
Colibri Digital Marketing
We’re the digital marketing agency for the twenty-first century. Based in San Francisco, we’ve got our fingers on the pulse of Silicon Valley, we’ve got an insider perspective on the tech industry, and we get a sneak peak at the future of digital marketing. If you’re ready to work with the best, drop us a line or click here to schedule a free digital marketing strategy session!
This post was originally written and posted by Andrew McLoughlin. It is republished here on on behalf of Colibri Digital Marketing. Thanks, Andrew, for a great read!