I have created a trivial/demo SPA with Ember.js. It has no meaningful content. Just filler. I created this specifically to try and understand how Google may crawl a production Ember SPA of mine.
Google says...
In 2009, we made a proposal to make AJAX pages crawlable. Back then, our systems were not able to render and understand pages that use JavaScript to present content to users.
...
Times have changed. Today, as long as you're not blocking Googlebot from crawling your JavaScript or CSS files, we are generally able to render and understand your web pages like modern browsers.
I have not created a sitemap.xml
or tried to enhance the HTML semantics in any specific way for Google. I just threw this together in about an hour. I was hoping to understand how Google crawls the links on a site like this.
The app renders without issue in production, and is backed by fastboot running server-side, which can route non-root URLs without a hashbang. So a server-side request for something like /about
or /api
is handled properly and renders the SPA with the correct route/pushState.
It's all fine and dandy, except when I ask Google to crawl the root (/
) "and linked pages". For reasons unclear to me, Google shows that it completely misses the nav bar when it crawls...but, the style is there, the dynamically loaded content is there.
So, where's the navigation?
I didn't think there was anything odd about the navigation. It's not hidden by my styles. It's certainly not ideal as far as I'm concerned. I'd rather see this as an li
, but again, I threw this together in an hour. I didn't imagine that having the navigation set up like this would make it completely hidden from the crawler for whatever reason.
<div class="nav-items">
<div><a href="/api" id="ember257" class="ember-view">API</a></div>
<div><a href="/about" id="ember266" class="ember-view">About</a></div>
</div>
A result of this seems to be that Google search results are not showing my /about
page, but do show my /api
page. The /api
page is linked to from both the body and the nav, but the /about
page is only linked from the nav. Since the nav isn't rendered, it seems like search results from Google do not know about my /about
page.
I'm sure that a sitemap.xml
and other changes would fix this, but my curiosity lies in why the nav bar is hidden. The "Fetching" tab in "Fetch as Google" shows that the nav bar HTML is present, but it seems like it is not rendered from Google's perspective. Why is the Google Crawl rendering process seemingly ignoring it?