How do you implement SEO best practices for dynamic and JavaScript-based websites?

A piece of software designed to find content by followinglinks. Content can be web pages, images, videos, PDFs, sound files, any just about content that can be hosted online. A web crawler is also referred to as just a bot, a spider, a robot, or even just a crawler. Google's web crawler is named Googlebot, Bing's is Bingbot. Both are easy to find within a web server's log files if one wanted to know how often their website was being visited by web crawlers. Crawling does not collect information about the page, this happens after the crawl, during the indexing phase, when search engines attempt to understand what the content is all about. Indexing is also when the search engine will decide to include the content within it's index or not.

Understanding how a web crawler works and how to adjust your website to accommodate (or limit) one can mean the difference from appearing for a keyword or missing out on your content appearing for a keyword. Indexing is important to learn, as it can help improve the efficiency of web crawlers who won't waste their time (or your "crawl budget" - the amount of URLs that a bot can and wants to crawl) on URLs you would prefer not be indexed. These pages are often redundant, or duplicate, such as those that exist on both www prefix and without, URLs with parameters for sorting, filtering, and refining, or confirmation pages meant only for users after making a purchase or completing an action on the website.

If it can't be crawled. It can't be indexed. If it can't be indexed it can't be found. If it can't be found then you won't be seen.

I'd add that reducing the size of content to just what the user and search engine need, versus excessive design or functionality, can improve how quickly a search engine can crawl and then analyze the content. Do you need 40 external JavaScript files, or can you reduce them to 1 or 3? Same for stylesheets. Using Google Search Console and Bing Webmaster Tools, one can easily submit a list of URLs they would like crawled, and then analyze the reports within both to troubleshoot for issues. Third-party tools such as Ryte, OnCrawl, Screaming Frog, and SiteBulb can be extremely insightful when it comes to discovering crawling and indexing issues, even for larger websites with billions of pages of content.

Despite following SEO best practices, it's possible that some of your site's pages may not get indexed. This can be due to several reasons: 1. Page exclusions due to meta tags, robots.txt, canonicalisation, duplications 2. Page redirections and errors 3. Other issues Please note that not every page needs to be indexed. And the above exclusions don't always require a fix. Go to Google Search Console > Indexing > Pages Here, you should be able to see the breakdown of pages that haven't been indexed for various reasons. If important site pages are showing up as not indexed in this section: 1. Check data on that URL in Google's index 2. Make necessary fixes 3. Validate the fix within Search Console, and request indexing

Things to avoid to prevent indexing issues: 1. Copying content from another website. 2. Copying content verbatim from large language models such as Bard and ChatGPT (or tools that use those technologies to create content). 3. Not customizing the HTML title or meta description tags for each URL. 4. Hosting a website in a "bad neighborhood" on an IP address that has been recognized as installing malware on a user's computer. 5. Writing a few sentences about a topic you would like a URL to appear for, versus creating the most helpful page when comparing the URLs that currently rank well. We call this "thin content." 6. Dynamically creating thousands (or millions) of pages at once on a website with few incoming links from other websites.

I would add that search engines don't like to be fooled. Serving up keyword-rich content to search engines, but not to users, is misleading and can lead to penalties, including omission from search results. Technologies such as REACT enable webmasters to load a static version of a page to search engines (aka: pre-rendering) and the final JavaScript-powered version to visitors. This is the opportunity to improve performance or to try cheating. I'd recommend against the latter. Fortunately, Google in particular is improving how to navigate JavaScript every day. If you are unsure if JavaScript is preventing search engines from seeing important content, use the URL Inspection Tool within Google Search Console to see how Google sees your page.

There's a lot of misinformation out there about JS websites being bad for SEO, which just isn't true. In some cases, JS websites are actually better for SEO given the significant performance improvements.

I would assemble the technology teams once per month and hold each other accountable to the following questions, having Ryte, OnCrawl, or other crawl tool to refer to: 1. Have we improved our website's crawlability and indexability in the last 30 days? 2. What challenges are preventing us from having any crawl errors or excessive redirects on our website? 3. How are we performing from a technical standpoint versus the competition? 4. Are the overall scores improving in the crawl tool we test our website with? 5. How have improved Page Experience and Core Web Vitals over the last 30 days? 6. How can management or leadership stakeholders help?