Some links here are affiliate links — if you buy through them we may earn a commission at no extra cost to you. It never changes what we recommend. Full disclosure.
Technical SEO is the foundation everything else sits on. The best content in your niche will not rank if search engines cannot crawl it, choose not to index it, cannot render it, or cannot work out how it relates to the rest of your site. Most technical problems are invisible to visitors — which is exactly why they go unnoticed for months.
This guide explains the core technical concepts in plain language: how crawling and indexing really work, crawl budget, JavaScript rendering, site architecture, mobile-first indexing, Core Web Vitals, structured data, international SEO, and why a brand-new site can take months to rank. For a hands-on checklist, see the DIY technical and on-page SEO audit.
How Search Engines Process a Site
Search engines work in stages:
- Discovery — finding URLs through links, sitemaps and previously known pages.
- Crawling — fetching those URLs.
- Rendering — running JavaScript to see the page as a browser would.
- Indexing — analysing the content and deciding whether to store it in the index.
- Ranking — choosing which indexed pages best answer each query.
A problem at any stage stops a page from reaching the next one.
Crawlability vs Indexing: Two Different Problems That Look the Same
When a page does not appear in search, it is tempting to assume one cause. There are two very different ones:
- Crawlability problems: search engines cannot reach the page — it is blocked by robots.txt, has no internal links pointing to it, sits behind a login, returns server errors, or loads its links only through JavaScript events crawlers do not follow.
- Indexing problems: search engines can reach the page but choose not to index it — a
noindextag, a canonical pointing elsewhere, content too thin or too similar to other pages, or simply not enough perceived value.
Search Console's URL Inspection tool tells you which: "URL is not on Google" with "Crawled – currently not indexed" points to quality or duplication, while "Discovered – currently not indexed" often points to crawl prioritisation or internal linking.
Crawl Budget: Why It Matters More on Larger Sites
Crawl budget is roughly how many URLs a search engine is willing and able to crawl on your site in a given period. For small sites (a few thousand pages), it is rarely a concern. On large sites — e-commerce catalogues, publishers, sites with faceted navigation — wasted crawling becomes a real problem:
- Common waste: parameter URLs, infinite filter combinations, session IDs, duplicate pages, redirect chains, soft 404s and slow server responses.
- Fixes: keep low-value URL patterns out of crawl paths, consolidate duplicates, fix redirect chains, keep sitemaps clean, improve server speed, and link prominently to important pages.
Search Console's Crawl Stats report shows how often Google crawls your site and how quickly your server responds.
JavaScript SEO: What Still Trips Up Modern Sites
Google can render JavaScript, but rendering is resource-intensive and can be delayed, and other search engines and AI crawlers handle JavaScript less reliably. Common problems:
- Content or links that appear only after user interaction (clicks, scrolls)
- Links implemented as click handlers rather than
<a href>elements - Critical content loaded from APIs that fail or time out for crawlers
- Titles, meta tags and canonical tags set only by client-side JavaScript
- Blocked JavaScript or CSS files
Server-side rendering, static generation or hydration of pre-rendered HTML avoid most issues. Check what Google sees with URL Inspection's rendered HTML and screenshot.
Site Structure
Site Architecture: Building a Structure Search Engines Can Actually Understand
Good architecture organises content into a clear hierarchy — home, categories, subcategories, individual pages — so both users and crawlers can understand what the site covers and which pages matter most.
Principles:
- Shallow enough: important pages reachable within a few clicks of the homepage.
- Logical grouping: related content grouped into sections and topic clusters.
- Strong internal linking: hub pages link to everything in their section; related pages link to each other.
- Consistent URLs that reflect the structure where sensible.
- No orphans: every indexable page linked from somewhere.
Architecture also distributes authority: pages linked prominently from strong pages tend to perform better than pages buried deep in pagination.
Mobile-First Indexing Isn't New, But It's Still Widely Misunderstood
Google predominantly uses the mobile version of a page for indexing and ranking. The implication is not just "have a mobile-friendly design" — it is that whatever is missing from the mobile version effectively does not exist for Google. Check that mobile pages include the same main content, headings, structured data, internal links, image alt text and metadata as desktop. Hiding content in tabs or accordions on mobile is fine; removing it is not.
Core Web Vitals: What Actually Moves the Needle
Core Web Vitals measure real-user experience: Largest Contentful Paint (loading), Interaction to Next Paint (responsiveness) and Cumulative Layout Shift (visual stability). They are part of Google's page experience signals, but they act more like a tie-breaker than a dominant ranking factor — relevance and content quality come first. What moves the needle on the metrics themselves:
- LCP: fast server response, a properly sized and prioritised hero image, fewer render-blocking resources.
- INP: less and lighter JavaScript, fewer heavy third-party scripts, breaking up long tasks.
- CLS: dimensions on images and embeds, reserved space for ads and dynamic content, careful font loading.
Use field data (the Core Web Vitals report in Search Console, based on real users) to prioritise, and lab tools like PageSpeed Insights to diagnose.
Page Speed: The Technical SEO Factor Users Feel Before Google Measures It
Speed matters to rankings at the margin, but it matters to users immediately: slow pages lose visitors, conversions and the engagement that keeps a site healthy. Treat speed as a product requirement — performance budgets, image discipline, limited third-party scripts and quality hosting — rather than something to fix after launch.
Making Content Machine-Readable
Structured Data: Helping Search Engines Understand, Not Just Read
Structured data (usually JSON-LD using the Schema.org vocabulary) labels the meaning of content explicitly: this is a product with this price and availability; this is a recipe with these ingredients; this is an organisation with this logo and contact details. Benefits:
- Eligibility for rich results such as review stars, prices, FAQs (in limited cases), breadcrumbs, events and recipes
- Clearer understanding of entities and relationships
- Consistent information across search features
Rules: mark up only content visible on the page, keep values accurate, use the most specific type that applies, and validate with Google's Rich Results Test. Rich results are never guaranteed — eligibility is not the same as display.
International SEO and Hreflang Without the Headaches
For sites serving multiple languages or countries, hreflang annotations tell search engines which version of a page to show to which audience. The essentials:
- Each language/region version lists all versions, including itself (return links are required).
- Use correct language codes (and optional region codes):
en-GB,en-US,de. - Add an
x-defaultfor a fallback page (such as a language selector). - Keep hreflang consistent with canonical tags — each version should be canonical to itself.
Choose a URL structure (country-code domains, subdirectories like /de/, or subdomains) and stick to it. Translation alone is not localisation; currency, examples and keyword research should be local too.
Keeping a Site Healthy
Server Uptime and Response Monitoring: The SEO Basics Everyone Forgets
If your server is down or slow when a search engine visits, crawling slows down; repeated outages can lead to pages dropping out of results. Monitor uptime and response time externally — DigitalOcean's Uptime monitoring includes a free check that works for any site — and investigate spikes in 5xx errors in Search Console's Pages and Crawl Stats reports.
New Site, No Rankings: What's Actually Happening Behind the Scenes
New sites commonly see weeks or months of minimal visibility. Behind the scenes, search engines are discovering pages, deciding which are worth indexing, and gathering signals about quality and trust — links, mentions and user satisfaction take time to accumulate. Speed things up by submitting a sitemap, linking all pages internally, earning a few genuine links and mentions, publishing consistently in a focused topic area, and targeting specific long-tail queries first. If pages are not indexed at all after several weeks, check for technical blocks before assuming it is just patience.
The Bottom Line
Technical SEO is about removing obstacles: make pages discoverable and crawlable, make them worth indexing, render content in HTML search engines can read, organise the site so importance is obvious, keep it fast and stable on mobile, label content with structured data, and annotate international versions correctly. Get these right and your content can compete on its merits.
Next step: Run a quick technical check with the free SEO Audit tool, and generate correct hreflang tags with the Hreflang Tag Generator.