WABBIT · ESSAYS

Payload CMS SEO in Next.js: the static sitemap that hid 55 pages from Google

The SEO plugin covers the fields an editor fills in; this is everything else a Payload 3 and Next.js site needs, with the sitemap bug that kept 55 of our pages out of Google.

What the SEO plugin covers, and what it leaves to you

@payloadcms/plugin-seo adds a meta group to your collections: a title, a description, an image, an overview that checks their lengths, and a search-result preview. You can let the plugin inject the group, or place the fields yourself. We place them by hand in a tab on Pages:

TypeScript
// src/collections/Pages/index.ts
import {
  MetaDescriptionField,
  MetaImageField,
  MetaTitleField,
  OverviewField,
  PreviewField,
} from '@payloadcms/plugin-seo/fields'
// …
{
  name: 'meta',
  label: 'SEO',
  fields: [
    OverviewField({ titlePath: 'meta.title', descriptionPath: 'meta.description', imagePath: 'meta.image' }),
    MetaTitleField({ hasGenerateFn: true }),
    MetaImageField({ relationTo: 'media' }),
    MetaDescriptionField({}),
    PreviewField({ hasGenerateFn: true, /* … */ }),
  ],
},

The plugin's config takes generateTitle, generateDescription, generateImage and generateURL. Those fill the fields and draw the admin preview. Nothing in the plugin writes to your <head>, builds a sitemap or emits structured data. Every one of those is your job, in Next.js.

Point generateURL at the same path function your routes use, or nested pages preview at the wrong address in the admin. One that returns ${url}/${doc.slug} is right for top-level pages and wrong for the nested ones covered below. It only affects the preview snippet, so it's easy to miss.

Canonicals: one self-canonical, from one function

Every route on our site builds its metadata through a single helper, generateMeta, which takes the Payload doc and the page's real public path. For a long time that helper set og:url and nothing else. Route comments said "self-canonical", but no page emitted <link rel="canonical"> at all. That mattered because an old WordPress copy of the same 15 chapters was still live and canonicalising to itself, so Google had no signal about which copy was ours.

Next.js renders alternates.canonical as the canonical tag, so the fix is one property:

TypeScript
// src/utilities/generateMeta.ts
const resolvedPath = path ?? deriveFallbackPath(doc)
const normalizedPath = resolvedPath.startsWith('/') ? resolvedPath : `/${resolvedPath}`
const fullUrl = getPublicURL() + normalizedPath
// …
return {
  // Self-canonical on the resolved path. …
  alternates: { canonical: fullUrl },
  description,
  openGraph: await mergeOpenGraph({ description, images: [{ url: ogImage }], title, url: fullUrl }),
  title,
}

Each route passes its path explicitly, because a slug alone can't tell you the URL prefix:

TypeScript
// src/app/(frontend)/compare/[slug]/page.tsx
export async function generateMetadata({ params: paramsPromise }: Args): Promise<Metadata> {
  const { slug } = await paramsPromise
  const page = await queryPage(slug)
  return generateMeta({ doc: page, path: pagePath({ slug, section: SECTION }) })
}

If a route has its own generateMetadata that skips the helper, it skips the canonical too. Search for every generateMetadata in your app and confirm each one goes through the same function.

One sitemap per collection, plus the routes Payload doesn't know about

We serve a sitemap index and one child sitemap per content type, each as a Next.js route handler using next-sitemap's getServerSideSitemap. Each child queries Payload directly for published docs:

TypeScript
// src/app/(frontend)/(sitemaps)/studies-sitemap.xml/route.ts
const results = await payload.find({
  collection: 'studies',
  overrideAccess: false,
  draft: false,
  // …
  where: { _status: { equals: 'published' } },
  select: { slug: true, updatedAt: true },
})
// …
export async function GET() {
  const sitemap = await getStudiesSitemap()
  return getServerSideSitemap(sitemap)
}

The query is wrapped in unstable_cache with a tag, so publishing a doc can revalidate its sitemap without a rebuild.

The gap with this pattern is any page that isn't a document. Our /blocks storefront and /docs are rendered from code, with no collection behind them, so no payload.find() ever returned them. Neither was in any sitemap until we added one for them. The blocks sitemap comment counts about 175 indexable pages, and the commit that added it notes the storefront and docs together earned 9 search impressions in three months. The fix is a sitemap built from the same lists the routes render from:

TypeScript
// src/app/(frontend)/(sitemaps)/blocks-sitemap.xml/route.ts
const paths = [
  '/blocks',
  ...categoryPaths,
  ...BUNDLE_SLUG_LIST.map((b) => `/blocks/${b}`),
  ...[...browsableBlockSlugs()].sort().map((s) => `/blocks/${s}`),
]

return getServerSideSitemap(paths.map((p) => ({ loc: `${SITE_URL}${p}`, lastmod })))

docs-sitemap.xml does the same with a fixed list of guide pages plus one entry per documented package, read from a build-time manifest so a package without docs is never advertised. Our pages sitemap also adds code-rendered hubs like /studies and /library by hand, for the same reason.

Now any route in src/app that isn't backed by a collection gets an explicit entry in some sitemap. A sitemap should also never list a URL that redirects or carries noindex. Ours leaves out /search and /posts (which 301s to /library/essays) for that reason.

The shadowing bug: a static file in front of a correct route

This is the one that cost us the most, and nothing about it looked broken.

We used next-sitemap in two ways at once. A postbuild script ran the CLI, which wrote a static public/sitemap.xml index from a list in next-sitemap.config.cjs. We also had a dynamic index at (sitemaps)/sitemap.xml/route.ts. Files in public/ are served ahead of app routes, so the static file answered every request for /sitemap.xml, and the route never ran.

When we added sitemaps for studies, series and chapters, we updated the route and not the config. The route was correct. The three child sitemaps were live and returning results the whole time. But the index Google actually read still listed only pages and posts. When we submitted the index in Search Console, it parsed to 20 URLs instead of 75. The missing 55 were the site's primary content: 13 studies, 4 series and 38 chapters. Search Console showed 37 pages indexed against 114 not indexed, and this was a large part of why.

The fix keeps both files and makes each one point at the other:

JavaScript
// next-sitemap.config.cjs
/**
 * KEEP IN SYNC with src/app/(frontend)/(sitemaps)/sitemap.xml/route.ts …
 *
 * `postbuild` writes a STATIC public/sitemap.xml from this list, and files in
 * public/ are served ahead of app routes, so the static index SHADOWS the
 * dynamic route entirely. …
 */
const CHILD_SITEMAPS = [
  'pages-sitemap.xml',
  'posts-sitemap.xml',
  'studies-sitemap.xml',
  'series-sitemap.xml',
  'chapters-sitemap.xml',
  // the /blocks storefront and /docs (code routes, no collection).
  'blocks-sitemap.xml',
  'docs-sitemap.xml',
]
// …
  exclude: ['/posts-sitemap.xml', '/pages-sitemap.xml', '/*', '/posts/*'],

The route carries the same array and a comment saying it's usually shadowed, so whichever one answers is right. The '/*' exclude stops next-sitemap crawling routes itself, since the route handlers own the URL lists.

The same thing happened with robots.txt. next-sitemap generated a static public/robots.txt that shadowed src/app/robots.ts and disagreed with its policy. We set generateRobotsTxt: false and let robots.ts own it. It advertises the index and all seven children from the same list.

If you use next-sitemap's postbuild alongside app-route sitemaps, check public/ after a build. Any generated file there with the same name as a route wins.

Nested URLs from one collection: section and pagePath()

Payload slugs are single segments, so a page that should live under /compare/ can't say so through its slug. We added a section select to Pages, in the sidebar next to the slug, and a single function that turns a page doc into its public path:

TypeScript
// src/constants/pageSections.ts
export const PAGE_SECTIONS = {
  compare: { path: '/compare', label: 'Compare (/compare/…)', crumb: 'Compare' },
  tome: { path: '/tome', label: 'Tome (/tome/…)', crumb: 'Tome' },
  trades: { path: '/websites-for', label: 'Trades (/websites-for/…)', crumb: 'Websites for' },
  // …
} as const

export function pagePath(doc: PageLike | null | undefined): string {
  const slug = doc?.slug || ''
  if (isPageSection(doc?.section)) return `${PAGE_SECTIONS[doc.section].path}/${slug}`
  return slug === 'home' || slug === '' ? '/' : `/${slug}`
}

The [slug] route, the section routes, the revalidation hook, the pages sitemap, the admin preview path, the link resolvers and the redirects resolver all call pagePath(), and that shared use is where the SEO value comes from. In the sitemap that's one line: loc: `${SITE_URL}${pagePath(page)}` . If two of those places build URLs differently, you get a sitemap URL that 404s, or a canonical that doesn't match the link Google followed. Adding a section is one entry here and one route folder.

Keep the select values to single words. They're also GraphQL enum values, which is why the URL comes from the path property.

Structured data: BreadcrumbList, FAQPage, Article

We emit JSON-LD from small server components that share one renderer. It escapes < so a title containing </script> can't break out of the tag:

TypeScript
// src/components/StructuredData/jsonLd.tsx
export function JsonLd({ data }: { data: Record<string, unknown> }) {
  return (
    <script
      type="application/ld+json"
      dangerouslySetInnerHTML={{ __html: JSON.stringify(data).replace(/</g, '\\u003c') }}
    />
  )
}

BreadcrumbJsonLd takes { name, path } items for nested routes and prepends Home. The crumb label in PAGE_SECTIONS feeds both the visible breadcrumb and this node, so they can't disagree. FaqPageJsonLd takes question and answer pairs, flattening a Lexical rich-text answer to plain text, and our FAQ blocks render it. ArticleJsonLd covers essays, chapters and case studies, and it leaves out any field whose source is missing. All of them reference a single Organization node by @id instead of restating it on every page.

Checking it

Build, then fetch /sitemap.xml and /robots.txt from the running site and compare them with your source lists. Then submit the index in Search Console and look at how many URLs it says it discovered. That number is how we found the shadowing bug: 20 discovered, when the child sitemaps held 75. After that, the Pages report shows which URLs are indexed and why the rest aren't, and that report is where a missing sitemap or a wrong canonical shows up first.

A checklist for your own Payload site

  • Every generateMetadata goes through one helper that sets alternates.canonical from the page's real path.
  • Each collection with public pages has a child sitemap that queries published docs only.
  • Code-rendered routes with no collection behind them are listed explicitly in some sitemap.
  • No sitemap lists a URL that redirects or carries noindex.
  • After a build, public/ holds no sitemap.xml or robots.txt that shadows an app route, or the two lists are kept in step and cross-referenced.
  • One function turns a doc into its URL, and the sitemap, links, redirects and revalidation all call it.
  • Your SEO plugin generateURL uses that same function.
  • Search Console's discovered count matches the URLs your child sitemaps actually return.

The sitemaps above cover the Tome block catalog at /blocks and the setup guide at /docs/get-started.

More in this series: Why every Payload page ships every block's CSS, and how we cut ours from 444 KB to 112 KB and Adding a custom block to Payload's layout builder (Payload 3 + Next.js).