Sitemap, robots and hreflang from one resource

The seo resource took over the whole search-engine surface of an app: sitemap.xml, robots.txt, the hreflang alternates and the head meta.

It started narrower, as a resource called sitemap, and the rename was the point at which it became clear these are not separate jobs. A sitemap needs the page list. Hreflang needs the page list and the languages. The canonical needs the current URL. The head meta needs the title and description. Every one of those was already declared somewhere in the app, so the resource reads them rather than asking for them again: %app->pages, %app->langs, %app->title, %app->description.

The most useful part is that robots.txt is generated and safe by default. Unless an entrypoint declares indexable, the resource serves Disallow: /, which means a dev or stage node is de-indexed simply by not saying otherwise. Only production sets the flag:

phlo_app(id: 'Site', host: 'example.com', indexable: true, ...);

With it set you get Allow: /, one Disallow: line per entry in %app->robotsDisallow, and a Sitemap: line. Because it comes from a route, there is no static robots.txt to maintain and none to accidentally deploy from the wrong machine.

Every app on this server adopted it within two days, and the static robots files disappeared the day after.

Engine 1.0. See the SEO chapter.