Keep /hm/ pages out of crawler traffic

Problem

A user received a Google Alert that linked to their profile on another Seed site's domain. The alert included text from a comment in their own space. The user was comfortable with the result, but wanted the team to understand how Google made the connection.

Seed can serve content from other spaces through /hm/ URLs. This can cause search results to associate a profile or comment with a different site's domain. We have not confirmed how Google discovered the reported URL; following a relationship between accounts is one possible explanation.

Solution

Apply the same crawler rules to sites and gateways. Both will serve Disallow: /hm/ in robots.txt, asking compliant crawlers to stop fetching these paths. Previously, gateways allowed general /hm/ crawling.

Keep the two image endpoint exceptions so link previews can load their images. Keep the existing restrictions on profile, comment, and other special views. Members-only sites continue to block all crawling. People can still open public /hm/ links in a browser.

Scope

The code and regression tests are complete locally: eight tests, the web typecheck, and formatting checks passed. The next step is to deploy the web app and check /robots.txt on each affected domain. A release date has not been set.

Removing existing Google results is a separate step. robots.txt does not guarantee removal from search results. Google must be able to fetch a page to read a noindex instruction, so removal needs a separate rollout and, if urgent, a temporary Search Console removal request. See Google's noindex guidance.

Rabbit Holes

    Reconstructing Google's exact discovery path is not required to apply this policy.

    A wider review of search indexing and duplicate content can follow separately.

No Gos

    Do not block people from opening public links.

    Do not remove the image exceptions needed for previews.

    Do not describe crawler rules as privacy or access controls: bots can ignore them.

    Do not claim that this change removes existing search results.

Do you like what you are reading? Subscribe to receive updates.

Unsubscribe anytime