AI crawler tracking

The PageDuel snippet runs in a browser. Crawlers do not execute JavaScript, so GPTBot, ChatGPT-User, Claude-User, Googlebot and the rest never run pd.js and never appear in your normal analytics. This traffic is not under-counted — it is invisible. The only place it exists is your own server logs, so crawler tracking needs a second, server-side install. Once your backend reports these requests, they show up under AI crawlers in the dashboard.

The three intents

Every recognized crawler is tagged with what the request is for.

Any other recognized bot is reported as other-bot. Requests from real browsers are discarded and never stored.

Get your write key

Open Settings → Integrations → Server-side tracking and choose Generate write key. Copy it into a server environment variable such as PAGEDUEL_WRITE_KEY.

The write key is not your snippet key. The snippet key is public and belongs in the <script> tag on your pages. The write key is a secret: it must stay on your server and must never appear in browser JavaScript, a client bundle, or a public repository. If one leaks, roll it from the same panel.

The same write key also authorizes server-side events on /api/track. Generating it does not conflict with a Stripe or Polar integration.

Send hits

Crawler hits are a plain HTTP POST, so any backend in any language can report them.

POST https://pageduel.com/api/crawler/hits
Authorization: Bearer YOUR_WRITE_KEY
Content-Type: application/json

The body is a batch of hits:

{
  "hits": [
    { "path": "/pricing", "statusCode": null, "userAgent": "Mozilla/5.0 (compatible; GPTBot/1.1; +https://openai.com/gptbot)", "ip": "203.0.113.10" },
    { "path": "/free-trial", "statusCode": 404, "userAgent": "ChatGPT-User/1.0", "ip": "203.0.113.11" }
  ]
}

A batch must contain between 1 and 500 hits. Send at most 50,000 batches per site per day.

A successful request returns 204 No Content with an empty body. You also get 204 when none of the hits in the batch were crawlers — PageDuel re-checks every User-Agent server-side, so a loose filter on your side is fine and costs you nothing.

Other responses: 400 for malformed JSON or a body that fails validation, 401 for a missing, malformed, or unknown write key, and 429 when the daily batch limit is exceeded.

Reporting should be best-effort. Never let a failed report break a page request, and never block the response while you wait for it.

Example: Next.js

Middleware runs before your route, so it can report traffic but cannot know whether the request ended in a 200 or a 404. Send statusCode: null here and handle 404s separately.

// middleware.ts
import { NextResponse, type NextRequest } from "next/server";

const CRAWLERS =
  /gptbot|chatgpt-user|claudebot|claude-user|perplexitybot|oai-searchbot|googlebot|bingbot|ccbot|bytespider|meta-externalagent/i;

export function middleware(request: NextRequest) {
  const userAgent = request.headers.get("user-agent") ?? "";

  if (CRAWLERS.test(userAgent)) {
    void fetch("https://pageduel.com/api/crawler/hits", {
      method: "POST",
      headers: {
        Authorization: `Bearer ${process.env.PAGEDUEL_WRITE_KEY}`,
        "Content-Type": "application/json",
      },
      body: JSON.stringify({
        hits: [
          {
            path: request.nextUrl.pathname,
            statusCode: null,
            userAgent,
            ip: request.headers.get("x-forwarded-for")?.split(",")[0]?.trim() ?? null,
          },
        ],
      }),
    }).catch(() => {});
  }

  return NextResponse.next();
}

To report the 404 half, make the same call from app/not-found.tsx with statusCode: 404. Middleware alone still gives you correct crawler traffic — you just will not see the content-gap list.

Example: Django

Django middleware wraps the response, so one call site covers both traffic and 404s.

# pageduel.py
import json
import re
import threading
import urllib.request

from django.conf import settings

CRAWLERS = re.compile(
    r"gptbot|chatgpt-user|claudebot|claude-user|perplexitybot"
    r"|oai-searchbot|googlebot|bingbot|ccbot|bytespider|meta-externalagent",
    re.I,
)
ENDPOINT = "https://pageduel.com/api/crawler/hits"


def _report(hit):
    request = urllib.request.Request(
        ENDPOINT,
        method="POST",
        data=json.dumps({"hits": [hit]}).encode(),
        headers={
            "Authorization": f"Bearer {settings.PAGEDUEL_WRITE_KEY}",
            "Content-Type": "application/json",
        },
    )
    try:
        urllib.request.urlopen(request, timeout=2).close()
    except Exception:
        pass  # reporting must never break a page request


class CrawlerReportingMiddleware:
    def __init__(self, get_response):
        self.get_response = get_response

    def __call__(self, request):
        response = self.get_response(request)
        user_agent = request.META.get("HTTP_USER_AGENT", "")

        if CRAWLERS.search(user_agent):
            forwarded = request.META.get("HTTP_X_FORWARDED_FOR", "")
            threading.Thread(
                target=_report,
                args=({
                    "path": request.path,
                    "statusCode": 404 if response.status_code == 404 else None,
                    "userAgent": user_agent,
                    "ip": forwarded.split(",")[0].strip() or None,
                },),
                daemon=True,
            ).start()

        return response

On a busy site, buffer hits and send them in batches of up to 500 instead of one request per hit.

The 404 content-gap report

statusCode is what powers the content-gap list. Report null for a normal request and 404 when the request did not resolve to a page. PageDuel then shows you every path a crawler asked for that does not exist.

Each of those paths is a page an AI assistant expected you to have. If ChatGPT-User keeps requesting /free-trial and you do not have a /free-trial page, that is not an error to fix in your router — it is a page to write.

Only null and 404 appear in the report. Other status codes are accepted and stored, but they are counted in neither the traffic total nor the gap list.

Reading the report

The crawler report is bucketed by UTC calendar day, not your site's local time zone. Any UTC day your selected range touches is counted in full, so the report can cover more than the range you picked. The dashboard says so above the numbers.

The verified share tells you what percentage of hits came from an IP inside a vendor's published range. Several crawlers — ClaudeBot, meta-externalagent, Amazonbot, Bytespider — publish no machine-readable range file, so they are self-declared by User-Agent and always show as unverified.

What is not tracked

• Nothing is collected automatically. This endpoint is the only source of crawler data. If your backend does not report a hit, PageDuel never sees it.
• Crawler hits do not count against your monthly event tier. They are metered and stored separately from visitors and events, so turning this on will not raise your bill.
• No crawler IP or raw User-Agent is stored. The IP is used at ingest to check the vendor range and then discarded; the User-Agent is reduced to a canonical crawler name.
• Existing analytics are unchanged, with one exception: Claude-User is now treated as a crawler rather than a human visitor, so a site receiving it will see a small dip in visitors and pageviews. That traffic appears here instead, under answer.

Related: server-side events and write keys.