LEADS SEO Automated Auditor (SnifferBot/2.0)
A single-file PHP crawler that audits websites for technical and on-page SEO, extracts company profiles, and stores every result in flat MySQL columns. Runs from cron. Never executes JavaScript. Enforces RFC 9309 robots.txt compliance.
Quick Facts
| Bot name | SnifferBot/2.0 |
|---|---|
| Language | PHP 7.4 or 8.x, command line only |
| Execution | Cron. One batch per invocation. Single-threaded. |
| Database | MySQL or MariaDB through PDO. Prepared statements only. |
| Pages per site | 5 by default, configurable via LEADS_MAX_PAGES |
| Sites per run | 10 by default, configurable via LEADS_BATCH_SIZE |
| Score range | 0 to 100 per page. Site score is the mean of page scores. |
| Checks per page | 11 technical and on-page SEO checks |
| JavaScript | Not executed. Only the raw HTML the server returns. |
| Dependencies | None. No Composer. No Node. No headless browser. |
What the Auditor Does
The auditor takes a list of URLs from the ph_web table, crawls up to five pages per site, runs eleven SEO checks on each page, and writes every finding into leads_page_findings as flat columns. Separately, it pulls company details (name, phone, address, about text) from About, Contact, and Footer pages and writes them into leads_company_profiles.
Every fetch identifies itself as SnifferBot/2.0. Every fetch checks robots.txt first. Every fetch blocks private, loopback, link-local, and cloud metadata IP ranges before sending a request. Every fetch has a hard 2 MB response cap. Every fetch has a timeout.
The auditor does not render JavaScript. It does not measure Core Web Vitals. It does not collect backlinks. It does not track keyword rankings. It reads static HTML and reports what the raw response contains.
How to Run the Auditor
- Apply the schema migrations from
migrations/2026_11_02_0001_create_leads_auditor_tables.sql. This createsleads_audit_runs,leads_page_findings, andleads_company_profiles. - Confirm your global bootstrap provides
$pdo. The auditor does not define database credentials and does not include any file. - Run
php leads_auditor.php --check. This prints the effective configuration, confirms the four required tables exist, and counts pending rows. Fix anything reported as missing or unwritable before moving on. - Run
php leads_auditor.php --limit=1to audit a single row. Queryleads_page_findingswherelisting_idmatches that row to confirm flat columns are populated. - Add the cron entry:
*/10 * * * * /usr/local/bin/php /path/leads_auditor.php >> /path/cron.log 2>&1
Features
Internal link discovery
Extracts every same-origin <a href> target from the current page. Breadth-first crawl capped at depth 5 by default via LEADS_DEPTH_LIMIT.
www and apex treated as the same site
example.com and www.example.com match. Without this, sites that redirect one to the other lose every internal link because the host string differs.
Per-domain page cap
Five pages per domain by default, enforced across the whole batch. Prevents one large site from consuming the crawl budget of ten smaller ones.
Sitemap fallback seeding
When the root page yields fewer than two crawlable links, the queue is seeded from sitemap.xml. Needed for sites whose navigation is rendered by JavaScript with no static anchors.
Manual redirect handling
Redirects followed one at a time. Every redirect target revalidated against the SSRF blocklist. Loop detection by visited set. Maximum 5 hops.
2 MB response cap
cURL aborts mid-stream when the response exceeds 2 MB. Protects against zip bombs, memory exhaustion, and servers that never close the response.
Only 2xx audited
A page is audited only when the HTTP status is 200 through 299. 3xx without a Location header, 4xx, and 5xx are skipped. No findings written for them.
Row timeout
150 seconds per row by default via LEADS_ROW_TIMEOUT. A slow site cannot block the rest of the batch.
Batch timeout
540 seconds per batch by default via LEADS_RUN_BUDGET. Unprocessed rows return to the queue for the next cron tick. Prevents overlapping cron executions.
Per-origin rate limit
2 seconds minimum between requests to the same origin, combined with the site's own crawl-delay using max(). Keeps the bot inside what servers already expect.
Stuck row recovery
Rows stuck in claimed state for more than 15 minutes reset to done, but only if no active audit run exists for that row. Cross-checks leads_audit_runs so live crawls are never interrupted.
Automatic pruning
Audit runs older than 90 days deleted in chunks of 1000. Child rows deleted explicitly in the same transaction. Does not rely on foreign key cascade behaviour.
Signal handling
SIGTERM and SIGINT caught and logged. In-flight rows recovered by the next run's stuck-row pass.
Fatal error logging
PHP fatal errors captured with file and line number. Every crash leaves a record.
Preflight check
--check verifies database connection, confirms four tables exist, counts pending rows, prints effective configuration, and checks log file writability.
Robots.txt Compliance
The auditor follows RFC 9309. Every URL passes through the robots checker before any page fetch. A site that disallows SnifferBot/2.0 is never crawled.
| Behaviour | Implementation |
|---|---|
| User-agent match | SnifferBot/2.0 matched against User-agent: blocks. Wildcard * and substring matching supported. |
| Group parsing | Consecutive User-agent: lines form one group. Rules apply to all agents listed in that group. |
| Longest match wins | Longer rule pattern takes precedence. Ties resolve to last rule seen. |
| Dollar anchor | Trailing $ treated as end-of-string anchor, following Google convention. |
| Wildcard matching | Two-pointer linear-time algorithm. No regex. No backtracking. No ReDoS surface. |
| Crawl-delay | Honored. Multiple crawl-delay values combine as the maximum. |
| Missing robots.txt | All paths allowed, per RFC 9309 section 2.3.1.4. |
| Pre-flight | robots.txt fetched and parsed before the first page request. Disallowed seeds abort the row immediately. |
Per-Page Data Extracted
Every crawled page produces these fields. Column names match the leads_page_findings table.
| Field | Source | Stored in column |
|---|---|---|
| Title | <title> element text | title |
| Title length | Character count of title | title_len |
| Meta description | <meta name="description" content> | meta_desc |
| Description length | Character count of meta description | desc_len |
| H1 text | First <h1> text, or "No H1 Found" | h1 |
| Word count | Words in body text after removing script, style, and noscript tags | word_count |
| Missing alt text | Count of <img> elements with empty or missing alt | missing_alts |
| Canonical URL | <link rel="canonical" href> | Used for scoring only |
| Noindex flag | noindex in robots meta | Used for scoring only |
| Schema types | All @type values from JSON-LD blocks | Used for scoring only |
| HTTP status | cURL response code | http_status |
| TTFB | Time to first byte in milliseconds | ttfb_ms |
| Load time | Total response time in milliseconds | load_time_ms |
Company Profile Extracted
Company data is pulled from About, Contact, Footer, and Profile pages. Every field is validated. Garbage is rejected, not stored.
| Field | Source and validation |
|---|---|
| Company name | JSON-LD Organization.name, then og:site_name, then first header link, then H1. Rejects domains, URLs, page titles like "Home" or "About Us", social names like "Facebook", and taglines. |
| Phone | JSON-LD telephone, then tel: link, then class or itemprop match. Rejects strings longer than 8 characters surrounding the number. Validates PH mobile (+63 9XX XXX XXXX) and landline (+63 2X or 3X or 8X) patterns. |
| About text | First three paragraphs over 50 characters from article, main, or about containers. Excludes nav, footer, aside, and product card paragraphs. Rejects paragraphs containing newsletter, cookie, copyright, or add-to-cart markers. |
| Country | Itemprop addressCountry, then JSON-LD, then phone dial-code inference. Mapped through 15-country table. |
| Service areas | JSON-LD areaServed only. Never from DOM. Empty when the site has no structured data. |
| Services | JSON-LD hasOfferCatalog only. Never from DOM navigation menus. Empty when the site has no structured data. |
| Address | JSON-LD address object, then address class, then footer address element. Length bounded 10 to 300 characters. |
| Year founded | Regex on page text for founded, established, or since followed by a 4-digit year between 1800 and now. |
| Source pages | JSON array of URLs that contributed to the profile. |
A company profile row is written only when at least two of four primary fields are populated: name, phone, about text, address. Rows that would be all NULL are skipped. An empty table is better than a table of navigation menus pretending to be company names.
Scoring Method
Every page starts at 100. Each failed check subtracts a fixed penalty. The final score is clamped between 0 and 100. Two numbers exist: the page score and the site score. The site score is the mean of all page scores in that run, rounded to the nearest integer.
The 11 checks and their deductions
| # | Check | Fires when | Points |
|---|---|---|---|
| 1 | Robots.txt | robotsAllowed is false | -20 |
| 2 | Sitemap | No valid sitemap.xml at origin | -15 |
| 3 | Indexability | robots meta contains noindex | -25 |
| 4 | Canonical | No canonical link element | -15 |
| 5 | Schema markup | No JSON-LD @type found | -15 |
| 6 | Title length | Under 40 chars deducts 20. Over 60 chars deducts 10. 40 to 60 deducts 0. | -20 or -10 |
| 7 | Heading hierarchy | Zero H1 deducts 15. Multiple H1 or zero H2 deducts 10. | -15 or -10 |
| 8 | Meta description | Under 120 chars or over 160 chars | -15 |
| 9 | Image alt text | Proportional: 10 multiplied by missing alt count divided by total images | 0 to -10 |
| 10 | TTFB | Over 800 milliseconds | -10 |
| 11 | Total load time | Over 3000 milliseconds | -10 |
Maximum possible deduction: 170 points. The clamp to zero means anything below -70 is indistinguishable from -170.
If robots.txt disallows the URL, the crawler exits before scoring begins. The row is marked aborted_robots. The -20 deduction is dead code, retained in case the robots state somehow changes between pre-flight and audit.
Why each check exists
| Check | Reason | Standard |
|---|---|---|
| Robots.txt | Disallowed pages produce no SEO value | RFC 9309 |
| Sitemap | Absence correlates with weak indexation hygiene | Google sitemap guidance |
| Indexability | noindex blocks ranking | Google Search Essentials |
| Canonical | Missing canonical risks duplicate content dilution | Google canonical guidance |
| Schema markup | JSON-LD enables rich results and AI citation | Schema.org, Google structured data |
| Title length | 40 to 60 fits inside SERP display without truncation | Google SERP display guidance |
| Heading hierarchy | Single H1 with H2s improves topical clarity | W3C heading semantics |
| Meta description | 120 to 160 avoids truncation while giving context | Google SERP snippet guidance |
| Image alt text | Accessibility and image search | WCAG 2.1 Level A |
| TTFB | 800 ms is Google's good threshold | Web.dev TTFB thresholds |
| Load time | 3000 ms is Google's good LCP threshold | Web.dev LCP thresholds |
What the score does not include
- Word count. Stored but not scored. Content length is not in the eleven checks.
- HTTP status. Non-2xx pages are skipped before scoring runs.
- Actual TTFB and load values. Only the two thresholds matter. A 799 millisecond TTFB and a 200 millisecond TTFB score identically.
- H1 count beyond the zero and multiple rules. Two H1s deducts the same as ten H1s.
- Absolute image count. Only the missing-alt ratio matters.
- Company extraction quality. Completely separate from page score.
When title, meta description, H1, canonical, and JSON-LD are injected by React, Vue, or Angular after page load, the raw HTML is a shell. The auditor sees empty values and applies full deductions. A well-optimised site can score 20 to 40 for this reason. This matches what Googlebot sees on the first pass. Google renders JavaScript in later passes. This auditor cannot. That is a documented limitation, not a bug.
Security Model
SSRF protection
IPv4 and IPv6 CIDR blocklists reject private, loopback, link-local, and cloud metadata ranges. DNS resolved once, validated, then pinned via CURLOPT_RESOLVE. DNS rebinding between validation and fetch cannot occur. Every redirect target revalidated.
XXE and entity expansion
Every DOM and XML parse uses LIBXML_NONET. Network access blocked during parse. LIBXML_NOENT and LIBXML_DTDLOAD never enabled. External entities cannot be resolved.
ReDoS prevention
robots.txt wildcard matching uses a two-pointer linear-time algorithm. No regex. No backtracking. A malicious robots.txt cannot freeze the crawler.
SQL injection
Every database operation uses PDO prepared statements with named placeholders. Two integers (recovery window and prune chunk limit) are inlined after explicit integer casting. Safe by construction.
CLI-only gate
The script exits with HTTP 403 if not invoked from the command line. This check runs before any include, any database access, any file operation. Cron is the only valid invocation path.
Bot identity
Every request sends SnifferBot/2.0 (+https://yourdomain.com/bot). Server logs identify the bot. WAF rules can block it. The bot does not spoof user agents or rotate them.
Host blocklist
Operator domains listed in Cfg::BLOCKED_HOSTS cannot be audited. Prevents self-scoring. Blocked rows marked unscannable before any fetch.
Resource caps
Memory capped at 512 MB. Response body 2 MB. JSON depth 32. cURL low-speed abort at 256 bytes per second over 15 seconds. Row timeout 150 seconds. Batch timeout 540 seconds.
Compliance
| Standard | Status |
|---|---|
| RFC 9309 Robots Exclusion Protocol | Full compliance. Group semantics, longest-match, crawl-delay, missing file means allow. |
| RFC 9309 section 2.2.1 User-Agent format | Compliant. Name, version, contact URL present. |
| HTTP caching headers | Not honored. Responses fetched fresh every run. Cache-Control, ETag, If-Modified-Since all ignored. |
| Conditional GET | Not implemented. No If-Modified-Since or If-None-Match sent. |
| Accept-Language negotiation | Not implemented. |
Configuration
Every operational setting is an environment variable. Missing values fall back to defaults and are logged. Silent default application never happens.
| Variable | Default | Range | Controls |
|---|---|---|---|
LEADS_UA | SnifferBot/2.0 (...) | any string | User-Agent sent with every request |
LEADS_MAX_PAGES | 5 | 1 or more | Pages per domain per batch |
LEADS_MIN_INTERVAL | 2.0 | 0 or more | Seconds between requests to same origin |
LEADS_RUN_BUDGET | 540 | 30 or more | Batch time budget in seconds |
LEADS_ROW_TIMEOUT | 150 | 30 or more | Per row time limit in seconds |
LEADS_BATCH_SIZE | 10 | 1 to 500 | Rows pulled per cron run |
LEADS_RETENTION_DAYS | 90 | 1 or more | Age before audit runs are pruned |
LEADS_PRUNE | 1 | 0 or 1 | Toggle pruning |
LEADS_DEPTH_LIMIT | 5 | 1 or more | Internal link depth cap |
LEADS_MAX_REDIRECTS | 5 | 1 or more | Redirect chain limit |
LEADS_CONNECT_TIMEOUT | 10 | 1 or more | cURL connect timeout |
LEADS_TOTAL_TIMEOUT | 30 | 1 or more | cURL total timeout |
LEADS_RECOVER_MINUTES | 15 | 1 or more | Stuck row recovery window |
LEADS_LOG_LEVEL | INFO | DEBUG, INFO, WARN, ERROR | Log verbosity |
LEADS_LOG_FILE | none | file path | Log destination. If unset, writes to stdout and stderr. |
Data Flow
One cron run walks through six stages. Every row and every page follows the same order.
Stage 1: Startup
main() runs recoverStuck() to release abandoned rows, then pruneOldRuns() to delete expired history, then selects up to LEADS_BATCH_SIZE pending rows from ph_web.
Stage 2: Row claim
The row is atomically claimed via UPDATE ph_web SET status='claimed'. A new audit run is created. Robots.txt is fetched and parsed. If disallowed, the row aborts here.
Stage 3: Crawl
Queue seeded with root URL. Each page fetched if 2xx, audited, and stored. Internal links added to queue until depth limit or page cap reached. Sitemap fallback fires when the root yields too few links.
Stage 4: Score and store
PageAuditor::audit() runs the eleven checks and returns the signals. DbWriter::saveFinding() writes flat columns into leads_page_findings with listing_id, run_id, and position.
Stage 5: Company profile
After crawl, Company::extract() pulls structured data from About, Contact, and Footer pages. saveCompany() writes the row only if two of four primary fields are populated.
Stage 6: Finalize
finishRun() writes pages crawled, site score, and final URL to the audit run. markDone() sets ph_web.status='done' and stamps last_audited_at.
Error Handling
| Situation | Run status | ph_web status |
|---|---|---|
| Robots.txt disallowed the URL | aborted_robots | unscannable |
| Host in blocklist | aborted_blocklist | unscannable |
| SSRF guard rejected the URL | failed | unscannable |
| Zero pages returned 2xx | failed | unscannable |
| Any other exception | failed | unscannable |
| Success | completed | done |
If the process is killed by the OS, the next cron run's recovery pass resets any claimed row older than the recovery window. That row is then available for re-audit. Stale audit run rows remain as running in the audit table. They do not block future audits of the same site.
Logging
Every log entry has four parts: an ISO 8601 timestamp with timezone, a level label, a plain message, and a JSON object with structured context.
| Level | Used for |
|---|---|
| DEBUG | Per page audit outcome, environment fallback, non-2xx skip, robots fetch failure |
| INFO | Batch start and end, row start and complete, sitemap seed, prune completion, quota hit, blocklist skip |
| WARN | Row timeout, budget exceeded, robots abort, fetch failure, stuck row recovery, signal received |
| ERROR | Zero pages fetched, row exception, fatal DB error, prune chunk failure, missing table |
[2026-11-02T14:37:02+08:00] INFO batch start {"rows":10,"limit":10}
[2026-11-02T14:37:02+08:00] INFO row started {"row_id":72,"run_id":33,"url":"https://smart.com.ph"}
[2026-11-02T14:37:03+08:00] INFO row completed {"url":"https://smart.com.ph","pages":1,"score":5}
[2026-11-02T14:37:27+08:00] ERROR row failed: 0 pages {"url":"https://pldt.com.ph"}
Limitations
These are design trade-offs. They are not defects. Each one has a reason.
- No JavaScript execution. Requires a headless browser such as Puppeteer or Playwright, which needs Node and Chromium. Not compatible with a single-file PHP deployment on cPanel.
- Single-threaded. One row at a time. Parallel requests need either curl_multi or a job queue. Both add complexity and increase WAF exposure from parallel egress.
- No proxy rotation. Would require a proxy pool with billing. Out of scope for a single-file tool.
- Simple mean site score. One low page drags the site down. No weighting by traffic, importance, or depth.
- 5-page default cap. A deep e-commerce site is not fully audited. The cap is there to keep ten sites finished inside one batch window.
- 2 MB response cap. Unusually large pages get truncated. The cap is there to prevent memory exhaustion.
- No duplicate prevention in the seed table. If
ph_webhas three rows for the same host, the first one consumes the 5-page quota and the other two hit the domain quota skip. Duplicate seed rows waste batch slots. - Stale audit run rows accumulate. Rows with
status='running'that never finished are not cleaned automatically. They do not block future audits. Cleanup would need a separate job.
Questions
How many sites does one run audit?
Ten by default. Configurable from 1 to 500 via LEADS_BATCH_SIZE. The effective ceiling is often lower than the batch size because each site takes 20 to 60 seconds and the batch has a 540-second budget.
How deep does it crawl?
Five pages per domain by default. The limit is enforced across the entire batch, not per row. A second row pointing at the same host is skipped with a domain quota message.
Does it execute JavaScript?
No. It fetches the raw HTML the server returns. React, Vue, and Angular pages that inject title, meta, or content on the client side will be audited as empty shells.
Does it respect robots.txt?
Yes. Fully. RFC 9309. Pre-flight check before any page fetch. If SnifferBot/2.0 is disallowed for a URL, that URL is never fetched.
What User-Agent does it send?
SnifferBot/2.0 (+https://yourdomain.com/bot) on every request.
How do I re-audit a site?
Set last_audited_at = NULL on the row. The next cron run picks it up.
Why is a JavaScript-heavy site scoring 5 out of 100?
Because the raw HTML contains no title, no meta description, no H1, no canonical, and no JSON-LD. All eleven checks that look for these fail. The site may rank perfectly in Google, but this auditor reads static HTML only.
What if a row never finishes?
It stays claimed for 15 minutes by default. After that, the recovery pass resets it to done if no active audit run is attached. Then it can be audited again.
Where are the results stored?
Three tables. leads_audit_runs holds one row per site audit. leads_page_findings holds one row per page. leads_company_profiles holds one row per site for company data. Every field in page findings is a flat column, not JSON.
Can a third party verify a finding?
Yes. Every finding has the URL, the HTTP status, and the timestamp. Anyone with access to the source page and the same eleven checks can reproduce the score.
Boundaries
These statements describe what the auditor does not do. They are deliberate.
- The score reflects the raw HTML returned to the crawler, not what a browser renders. Sites that rely on client-side rendering will score lower than their real SEO posture.
- The eleven checks are a fixed heuristic. They are not Google Lighthouse. They are not Core Web Vitals. They are a proprietary scoring model aligned with published Google guidance on the specific points each check covers.
- Company extraction only fires when the site publishes structured data or obvious footer phone links. Sites without JSON-LD
OrganizationorLocalBusinessmarkup will often produce a name and a phone but nothing else. - Service lists and service areas come only from JSON-LD. If the site does not publish them, the columns stay empty. This is a design choice. A column full of navigation menu items pretending to be services is worse than an empty column.
- The auditor is not a search engine rank tracker, backlink checker, keyword rank monitor, or accessibility certifier. Those are separate tools.