How to automate competitor monitoring

What this automation does

Instead of someone remembering to check five competitor sites every Friday, the system watches public sources and tells you only what changed and why it might matter.

1. Source list. For each competitor, a config table lists the URLs to watch and the method for each: RSS feed, sitemap, page snapshot, or a licensed data API.
2. Polite fetch. On schedule, the workflow checks robots.txt, fetches each allowed page once with an honest user agent, and backs off on errors. Disallowed URLs are skipped and logged.
3. Extract and normalize. Page HTML is reduced to the meaningful text: plan names, prices, feature lists, headings and body copy. Dates, counters, cookie notices and rotating elements are removed.
4. Diff. The new text is compared with the last stored snapshot. Unchanged pages stop here.
5. Judge. Changed sections go to an LLM with the competitor’s context and your positioning. It classifies the change, rates its importance, and writes one or two sentences on what changed.
6. Digest and alert. Material changes are collected into a weekly digest in Notion and Slack. Pricing changes alert the sales channel the same day.

When it is worth building

It is worth building when competitive information shapes deals or roadmap decisions and the current process is ad hoc, such as a rep mentioning on a call that a competitor dropped their price last month. If losing a deal to a pricing change you did not know about has happened more than once, automated monitoring pays for itself.

If you have two competitors and a curious product marketer, set up RSS in a feed reader and Google Alerts and stop there. Commercial competitive intelligence platforms are the better answer if you need broad coverage of dozens of competitors, review-site tracking and battlecard management in one product. A custom build fits teams who want a short, high-signal list of sources, their own judgment criteria and output in tools they already use.

Respecting site rules

Build these rules into the code, not into a policy document:

• Obey robots.txt. Check it before each run and skip disallowed paths, even if the page is public.
• Follow terms of service. If a site’s terms prohibit automated access, remove it from automated monitoring and use a manual check or licensed source.
• Stay public. Never log in to a competitor’s product, use a trial account to scrape the app, or access gated content.
• Keep the load low. One request per page per check, spread out over time, with a clear user agent and contact address.
• Do not evade blocks. If a site blocks you with a captcha or bot protection, stop. Rotating proxies to get around it is where monitoring turns into a problem.

Architecture

How to build it

1. Choose sources deliberately. Pricing page, changelog or release notes, blog feed, careers page and press page per competitor is a strong start. Five sources each beats fifty.
2. Record the rules per source. Note the robots.txt status and any terms that apply, and the check frequency. Review this list quarterly.
3. Write extractors. Use a generic text extractor for most pages and a specific one for pricing pages that pulls plan names, prices and limits into structured fields.
4. Store snapshots and diff. Keep every snapshot. Diff at the section level so the model sees what changed in context.
5. Write the judgment prompt. Describe your product, your main differentiators and what counts as material. Ask for a category, an importance score and a plain description of the change.
6. Build alerts and the digest. Same-day Slack alerts for pricing and packaging; a weekly digest grouped by competitor for everything else.
7. Add feedback. Let readers mark items useful or noise. Use that monthly to tune the noise filters and the prompt.

What breaks in production

• JavaScript-rendered pages. Some pricing pages load prices with scripts and return empty HTML. A headless browser can render them, but only where site rules allow automated access.
• A/B tests and geo pricing. You may see a test variant or a regional price. Note the fetch location and flag one-off changes for confirmation before alerting sales.
• Redesigns. A full site redesign makes every page look changed. Detect large structural diffs and summarize them as one event.
• Blocked fetches. If a site starts returning 403s or a challenge page, stop checking it and alert the owner, rather than retrying.
• Alert fatigue. If the digest is long every week, people stop reading it. Tighten what counts as material before adding sources.

Timeline and cost

Monitoring for up to ten competitors with five sources each, diffing, AI change judgment, Slack alerts and a Notion digest usually takes one to three weeks. Lumeor builds this in the range of $4,000 to $10,000, depending on the number of competitors and sources, how many pricing pages need custom extractors, and whether SEO data is included. Running costs are hosting plus small model usage, and any SEO data subscription you license directly. Our n8n automations suite case study shows the same ingest, classify and deduplicate approach applied to high-volume messaging data. To turn insight into content, pair this with SEO content briefs.

How to measure success

• Share of digest items readers mark as useful
• Days between a competitor pricing change and your team knowing about it
• Hours per week previously spent checking sites by hand
• Battlecard or sales talk track updates triggered by the digest
• Number of sources skipped for robots or terms reasons, reviewed so coverage gaps are deliberate