AGC (auto-generated content) websites can pick up posts from feeds, HTML, or crawlers and republish them without permission. However, blocking an IP or ASN from quick observation doesn’t necessarily stop perpetrators: a single address can be shared by many legitimate services and bots. In 2026, mitigation is more likely to be achieved through log evidence, Cloudflare Security Settings/WAF, endpoint protection, and copyright enforcement.
This change preserves the context of the author’s experience as well as the original image. Technical details that are risky, unverified, or potentially cause false positives are not presented as steps to be followed blindly.
| Problem | Main solutions | What to check |
|---|---|---|
| Aggressive scraping | WAF, bot settings, rate limits | Logs and legitimate traffic |
| Copied content | Proof of copyright, host, and Google | Specific URLs and permissions |
| Incorrect blocks | Review Security Events | Verified bots, API, visitors |
What is AGC content theft and how to prove it?
AGC does not always mean violation. Sites that create summaries with permission are different from full copying without a license. Compare dates, identical paragraphs, images, and the original source. data crawling and process crawling explain the crawling mechanism. Don’t accuse an IP just because it appears in the access log.
1. Collect server logs and evidence before blocking
Save the UTC/WIB timestamp, accessed URL, response code, user agent, request frequency, and evidence of copied content. Differentiate search bots that are useful for how crawlers work and indexing from automated access that wastes resources. Good documentation will help DMCA reporting to Google if legal reporting is needed.
2. Enable Cloudflare bot protection available on your plan
In the Cloudflare dashboard, check the Security Settings and Bot traffic categories. Bot grouping, Super Bot Fight Mode, and Bot Management features vary across plans. Granular bot scoring requires an Enterprise Bot Management subscription. Use Managed Challenge or logging first, rather than Blocking entire countries or ASNs.
3. Create limited custom rules based on evidence
Set rules only for attacked paths, unverified traffic, and proven patterns. View Security Events before and after. Test the impact on logins, REST APIs, cron, and dynamic websites. For basic reinforcement, audit optimal Cloudflare configuration and its relationship to handling DDoS during a surge in requests.
4. Don’t block Googlebot and legitimate visitors
Incorrect challenges can hinder verified crawlers, monitoring services, and potential customers. Cloudflare provides a Verified bots category; exceptions depend on the plan and features used. Compare the results with technical SEO and XML sitemap, then ensure that important URLs remain indexable.
5. What is the role of Wordfence and origin protection?
Wordfence can provide a layer of security for WordPress, but the IP rules on the server must take into account reverse proxy and real client IP configurations. Cloudflare origin security explains the need to secure the origin; don’t use third-party IP lists as a blocklist without auditing. Smart rate limiting is more resilient than an address block war.
6. SEO and copyright follow-up
Add canonical URL for your own URL variations, keeping SEO-friendly articles and E-E-A-T, and monitor monitor SEO traffic. If theft is proven, use DMCA report to Google and report it to the violating host. For videos, the guide report YouTube reuploads has a different mechanism.
7. How to evaluate after the rule is implemented
Compare request rates, error rates, bot traffic and conversions before and after. If many visitors fail to access the site, roll back the rule aggressively. Focus on the reader experience according to search intent, not claims that one firewall can wipe an entire AGC site off the internet.
AGC incident response procedures without blocking genuine visitors
When scraping traffic spikes, start by taking a log snapshot and identifying patterns. Check to see if requests are arriving to the same page at an unusual frequency, generating repeated errors, or putting pressure on database resources. Don’t use user-agent as the sole evidence, as it’s easily forged.
Test the rule on the smallest scope first
If Cloudflare provides Log or Managed Challenge mode as part of your plan, start with the easiest actions to evaluate. Focus on the endpoints that are actually being abused. Once the rule is activated, open a public page, log in to the admin, submit a form, run a WordPress function that requires a loopback, and check with verified search engine bots. Make sure blocking does not cause a decrease in indexability.
| Indicators | Interpretation | Response |
|---|---|---|
| Requests increase from one ASN | Not necessarily one perpetrator | Check IP, path, time, and additional evidence |
| High challenge on real visitors | False positives may occur | Narrow the rule conditions |
| Googlebot is hampered | Technical SEO risk | Evaluate verified bots categories and exceptions |
| AGC is already Publish a copy | Separate copyright issues | Save evidence and report to host/Google |
Save the rule number, reason for change, date, before-after measurements, and rollback steps. A WAF is about controlling access to your own website, not a tool that removes posts from other servers. Rightsholders need to address both issues—server protection and copy distribution—through the proper channels.
Image archive and original tutorial experience
The following screenshots are from a scraping case the author recounted in 2020. Historical IP/ASN attribution should not be considered evidence against the current address user.
See 10 original screenshots
FAQ: how to block content-stealing AGC websites
1. Is AGC always illegal?
No. The compliance rate depends on the permission, usage method, and applicable law.
2. What is the difference between legitimate crawling and unauthorized scraping?
Crawling can be done for legitimate indexes; copying without a license can violate copyright.
3. Can Cloudflare Free block bots?
There are protection features, but the level of control depends on the package.
4. Is Bot Score available on all plans?
No. Granular scoring is a specific Bot Management feature.
5. What are the risks of blocking a single ASN?
You can block legitimate users because many services share ASNs.
6. Is Wordfence enough?
Not necessarily; combine WordPress and edge security with the right configuration.
7. Is it okay to block Googlebot?
Don’t intentionally hinder legitimate crawlers because it hurts the index.
8. What are the important pieces of evidence for scraping?
Request logs, timestamps, original and copied URLs, and proof of ownership.
9. Does the WAF remove scraped articles?
No. The WAF protects your site from access, not deleting files on other sites.
10. How to report copyright infringement?
Use Google’s legal channels and report to the violating host.
11. How long to monitor new rules?
Monitor before and after the rule, including traffic and errors.
12. How to check for false positives?
Test verified crawlers, real visitors, forms, and APIs.
