Rate Limiting and Bot Protection for NZ Sites

Quick answer

Most "attacks" against an NZ school or small-business site are not attacks — they are automated scan-probing from bots that crawl the public web looking for known vulnerabilities. You do not need to defend every probe like an active attacker; you need to make your site expensive to scan, log the sources, and rate-limit connections that exceed normal human browsing patterns. The two workhorse tools are nginx's limit_req module (per-IP request rate cap) and fail2ban (log-driven automatic IP blocking after a threshold of suspicious requests).

For the headers that complement rate limiting — CSP, HSTS, Referrer-Policy — see the Security Headers Explained guide. For a free scanner that classifies the traffic hitting your site as human, bot, or scanner, run the Free Website Security Check.

Why scan-probing is not an attack — and why that matters

Almost every New Zealand school or small-business website that ever appeared in a CERT NZ or NCSC advisory was reached by an automated scanner, not a targeted attacker. The bots crawl the internet looking for known vulnerabilities — WordPress plugins with a CVE, exposed .env files, missing security headers, login pages with no rate limiting, exposed phpMyAdmin endpoints (1).

The important property of scan-probing is that it is indiscriminate and high-volume. The scanner does not care whether your school has 50 students or 5000; it cares whether you happen to be running a vulnerable version of some software it knows about. So:

• The traffic looks like an attack on the access log — hundreds or thousands of requests from a small set of IP addresses.
• The traffic is not an attack on your data — the bot is not interested in your school, your students, or your staff. It is looking for a generic vulnerability.
• The traffic is still a problem: it consumes bandwidth, fills log files, and burns IT time investigating.

The right response to scan-probing is not "find the attacker and block them"; it is "make scan-probing expensive enough that the bot moves on, log the sources, and rate-limit obviously-abusive clients".

Treating scan-probing as a targeted attack is the most common mistake. You do not need forensics, threat-intel feeds, or incident response. You need rate limits, log review, and a block list that updates itself.

How to recognise scan-probing in your access log

Before tuning any rate limits, look at your actual traffic. On nginx, the access log line for a probe looks like:

185.220.x.x - - [20/Sep/2026:03:14:22 +0000] "GET /wp-login.php HTTP/1.1" 404 162 "-" "Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1)"
185.220.x.x - - [20/Sep/2026:03:14:23 +0000] "POST /xmlrpc.php HTTP/1.1" 403 162 "-" "Mozilla/5.0 (X11; Linux x86_64)"
185.220.x.x - - [20/Sep/2026:03:14:24 +0000] "GET /.env HTTP/1.1" 404 162 "-" "curl/7.88.1"

Three signs you are being scanned, not attacked:

• Many requests, short interval — the same client hits many endpoints within seconds.
• 404 / 403 spikes — the bot is probing for known paths that do not exist on your site.
• Suspicious User-Agents — curl, python-requests, masscan, zgrab, Go-http-client, or an old generic browser string.

On Apache, the same indicators appear in /var/log/apache2/access.log (Debian/Ubuntu) or /var/log/httpd/access_log (RHEL).

A one-liner to count requests per IP in the last hour:

awk '$4 > "[20/Sep/2026:02:00:00"' /var/log/nginx/access.log | awk '{print $1}' | sort | uniq -c | sort -rn | head

Anything over a few hundred requests from a single IP within an hour is almost certainly a scanner, not a real visitor.

The two tools you actually need

For most NZ school and small-business sites, two tools cover 95% of scan-probing defence.

nginx limit_req — request-rate cap

The limit_req module caps the number of requests a single IP can make to a given location in a given time window. Requests over the cap are either delayed (burst) or rejected outright (nodelay). This is the right tool for:

• Login endpoints (preventing brute-force password guessing)
• Search endpoints (preventing expensive query floods)
• Comment forms (preventing spam)
• Any path that has no legitimate reason to be hit thousands of times per minute

fail2ban — log-driven automatic IP blocking

fail2ban watches a log file (typically nginx access log or ssh auth log), counts occurrences of a pattern (a 404 on a sensitive path, multiple login failures), and adds the offending IP to a firewall block list for a configurable period. This is the right tool for:

• SSH brute-force attempts (default behaviour on most Linux distros)
• Repeated scanning for known vulnerable paths
• Repeated probing of administrative endpoints

You do not need both for the same job — limit_req is faster and cheaper; fail2ban is more thorough but adds latency and OS-level side effects.

nginx limit_req — copy-paste configurations

Below are the three most useful configurations for an NZ school or small-business site.

Global rate limit on all requests

Inside your http { … } block, define the rate-limit zone:

limit_req_zone $binary_remote_addr zone=global:10m rate=10r/s;

Then in your server { … } block, apply it:

limit_req zone=global burst=20 nodelay;

This caps every IP at 10 requests per second, with a burst allowance of 20. A typical human visitor makes 1–2 requests per second; a scanner hits dozens per second. The cap applies at the nginx layer, before the request reaches your application (2).

Strict rate limit on the login page

limit_req_zone $binary_remote_addr zone=login:10m rate=1r/s;

server {
    location /login {
        limit_req zone=login burst=5 nodelay;
        proxy_pass http://backend;
    }
}

One request per second per IP, with a 5-request burst. This stops password brute-forcing dead without slowing legitimate logins — a human typing credentials might briefly burst, but cannot sustain 1 r/s for long (2).

Strict rate limit on wp-login and xmlrpc (if you have WordPress)

location ~* ^/(wp-login|xmlrpc)\.php$ {
    limit_req zone=login burst=5 nodelay;
    include fastcgi_params;
    fastcgi_pass unix:/run/php/php-fpm.sock;
}

wp-login.php and xmlrpc.php are the two WordPress endpoints most scanned. If you do not use WordPress, replace this with whatever login / API endpoint your application has.

Reload nginx after any change:

sudo nginx -t && sudo systemctl reload nginx

The -t test catches syntax errors before reload.

fail2ban — copy-paste configuration

fail2ban is included in the package repositories of every common Linux distribution.

Install

# Debian / Ubuntu
sudo apt install fail2ban
# RHEL / Rocky / Alma
sudo dnf install epel-release fail2ban
sudo dnf install fail2ban

The default configuration in /etc/fail2ban/jail.conf enables an SSH jail only. Most NZ sites want at least one additional jail for nginx 404 scanning.

Add an nginx scan-probe jail

Create /etc/fail2ban/jail.d/nginx-scan.conf:

[nginx-scan]
enabled  = true
filter   = nginx-scan
logpath  = /var/log/nginx/access.log
maxretry = 30
findtime = 60
bantime  = 3600

maxretry = 30 failed requests, findtime = 60 seconds, bantime = 1 hour. A scanner hitting your site 30 times in a minute is not a real user; block it for an hour.

Create /etc/fail2ban/filter.d/nginx-scan.conf:

[Definition]
failregex = ^<HOST> .* "(GET|POST|HEAD) [^"]*" (404|403) 
ignoreregex =

This regex matches any GET/POST/HEAD request that returned 404 or 403 — the typical probe signature (3).

Reload fail2ban:

sudo systemctl reload fail2ban

Verify the jail is active

sudo fail2ban-client status nginx-scan
sudo fail2ban-client status sshd

You should see Currently banned: N for each jail. After a day or two of operation, watch the ban counts climb as scan traffic is caught.

Apache equivalents

If you run Apache, the equivalents are mod_ratelimit (built-in, simple) and mod_evasive (third-party, more powerful).

mod_ratelimit (built-in)

<IfModule mod_ratelimit.c>
  <Location />
      SetOutputFilter RATE_LIMIT
      SetEnv rate-limit 400
  </Location>
</IfModule>

This caps each client at 400 KB/s per connection — slows scans without breaking normal browsing (4).

mod_evasive (third-party)

mod_evasive watches for the same pattern as fail2ban (many requests from one IP in a short window) and 403s the parent IP at the Apache layer. Install from your distribution's package manager (libapache2-mod-evasive on Debian/Ubuntu) and configure:

<IfModule mod_evasive20.c>
    DOSHashTableSize 3097
    DOSPageCount 2
    DOSSiteCount 50
    DOSPageInterval 1
    DOSSiteInterval 1
    DOSBlockingPeriod 60
</IfModule>

DOSPageCount 2 = a single IP cannot hit the same page more than twice per second; DOSSiteCount 50 = a single IP cannot hit more than 50 pages per second overall (5).

Cloudflare, Fastly, and other CDNs as rate limiters

If you front your site with Cloudflare, Fastly, or another CDN, you already have a powerful rate-limiting layer in front of your origin. The CDN sees the attack class before it ever reaches your server, blocks it at the edge, and your origin stays clean.

Cloudflare's free tier includes basic rate limiting; the Pro tier adds custom rules (e.g. block any IP that hits more than 50 URLs in 10 seconds). The trade-off is that the rules run at the CDN edge, not on your server, so if the CDN is bypassed (a direct API call to your origin) the rules do not fire. Always set limits at both layers where you can (see the Security Headers Explained guide for the parallel pattern for response headers).

A scan probe that gets rate-limited at the CDN edge never costs your origin bandwidth — that is the whole point of the edge. But a CDN rate-limit that logs the rejection as a 200 OK (because the block is served from a cached "you have been blocked" page) hides the ban from your access log. Check the CDN dashboard, not your origin log, for the actual rejection count.

Logging and reviewing rate-limit decisions

Rate limits and fail2ban bans are silent: a blocked request never reaches your application. To verify the limits are working, log the rejections.

nginx: log rejected requests

limit_req_zone $binary_remote_addr zone=global:10m rate=10r/s;
limit_req_log_level warn;
limit_req_status 429;

limit_req_log_level warn sends rejections to the error log at warning level. limit_req_status 429 (Too Many Requests) is the right HTTP status — clients can read it and back off; 503 is wrong because it implies the server is broken (2).

Then watch the error log:

sudo tail -f /var/log/nginx/error.log | grep 'limiting requests'

fail2ban: ban and unban explicitly

sudo fail2ban-client status nginx-scan
sudo fail2ban-client set nginx-scan banip 185.220.x.x
sudo fail2ban-client set nginx-scan unbanip 185.220.x.x

The set … banip command is useful for manually blocking a known-bad IP without waiting for the log threshold (3).

Common mistakes to avoid

From running the free scanner across New Zealand school and small-business sites, the most frequent missteps are:

• Rate-limiting the entire site at 1 r/s. Most real visitors will hit the limit on the first page load. A sensible global cap is 10–30 r/s per IP; per-endpoint caps are stricter.
• Forgetting to whitelist your monitoring or office IP. If your IT provider monitors the site from a fixed IP, the rate limit may block their health checks. Add the monitoring IP to a limit_req_zone … key= exception or use geo and map.
• fail2ban ban time too short. 60-second bans encourage scanners to retry. A 1-hour minimum is usually the right floor.
• fail2ban ban time too long. Multi-day bans of legitimate users are hard to undo. Stay under 24 hours for soft rules, and reserve multi-day bans for explicit SSH brute-force detections.
• No monitoring on the ban list. A fail2ban configuration that bans correctly but never alerts is useless operationally. Pipe the bans into a Slack or email notification (or at minimum, an hourly cron email of fail2ban-client status output).
• Capping on the CDN but not the origin. CDN bypasses (direct origin requests, API calls) skip your CDN rules. Always set limits at both layers.
• Trusting the X-Forwarded-For header blindly. If your service is behind a reverse proxy, the client IP comes from X-Forwarded-For. Configure your reverse proxy to set this and your rate limiter to read it (nginx: real_ip_header X-Forwarded-For; set_real_ip_from <proxy_cidr>).

Knowledge check

Test your understanding of rate limiting and bot protection with these questions. Click or tap each card to flip it and see the answer.

Sources and references

[1] CERT NZ. (2025). Critical controls for small organisations. https://www.cert.govt.nz/

[2] Nginx. (2025). Module ngx_http_limit_req_module. https://nginx.org/en/docs/http/ngx_http_limit_req_module.html

[3] fail2ban. (2025). fail2ban manual. https://github.com/fail2ban/fail2ban

[4] Apache Software Foundation. (2025). mod_ratelimit. https://httpd.apache.org/docs/2.4/mod/mod_ratelimit.html

[5] Apache Software Foundation. (2025). mod_evasive. https://github.com/jviera/apf-mod_evasive_Apache2

[6] Cloudflare. (2025). Rate limiting rules. https://developers.cloudflare.com/waf/rate-limiting-rules/

[7] Mozilla. (2025). MDN Web Docs — 429 Too Many Requests. https://developer.mozilla.org/en-US/docs/Web/HTTP/Status/429

What to do next

• Run the Free Website Security Check to see what your site looks like to a scanner. The report shows whether common endpoints (login, admin, .env) respond in a way that draws more probing.
• For the response-header side (CSP, HSTS, X-Frame-Options) that prevents many of the attacks scan-probing looks for, see Security Headers Explained for NZ Sites.
• For the strict-Transport-Security side that closes the downgrade attack class scan-probing sometimes tests, see HSTS Strict-Transport-Security Explained for NZ Sites.
• For the Content-Security-Policy walkthrough that stops a probe from succeeding even when it finds an XSS bug, see Content-Security-Policy Deep Dive for NZ Sites.
• For the secure-cookie side that prevents session hijacking when scan-probing succeeds, see Secure Cookie Flags for NZ Sites.
• Rate Limiting and Bot Protection for NZ Sites — why scan-probing differs from real attacks, with copy-paste nginx limit_req and fail2ban-style heuristics for nginx, Apache, mod_evasive and Cloudflare.
• TLS Certificate Renewal Automation for NZ Sites — Certbot, Let's Encrypt, DNS-01 vs HTTP-01 challenges, and how to keep HTTPS certificates renewed automatically on NZ-hosted sites.
• Phishing and Fake Login Forms: NZ School Context — how exposed-secret findings connect to phishing campaigns, with fake-login-form detection patterns for NZ schools.