Do You Need Cloudflare for a Small Docker Compose App?

My reader portal was using about 1.3 GB of memory on a server with 8 GB. I was paying for the other 6.7 GB to sit there and look reassuring. So I did a little optimisation with the steps performed and shared in this post.

At the same time, everyone I asked about bots said the same thing: put Cloudflare in front of it. So I set myself two questions. How small can this server go? And do I actually need a protective layer, or do I just feel that I should?

I answered the first by moving to a 4 GB server. I answered the second by reading my own logs. This post covers both, and the three bugs that only showed up after the move. If you run a small Docker Compose app and wonder whether you are being naive without a CDN in front, this one is for you.

How Much RAM Does a Small Flask and Celery App Need?

The portal is a Flask app (a Python web framework) with a Celery worker (a separate process that runs background tasks), a scheduler, Postgres and Redis, all in Docker Compose behind Nginx. I described how to set up a stack like this in From Localhost to Live, and how to deploy it from GitHub Actions in Docker Compose CI/CD with GitHub Actions.

I measured before deciding anything, and it is worth saying exactly what I measured. After the move, free -h on the new server showed 1.3 GiB used and 2.0 GiB available out of 3.3 GiB. That is roughly 40 per cent of the host, which leaves room for a busy day, a backup and an unexpected spike.

That is a host-wide figure, not a per-container one. docker stats told a smaller story: the five containers together used about 733 MiB (web 368, Celery worker 220, scheduler 79, Postgres 56, Redis 10). The other 0.5 to 0.6 GiB is the operating system, the Docker daemon, Nginx and similar overhead. free also keeps page cache out of used, because the kernel gives it back when needed. On the old server, free -h showed 1.3 GiB used of 7.7 GiB; I never ran docker stats there.

RAM is not the only sizing question. CPU saturation, disk I/O, Postgres connection limits, Celery concurrency and the memory peak during a backup matter too, which is one reason I chose a 4 GB plan with two dedicated CPUs.

The old server cost about $55 a month. The new one is $40 plus $8 for the provider’s automatic backups, $48 in total, so I save $7 a month. Honestly, halving the RAM saves less than you might expect. The real gain is that the server now matches the app, and I understand why it is the size it is.

What Do Nginx Access Logs Show About Bot Traffic on a Small Site?

Before adding any protection, I wanted to know what I was protecting against. Nginx records every request in its access log, a plain text file with one line per request, so I counted.

# Status codes, most common first, across the active and rotated logs
# (Nginx's predefined `combined` log format: field 9 is the status)
sudo zcat -f /var/log/nginx/access.log* | awk '{print $9}' | sort | uniq -c | sort -rn | head

# Obvious scanner probes
sudo zcat -f /var/log/nginx/access.log* | grep -cE '\.env|wp-login|xmlrpc|\.git/|phpmyadmin'

# Scanner share = matching requests / all requests x 100
sudo zcat -f /var/log/nginx/access.log* \
  | awk '/\.env|wp-login|xmlrpc|\.git\/|phpmyadmin/ {m++} END {printf "%.0f%% of %d requests\n", 100*m/NR, NR}'

zcat -f reads plain and compressed files alike, so one command covers the active log and the rotated ones. It is available on common GNU/Linux systems, though not on every variant. You need sudo because Nginx logs are readable by an admin group only. Check your log retention first, because fifteen days of analysis needs fifteen days of logs, and note that the commands assume the combined format.

A high 4xx percentage on its own says little. Count by URL (awk '{print $7}') and by source IP (awk '{print $1}') as well, to see what is being asked for and by whom.

Here is what my fifteen days looked like:

Read that last row again. Fifteen days, and a few real people. The rest were robots knocking on doors that don’t exist. A scanner looking for wp-login on a Flask app gets a 404 and moves on. That 404 costs me one log line.

Most of these requests looked like routine internet scanning rather than a targeted attack. I cannot know what each scanner intended, and the logs do not prove that nothing hostile happened. A 404 means the resource was not found, and costs me one log line. The same scanners would become much more interesting if they started triggering 5xx errors, reaching sensitive endpoints or eating server resources. The distinction matters, because the cure for noise is patience and the cure for an attack is a different set of tools.

Why Cloudflare Challenge Pages Break Fetch Calls Between a Blog and a Portal

I am not against Cloudflare. It is a good product, and I may well use it one day. I skipped it for a specific technical reason, and I think it’s worth sharing because it is easy to miss.

My blog lives on GitHub Pages at daehnhardt.com. My portal lives at app.daehnhardt.com. Pages on the blog call the portal from the visitor’s browser with fetch, using credentialed CORS requests, to show things like bookmarks and gated content.

The blog and portal addresses are cross-origin (different hostnames) but same-site (the same registrable domain), a distinction explained well on web.dev. The same-site detail does not change the argument, so I will not dwell on it. What matters is what a bot challenge does to a fetch call.

A Cloudflare challenge page is an HTML interstitial that a browser must render and solve. A fetch call from my blog cannot show that page to anyone. If a rule challenges one of those API requests, fetch receives HTML where it expected JSON, the challenge cannot complete, and my bookmarks stop working. Only challenged requests break, but I would rather not find out which ones from a reader. Cloudflare’s own documentation describes a cf-mitigated: challenge header you can use to detect this in fetch and XHR code. Cloudflare documents it because it expects the situation to happen.

Bot Fight Mode, the simplest switch, makes it harder still. Cloudflare states that it may challenge API or mobile app traffic, and that you cannot bypass it with WAF custom rules or Page Rules. Super Bot Fight Mode is a different product: it runs on Cloudflare’s ruleset engine and supports Skip rules, so the limitation is not universal across Cloudflare’s bot protection. Both can still challenge API traffic. You could configure around all of this, but I would be configuring around a problem my logs say I don’t have.

The three Cloudflare options compare like this for a blog that calls a portal through fetch:

There is another route I considered. Cloudflare recommends Turnstile Pre-clearance for API calls: the widget issues a cf_clearance cookie after a visitor passes on an HTML page, and later API requests can then pass WAF rules without a full-page challenge. It is a real alternative. Whether it suits a blog on GitHub Pages calling a portal on another host is something I would have to test, and it adds moving parts my logs do not justify yet.

There is a lighter option I may use later: Turnstile on the sign-up and contact forms. Turnstile puts a widget in the form, which may or may not ask the visitor to interact, and your server then validates the token with Cloudflare. The token lasts five minutes and works once. This protects the forms without touching the API calls.

For now I rely on rate limits inside the app, a firewall that exposes only the ports the app needs, and brute-force protection on SSH. I considered extra log-based banning for Nginx, but it would mostly tidy noise that is already harmless.

When I would change my mind: if the logs turn hostile. Here is how I would read them:

How to Move a Docker Compose App to a New VPS With Little Downtime

I wanted the move to be dull, and it was, thanks to a written runbook and a rehearsal. The order was:

1. Build the new server and lock it down first. SSH hardened, firewall on, before any app was installed.
2. Rehearse the data move. A script dumped the old database, copied it, restored it on the new server and compared the row count of every table. I ran it in rehearsal mode first and only went live once the counts matched exactly.
3. Get the HTTPS certificate before switching DNS. I used Let’s Encrypt’s DNS challenge, which proves you control the domain with a temporary DNS record, so the new server had a valid certificate before any visitor reached it.
4. Lower the risk of stale DNS. My DNS records had a 600-second lifetime. Most DNS caches should pick up the new address within roughly ten minutes, although some clients may take longer.
5. Cut over in a short window. Stop the old app, do a final sync, start the new one, switch DNS, then test from a private browser window.
6. Keep the old server for a week. The old server stays as a fallback until I am sure nothing was missed.

Matching row counts is a good first check, but it doesn’t prove a successful restore. Also confirm the restore command exited cleanly, that the old and new PostgreSQL versions suit your dump method, and that schema objects, indexes, extensions and sequences came across. Finish with a real application smoke test.

One lesson from step 5 deserves its own box.

Retire the old host with docker compose down, not docker compose stop.

A restart policy is a Docker setting that decides whether a container returns after it exits or the server reboots. My containers use restart: always. Docker’s documentation says a manually stopped container is restarted when the Docker daemon restarts, so a stopped container comes back after a server reboot. If my old scheduler had come back, I would have had two schedulers on two servers sending every scheduled email twice. down removes the containers, so nothing comes back.

Three Bugs After Moving a Docker Compose App to a New Server

The move went well. Things that had been quietly wrong for months did not.

1. Celery Worker Missing a Volume Mount: “master file missing”

My web container and my Celery worker share one image but not one set of mounts. I had mounted the backup folder only where I remembered it.

The nightly database dump runs in the worker, so for a while it wrote dumps inside the container, where a redeploy erased them. Later I added a feature where a background task attaches a protected PDF to an email. The web container could see the PDFs. The worker could not, so the task failed with “master file missing” after the page had already told the reader the email was on its way.

The fix was one extra line in the Compose file (the second volume below), plus a test so it cannot quietly regress:

celery:
  volumes:
    - ./backups:/backups
    - private_files:/app/private_files:ro

The lesson: a task runs in the worker’s container, so the worker needs every file the task touches. A missing mount shows up in the worker log, not as a web error, and nobody sees the log until they go looking. Now I check it after each deploy, using a harmless test PDF:

# Verify the worker can see the protected files
docker compose exec celery test -r /app/private_files/example.pdf \
  && echo "Worker can read the file"

A deployment is not finished when the containers are green. A readable file is also not the same as a working task, so I send one real task to a non-production address as well.

2. Alert Emails Bounced Because the Recipient Was the Default Sender

I had built a small resource monitor that emails me when memory, disk or backups look unhealthy. The first alerts arrived as bounce messages. I had not set an admin recipient, so the app fell back to its default sender address, a no-reply address on my own domain, and mailed that. My mail provider rejects mail to that address with a 550 error.

I set the recipients explicitly and sent a test. Fixing the recipients took five minutes. The lesson is that the first test of any alert should be a real one, delivered to a real inbox, before you trust it.

3. Rate-Limit IP Blocks That Never Expire

My app blocks an IP address after repeated rate-limit breaches. The blocks were permanent. Quietly, one of them locked me out of my own site. When I dug in, I also found that signed-out visitors’ bookmark checks, which the blog fires on every page view, were counting towards the limit. Those bookmark checks are ordinary page-view traffic, not abuse.

Now blocks expire after a short, fixed time; a block I place by hand is never overwritten; the same expiry applies to the account flag that comes with the block; and that polling no longer counts. Automatic defences should expire unless there is a good reason for a permanent block.

Cloudflare as a Doorman: a Mental Model

Think of the server as a flat and of Cloudflare as a doorman. He is excellent at keeping strangers out, but he also stops the postman and your cousin with a key, and you have to teach him their faces. Before hiring one, look at who actually rings. Mine mostly got people ringing doorbells that did not exist.

Checklist: Right-Sizing a Small Server and Skipping a CDN

1. Measure real memory use before choosing a server size.
2. Read your access logs for a couple of weeks before buying protection.
3. List every cross-origin fetch your front end makes, and ask what a challenge page would do to each.
4. Rehearse the data move and compare row counts per table.
5. Get the certificate ready before switching DNS.
6. Retire the old host with docker compose down, not stop, when it must never come back after a reboot.
7. Give workers the same mounts as the web container, and check them after each deploy.
8. Send a real test alert to a real inbox.
9. Make automatic blocks expire and exempt yourself.
10. Write down when you would change your mind about the protection you skipped.

Want these commands on paper? I put the ones I used into a two-page cheat sheet: reading Nginx and container logs, routine server maintenance, and the database copy, certificate and DNS steps of a move. I ran the log and database commands before they went on the page. It is free with a free membership, along with the other cheat sheets. Print it and keep it next to the keyboard.

Final Thoughts: Cloudflare Not Yet, With a Trigger to Revisit

I did not decide against Cloudflare. I decided not yet, with reasons I can write down and a trigger for reconsidering. That feels more honest than either installing it because everyone does, or ignoring it because I was in a hurry.

The server move was the easy part. The surprises came from small gaps that a bigger, emptier server had been hiding. If you are right-sizing your own setup, expect the same, and expect to learn something useful.

The most useful part was not the decision but the method: measure what the server uses, read the real requests, understand how the pieces talk to each other, rehearse the move, and write down what would change your mind. Swap in your own server and your own security product; the method still works.

Have you looked at your own access logs recently? What did they say?

References

• Cloudflare: Bot Fight Mode, including the note that it may challenge API traffic and cannot be skipped with WAF custom rules.
• Cloudflare: Detect a challenge page response, the cf-mitigated header for fetch and XHR.
• Cloudflare: Turnstile server-side validation.
• web.dev: “Same-site” and “same-origin”.
• Let’s Encrypt: Challenge types, the DNS-01 challenge.
• Docker: Start containers automatically, restart policies.
• My earlier posts: From Localhost to Live and Docker Compose CI/CD with GitHub Actions.