How to Audit a Flask App With Reusable AI Security Skills

I do not enjoy security audits. I suspect most developers do not put them at the top of their Friday-afternoon wish list either. They are the kind of task everyone agrees is important and nobody wants to do this week, which is exactly why “we will get to it eventually” projects stay unaudited until something goes wrong. I have a small Flask app I built to track my own AI subscriptions — which tools I actually use, what they cost, whether they are earning their place — and I wanted it checked properly before I trusted it with my own data. Not a five-minute glance. A real review.

So instead of asking an AI agent to vaguely “check the security” of my app, I built something I can reuse. This post is about that process: how a ChatGPT brainstorm turned into four reusable Claude skills, what tools they run under the hood, and — this is the part I actually want you to remember — how testing them against a real app immediately caught mistakes in their own instructions, before those mistakes could waste anyone’s time later.

If you build small tools with AI assistance the way I do (I wrote about my general workflow for turning prompts into deployed apps a while back), this is the missing piece: a repeatable way to check what you shipped actually holds up.

Why a Reusable AI Security Skill Beats a One-Off “Check My Security” Prompt

An AI skill is a versioned instruction file — scope, permitted tools, required output format — that an AI agent loads and follows the same way on every run, rather than a prompt improvised per conversation. That difference is the whole point of this post.

“Please check this app for security issues” is not an instruction. It is a wish. An AI agent given that prompt alone will either invent plausible-sounding vulnerabilities that are not real, or miss the boring, well-documented ones that actual scanners catch in seconds. Neither outcome is useful, and both waste your time verifying claims that should have been solid in the first place.

I asked ChatGPT to sketch out what a proper version of this would look like, treating it the way a security consultancy would: narrow, repeatable checks, each with a defined scope, a list of permitted tools, and an expected output format. ChatGPT’s answer was a ten-skill breakdown — a code reviewer, an authorisation tester, a test generator, a dependency auditor, a headers auditor, a dynamic scanner, an authenticated-journey scanner, an abuse tester, a secrets reviewer, and a findings verifier — plus its own advice not to start with all ten at once. Fewer, more reusable skills beat ten independent agents nobody keeps maintained.

I fed that whole brainstorm to Claude, which had just finished building the subscription-tracking app, and asked it to turn the recommendation into something real: a skill set generic enough to run against this app today and any other Flask app I build later, saved in my own reusable skills repository rather than living only in one conversation.

The Four Flask Security Skills: Code Reviewer, Authorisation Tester, Runtime Scanner, Findings Verifier

Claude and I settled on the lean version of ChatGPT’s list — four skills, not ten. Three of them look at the app from different angles; the fourth sits downstream and refuses to pass anything through without evidence:

Flask repository
      │
      ├── Security code reviewer ──── Bandit + Semgrep + manual checklist
      │                                         │
      ├── Authorisation tester ────── Route matrix + generated pytest
      │                                         │
      ├── Runtime scanner ─────────── pip-audit + Trivy + headers/cookies
      │                                         │
      │                                         ▼
      └────────────────────────────► Findings verifier
                                     evidence · severity · fix · re-check
                                                │
                                                ▼
                                          Report I act on

And in more detail:

Security code reviewer. Reads the Flask app’s routes, config, Jinja2 templates, and database queries against a fixed checklist — DEBUG mode left on in production, a weak or default SECRET_KEY, missing CSRF protection on state-changing routes, unsafe render_template_string (server-side template injection), raw SQL string interpolation (SQL injection), unrestricted file uploads, and so on. It also runs Bandit — a static analyser that builds an AST of each Python file and runs security plugins against it — and Semgrep, a pattern-matching static analysis engine whose rules are written in the syntax of the language being scanned. The skill interprets both tools’ output rather than dumping it raw.

Authorisation tester. The skill I care most about for a multi-user app. It maps every route to who should be allowed to reach it — anonymous visitor, logged-in stranger, the actual owner of that data, an admin — and generates real pytest tests proving a stranger cannot read or edit someone else’s records. That class of bug has a name: broken object-level authorisation, and it is the one generic scanners structurally cannot find, because they do not know your app’s ownership rules.

Runtime scanner. Runs PyPA’s pip-audit, a tool that checks Python environments and dependency files against known-vulnerability databases, and Trivy, Aqua Security’s open-source scanner for filesystems, secrets, and container configuration — plus a small check of the app’s actual HTTP response headers (Content-Security-Policy, X-Content-Type-Options, Strict-Transport-Security) and session cookie flags (Secure, HttpOnly, SameSite).

Findings verifier. The sceptic. Every finding from the other three skills has to arrive with an exact file and line, the attacker-controlled input, where that input lands unsafely, a reproduction, and a suggested fix. Anything vaguer — “this may be vulnerable to SQL injection,” with no evidence — gets bounced back rather than reported. The findings verifier also assigns each surviving finding a priority: P0 for a demonstrated compromise, down to P3 for low-impact hardening, with an “Informational” tier for accepted risks and false positives that are worth recording, not silently dropping.

“Skill” can sound like a fancy word for a well-phrased prompt. It is not, and the difference shows on the page. Here is the section of the findings verifier that does most of the work:

## Required evidence

Do not report a vulnerability unless you can provide all six:

1. The affected file and line number
2. The attacker-controlled input, and how it reaches the code
3. The unsafe operation, or the security control that is missing
4. Reproduction steps, or a failing test that demonstrates it
5. A proposed remediation
6. Verification that the remediation works, after it is applied

If any item is missing, downgrade the finding to "Needs investigation"
and state which item you could not supply. Do not guess.

That is a specification, not a request. It is the difference between hoping an agent is rigorous and giving it a definition of rigour it can be held to — and it is the same file every time, rather than however I happened to phrase things on a Thursday.

We deliberately left two of ChatGPT’s ten skills for later — a dynamic scanner (OWASP ZAP) and an authenticated-journey tester. Both need a real deployed target worth spidering, and my subscription tracker only runs on my own machine so far. Building them now would have been effort spent on a problem I do not have yet.

Behind all four sits OWASP’s Application Security Verification Standard (ASVS), a catalogue of application security requirements written as testable statements rather than advice. Its 5.0.0 release went live on 30 May 2025 at Global AppSec EU Barcelona — the first major version in five years, with roughly 350 requirements across 17 chapters. ASVS gives the four skills one shared definition of what “secure” means in testable terms, instead of each skill reinventing its own idea of what to check.

Two Scanner Bugs Found by Testing the Skills: Bandit’s -x Exclude Flag and Trivy’s Docker Compose Gap

Here is the part I actually wanted to write this whole post about. Claude did not just author four markdown files and call it done. Claude installed all four scanners and ran a real pass against my Flask app the same day — and that single test caught two mistakes in its own freshly written instructions before those mistakes could quietly mislead the next person who ran them.

The first mistake was in the Bandit command:

bandit -r . -x .venv,tests,migrations -f json -o reports/bandit.json

That -x flag is supposed to exclude the virtual environment and test suite from the scan. It looked correct. It did nothing, and the reason is worth understanding rather than memorising. Bandit’s manual page describes -x as a “comma-separated list of paths (glob patterns supported)”, and its own default value is a list of bare names — .svn,CVS,.bzr,.hg,.git,__pycache__,.tox,.eggs,*.egg — which reads as though bare directory names are the normal case and globs are the optional extra. In practice it is the other way round: Bandit matches each file path it is about to scan against your exclude patterns using fnmatch. During a recursive run those paths look like ./.venv/lib/python3.12/site-packages/foo.py. The pattern .venv matches none of them. It would only match a file literally named .venv. I am not the first to walk into this — it is the same behaviour reported against Bandit 1.6.0 when excluded directories stopped being respected, and it still behaves this way on Bandit 1.9.4, which I checked against a throwaway tree with one file inside .venv and one outside: -x .venv excluded nothing, -x "./.venv/*" excluded exactly the file it should.

So the first run scanned the entire .venv folder along with the app itself and came back with 4,011 results, almost all of them noise from third-party library code that was never mine to fix. The corrected version:

bandit -r . -x "./.venv/*,./tests/*,./migrations/*,./__pycache__/*" -f json -o reports/bandit.json

brought that down to 2 results — the actual number of things in my own code worth looking at.

A four-digit result count from a small app is not evidence of a disaster. A four-digit result count is evidence the exclude flag did not work, and the fix went straight back into the skill’s own instructions so nobody hits that wall again.

Worth saying plainly: an exclusion list is the awkward way round. If your layout allows it, naming what you do want scanned is easier to reason about than maintaining an ever-growing list of what you do not:

bandit -r app tests -f json -o reports/bandit.json

My repository has application code scattered outside a single package directory, which is why the skill still defaults to -r . with exclusions. If yours does not, take the narrower route.

The second mistake was more interesting. My app has no Dockerfile — only a docker-compose.yml running Postgres, with the Flask app itself running on my machine directly. The skill’s instructions assumed Trivy’s built-in misconfiguration checks would inspect that Compose file for the usual sins — containers running as root, exposed ports, hardcoded passwords. They do not. Trivy’s own list of covered infrastructure-as-code formats is Azure ARM templates, CloudFormation, Docker, Helm, Kubernetes and Terraform — and the “Docker” entry there means Dockerfiles. Docker Compose is not on that list. I confirmed it by pointing Trivy straight at the file and watching it report zero config files detected. Trivy can be pointed at arbitrary YAML with custom Rego checks, so this is a gap in the built-in rules rather than a hard limit of the tool — but “write your own policy” is a different job from “run the scanner”. The fix was not a command-line flag. It was rewriting that section of the skill to say plainly: read the Compose file yourself, because the tool will not do it for you.

Neither the Bandit exclude bug nor the Trivy Compose gap would have mattered if the skills had just been written and filed away. Both mattered because Claude ran the skills for real against something with actual code in it, the same day, before handing me a report I might have trusted at face value.

Flask Vulnerabilities Found: Open Redirect and Host-Header Password-Reset Poisoning

Once the Bandit exclusions were corrected, the same scanning pass surfaced two genuine issues in my subscription tracker, plus a handful of false positives worth mentioning because they show why a findings verifier matters at all.

An open redirect. An open redirect is a vulnerability where an application sends a user to a destination taken from untrusted input without checking it belongs to the application’s own site. My login page accepted a next parameter — the page to return you to after logging in — and passed it straight to a redirect with no check that it actually pointed somewhere on my own site. Someone could send a link like mysite.com/login?next=https://looks-like-a-scam.example, and after you logged in — trusting my domain, because you were really on it — you would land somewhere I never sent you. The fix rejects any next value that carries a web address of its own, and only allows a plain path on my own site.

A password-reset link that trusted the wrong thing. The formal name for this one is host header injection, and the attack it enables is password-reset poisoning. When my app emails you a password-reset link, it builds that link’s web address from the incoming request. Normally that is your browser, asking politely. But nothing stopped someone else from asking on your behalf with a forged host value in the request — in other words, the request could lie about the domain the app was running on. That would land your genuine reset email in your inbox — sent by my real server, about your real account — with a link pointing at their address instead of mine. If you clicked it, your reset token would go to them, not to me. The fix stopped building that link from the request at all, and uses one address I set on purpose instead.

The open redirect and the password-reset poisoning are both the kind of bug a routine glance at the code would not catch — they only show up once you ask “what happens if the exact thing the code assumes is well-behaved… is not?” Both were fixed and verified the same afternoon, but only after Claude asked me first: “Want me to fix those two now?” I approved the changes. Claude patched the code and added regression tests for both vulnerabilities; I then checked the behaviour myself against the running app before calling it done.

The false positives were just as instructive. Bandit flagged a fixed string in my token-signing code as a “possible hardcoded password” — it was not a password at all, just a label that keeps a password-reset token from being replayed as an email-verification token. That check is B105, hardcoded_password_string, and all it does is spot a string literal assigned to a name matching one of seven words: password, pass, passwd, pwd, secret, token, secrete. Bandit’s own documentation says the plain part out loud — “this can be noisy and may generate false positives.” Semgrep, meanwhile, flagged twelve of my templates for “missing CSRF protection” using a rule written for Django, a different framework, that does not recognise how Flask actually implements the same protection. Both got marked and explained, not silently dropped — a finding that turns out to be nothing is still worth a sentence, so the next person does not wonder whether it was ever checked at all.

Checklist for Building and Testing an AI Security-Audit Skill

If you take nothing else from this post, take these six:

1. Write the check down as a file, not a prompt. A skill you can point at any repository beats a conversation you have to reconstruct.
2. Run it against real code the same day you write it. Instructions that have never been executed are guesses with good formatting.
3. Sanity-check the result count. Four thousand findings from a small app means your exclusions failed, not that your app is on fire.
4. Prefer naming what to scan over listing what to skip. bandit -r app tests is easier to keep correct than an exclusion list that grows every time you add a folder.
5. Check what your scanner does not cover. Mine skipped docker-compose.yml entirely, and said nothing about it.
6. Demand evidence per finding — file, line, input, reproduction, fix. Anything vaguer goes back for investigation rather than into the report.

What an AI Security Audit Does and Does Not Prove

Building the skills, testing them, fixing what they found, and writing this post took a real afternoon of work. None of it was instant. But the four skills now sit in my reusable skills repository, ready to run against whatever Flask app I build next, with no re-explaining what “check the security” is supposed to mean. That is the actual return on the time: not this one audit, but every audit after it.

I also like that the skill set stayed honest about its own limits. Nothing here fixes authentication code without asking me first, and every finding had to survive a sceptical second pass before I ever saw it as something to act on, rather than a list of scary-sounding maybes. But it is equally important to be clear about what a clean run does not buy you:

What this audit does not prove

Passing these checks does not make an application “secure.” The skills review source code, dependencies, authorisation rules, configuration, response headers and cookies. They do not perform authenticated dynamic testing, penetration testing, infrastructure review, or production monitoring. A green report means the things I chose to check came back clean — nothing more.

Nothing here scans a live production site, either. I do not have one yet for this app, so building that skill now would be solving a problem I do not have.

Should You Build Reusable AI Security Skills for Your Own Apps?

If you build small tools the way I do — quickly, with AI help, and more of them than you can manually audit one by one — a rough brainstorm plus a few hours turning it into something reusable is a much better trade than either skipping the audit or redoing the same conversation from scratch every time. The audit you actually run beats the thorough one you keep postponing. Mine now takes an afternoon once, and minutes thereafter. If you want the other half of this — keeping yourself safe while the assistant writes the code in the first place — I covered that in Using AI Code Assistants Safely.

References

1. OWASP Application Security Verification Standard (ASVS) — and the v5.0.0 release of 30 May 2025
2. Bandit — PyCQA, its manual page documenting the -x exclude flag, issue #488 on excluded directories, and check B105: hardcoded_password_string
3. Semgrep
4. pip-audit — PyPA
5. Trivy — Aqua Security, its infrastructure-as-code coverage list, and custom Rego checks
6. OWASP ZAP