Ghost Nil

01Why this exists

DeFi protocols lock real money behind Solidity. Traditional audits are episodic. Immunefi bounties are always open, first reporter wins, and they pay for impact rather than for a PDF. That combination is a reasonable place for an automated hunter to sit — if the hunter is conservative about what it claims.

The working hypothesis was simple. Pattern-matching tools already catch a lot of known vulnerability classes: reentrancy, unprotected selfdestruct, spot-price oracles. They miss most business-logic and economic bugs. Language models are the opposite: they can reason about “what happens if an attacker flash-loans this token and donates it before a snapshot,” but they get overwhelmed when you dump an entire protocol into one prompt. A useful system would run the cheap, deterministic tools first, then force a model to look at one file at a time with the right historical examples in context.

This was focused on a local model running on hardware that can go for hours. That was the point. If compute is free, and security is a function of compute, then you run it long enough and you eventually find something — even a needle in a haystack.

There was a second, non-technical reason. The person building this is security-conscious about supply-chain risk. The scanner had to make every outbound network call explicit, prefer local tools, and never treat model output as a finding until a human had read it. AI submissions without review are how you spam bounty programs and burn credibility.

02Design constraints

Several decisions were made early and kept, because they keep the system small enough to understand.

• EVM only. No Cairo, Move, or Solana. One compiler family, one bytecode model, one set of tools.
• Two tiers, no middle ground. Either a tool can run on every contract with zero per-protocol setup, or it belongs in the LLM loop. Fuzzers such as Echidna and Medusa were deferred because they need hand-written invariants.
• Benchmark before hunting. If the pipeline cannot find known bugs in Damn Vulnerable DeFi, it has no business claiming it will find unknown ones in production.
• Immunefi first. Always-open programs, ranked by bounty size, language, and whether a GitHub scope exists. Competitive audit contests were deferred.
• Human last mile. The database stores candidates. A person decides whether anything is real, in scope, and worth writing up.
• Long-running is acceptable. A deep-scan of an 80-file protocol can take on the order of an hour. That is a feature if the alternative is a shallow one-shot pass.

03Two-tier architecture

The limitation of the one-shot analyzer is the reason Tier 2 exists. When the model sees an entire contract plus every scanner finding at once, it tends to name one or two obvious issues and stop. File-by-file prompting removes that collapse: twenty files become twenty focused calls, each with its own few-shot examples.

An aggregator then deduplicates across tools by normalized detector category, contract, and severity. If Slither and Mythril both report reentrancy on the same address, the more detailed finding is kept. Deep-scan results are tagged separately so they can be triaged as higher-volume, lower-precision candidates.

The model is instructed to return JSON only, and to return an empty list if the file is clean. In practice, reasoning models still narrate. A chain-of-thought extractor therefore has to distinguish “this is a reentrancy” from “there is no reentrancy here.” Strict mode requires affirmative attack language, uses an expanded dismissal list, and defaults severity to Medium. That change cut extractor noise by about 71% in the post-Mt-Pelerin revision.

04How a protocol moves through the system

Progress is a SQLite state machine. The queue is resumable: stop at any time, run again, continue. That matters more than it sounds. Mythril can take minutes per contract; a deep-scan can take most of an hour. A pipeline that cannot survive Ctrl+C is not a pipeline you will actually use.

Solidity versions in the wild span 0.4 through 0.8. Before Slither or Mythril run, the scanner reads the pragma and uses solc-select to activate a matching compiler, falling back to 0.8.28. Semgrep does not need that; it matches source text. SMTChecker is a compiler flag on the same solc binary.

A Streamlit UI sits on top of the same database — dashboard, queue, scan, reports, settings — but the CLI is the real interface: fetch, run, deep-scan, benchmark, bounties, status, report, retry.

05The knowledge layer

A generic “find a vulnerability” prompt is the caveman version of this idea. The useful version tells the model what usually goes wrong in this kind of protocol, then shows it two real exploits that look like the file under review.

Three local corpora feed that context. None of them are queried over the network at scan time.

• DeFiHackLabs — 682 reproduced Foundry PoCs, indexed by attack category (reentrancy, oracle manipulation, flash loan, donation, proxy initialization, and others). The deep-scan guesses categories from the source text and retrieves two nearby examples, including a short code snippet.
• Incident index — a combined store that can include a large REKT-style incident list and a structured reentrancy-attack collection. When a scan is labeled lending, dex, bridge, vault, staking, token, or governance, the prompt receives a short historical summary of how those archetypes actually fail.
• Custom Semgrep rules — twenty patterns written from real incidents rather than generic CWE lists: state-after-.call(), transfer-before-state (including safeTransfer), unprotected selfdestruct / delegatecall, tx.origin auth, spot-price and single-block TWAP oracles, balance / balanceOf(this) used as accounting, ecrecover without nonce or zero-address check, unsafe downcasts, division before multiplication, uninitialized implementations, missing slippage or deadline, unchecked low-level calls, and arbitrary storage writes.

A local Anvil harness can fork a live chain (GetBlock or a public RPC) so a suspected issue can be poked without sending a mainnet transaction. Full automatic PoC generation was started and not finished. In the field experiments below, the verified High findings were reproduced with hand-written Foundry tests against that local setup.

06Evaluation: Damn Vulnerable DeFi

Damn Vulnerable DeFi is a sequence of 18 challenges, each a small but realistic protocol with a known vulnerability, ranked from Level 1 to Level 5. Ghost Nil treats it as a regression suite. Ground truth is a set of attack categories per challenge — for example, puppet-v2 is tagged oracle-manipulation, spot-price-oracle, and flash-loan-attack.

Scoring is intentionally loose on detector names and strict on coverage:

• Pass — every expected category was matched by at least one finding.
• Partial — at least one expected category matched.
• Miss — nothing in the right ballpark.

That means a Partial is not a win. It means the pipeline noticed a related smell and still failed to name the full bug. The table is a progress meter, not a marketing score.

PassPartialMiss

Read the delta the unglamorous way. The system got better at oracle and flash-loan families — the classes it was explicitly primed for — and stayed weak on pure logic bugs. That is consistent with the design, not a surprise.

07Field experiments

After the v2 benchmark, the same pipeline was pointed at live Immunefi programs. Targets were selected from the public bounty API, ranked by reward, Solidity scope, and whether a GitHub repository existed. What follows is the triage record, not a claim of paid bounties. Operational detail that would help an attacker is omitted.

08What the tools taught us

The first live scans changed the pipeline more than the benchmark did. Six problems showed up immediately on Mt Pelerin; most of them were then implemented before TermMax.

1. Chain-of-thought is not a finding list. Qwen3 reasons out loud. A keyword extractor that sees the word “reentrancy” in “so no reentrancy risk here” will invent a High. Strict extraction — dismissal phrases, required affirmative language, Medium by default — was the single largest noise cut (about 71%).
2. Do not scan interfaces. An interface file has no executable body. Every finding on one is a false positive.
3. Do not scan mocks or tests. Same reason. Combined with interface skipping, file filtering removed about 42% of deep-scan work.
4. Scope must be applied before the model runs. Immunefi publishes an in-scope asset list. Scanning 71 files when 29 are in scope wastes the expensive calls and pollutes triage. get_scope_files(slug) now feeds the loop.
5. Severity has to be earned. A keyword in a paragraph is not High. High or Critical requires an attack scenario, a fund-flow story, or a reproduction step in the model’s own words.
6. Standard libraries are a targeting signal. Folks Finance was clean because it was conventional. Custom math, custom oracles, and custom access control are where this stack is more likely to earn its keep.

Semgrep also needs its own humility. Twenty rules written from real hacks will fire on SafeCast downcasts and on ecrecover that already checks address(0). A rule that cannot see a nearby guard is still useful as a hint to the model, and still noisy as a report line.

09Limits, non-goals, and honest reading

Ghost Nil does not replace an audit. A 2/18 pass rate on a teaching benchmark is a system that sometimes names the right bug class, occasionally names the whole bug, and usually needs a person. The field scans confirm the same ratio: hundreds of candidates, a handful of real issues, and scope rules that can make the best issue unsellable.

Things that are explicitly out of scope, for now:

• Fuzzing and invariant testing (per-contract setup).
• Bytecode-only decompilation as a primary path. Verified source is the target; unverified contracts were a documented idea and not the built pipeline.
• Automatic bounty submission, or any model output sent to a vendor without review.
• Code4rena contest integration — planned on paper, waiting until the scanner is better than “partial on DVD.”
• Non-EVM ecosystems.

Things that were started and are not finished: richer REKT-database prompting as a default, a dedicated Immunefi submission formatter, and fully automatic exploit synthesis on the Anvil fork. The TermMax reproductions used the harness and human tests, not a generated attacker contract.

If you read only the headline numbers, you will overfit. The interesting result is the shape of the work: deterministic tools for known patterns, a constrained local model for file-local reasoning, historical exploits as few-shot memory, and a human who is willing to throw 93% of the output away.

10What leaves the machine

This section exists because the project was built under a supply-chain constraint: if a call is not listed here, the scanner is not supposed to make it. Static tools run as local subprocesses. Semgrep is invoked with metrics off and a local rule file, not the Semgrep registry.

Etherscan is rate-limited with a token bucket at five requests per second and three retries. Immunefi responses are cached for six hours. Fork testing, when it happens, executes on the local Anvil process. No exploit transaction is sent to a public mempool by this design.

11Status

Ghost Nil is a finished-enough experiment to talk about and an unfinished hunter. The pipeline works end to end: queue, discover, acquire, scan, report. The deep-scan loop is real. The DVD numbers above are from April 2026. The three Immunefi scans are from the same week. The source is not being released. A public repo would invite people to run an unattended model against production protocols and paste the output into bounty forms, which is the failure mode this project was designed to avoid.

What I would still want, if this were continued: less Semgrep noise around SafeCast, default protocol-type prompts instead of an optional flag, and a tighter loop from “the model described an attack” to “a Foundry test failed on a fork.” Discovery remains the hard part. Repair and reproduction are easier once you know what you are looking at — a point also made by EVMbench, which this project read as a design note rather than as a leaderboard to chase.

If you work on a protocol that was mentioned here and want the private write-up, that is what responsible disclosure is for. This page is the method, the scores, and the limits — not a vulnerability feed.