security research · lab notes
an isolated lab for measuring open-weight models on authorized offensive-security work.
There's a persistent claim that "abliterated" models, open-weight models with their refusal behavior surgically removed, unlock serious offensive-security capability. I wanted to measure it rather than repeat it. So I built a rig to answer one question: can an uncensored local model actually drive a penetration-testing toolkit competently and safely, or is that mostly hype?
Everything here ran against targets I own or am authorized to test, in a sandbox that couldn't see my machine. This was a capability study, not a weapon, and the most useful output was an honest verdict.
A disposable Debian container built to Kali parity, running on OrbStack, driven by a local model served through Ollama on the same Mac. A deliberately vulnerable web app stood in as the practice target, with the correct answers established by hand so I could grade against ground truth. The only thing shared between the sandbox and my machine was a single evidence folder; the host stayed invisible to the model.
The path there was a series of dead ends worth keeping. Running a full coding agent on top of the local model was the wrong shape and got abandoned. Pure chat over Ollama (through Msty) was fine for one-shot questions but weak at actually doing the work. Giving the model a full container plus a workspace is where the recursion problem showed up, and where the project finally became a real graded benchmark instead of vibes.
Bounded, known-technique recon (service discovery, fingerprinting, textbook SQL injection against a target I already knew was vulnerable) worked consistently.
Anything needing novel construction, or simply knowing when to stop, fell apart. Same prompt, same target, back-to-back graded runs swung a single task from 9 out of 10 to 0.
The failure mode I kept hitting was recursion: the model would get the right answer, then re-run the same command and narrate itself in circles. The fix was not sampling tuning, it was a blunt system-prompt rule: you already ran this, do not run it again.
My grader ran an adversarial verifier on every claimed exploit, trying to disprove it. It caught the model reading a local file and passing it off as a live result. Teaching it to say NOT CONFIRMED cost score but bought honesty.
A larger abliterated model was the better hacker (it landed a real injection the smaller one gave up on) but the worse operator: it looped to timeout on most tasks at the vendor-recommended settings.
Peak score across five tuning rounds was about 7.6 out of 10, but which 7.6 you got was a dice roll. Neither local model I tested is a dependable autonomous operator. My considered conclusion: use a frontier hosted model as the primary brain for authorized work, and keep the uncensored local rig only for the narrow cases where it genuinely earns its place, when data can't leave the machine, when the work is air-gapped, or for unattended runs at scale.
There's a quiet irony in the result: authorized security testing is permitted work, so mainstream models mostly don't refuse it anyway. The "uncensored" premise turned out to be the least interesting thing about the experiment. The reusable part is the harness, an adversarial grader that assumes every finding is a lie until proven otherwise. That habit outlasts whichever model is driving.
Ongoing, and model-agnostic by design. Standardized cyber-LLM benchmarks are wired up and waiting for the next round; that's the honest gap in the current results.