Abstract
Large language models are increasingly used for software development tasks, includingbug localization and repair. Their reported performance, however, can be difficult to
interpret when benchmark programs are familiar, clearly written, or rich in surface
cues such as meaningful identifiers and conventional structure. This thesis investigates
whether large language models can still identify and repair injected bugs when the
same programs are transformed through source-code obfuscation, and whether access
to dynamic debugging tools can compensate for the resulting loss of readability.
The thesis therefore introduces Grenzer, an experimental framework for constructing
controlled Python debugging tasks. The thesis uses the framework with a selection of
executable LeetCode-style programs to inject one observable defect per sample, validate
the defect with tests, filter for samples that are solvable before obfuscation, applying
source-level obfuscations while tracking bug locations, and evaluating model diagnoses
using a panel of judge models. The evaluated bug families are off-by-one errors, wrong
variables in expressions, wrong logic operators, and wrong arithmetic operators.
The results show that the evaluated models are robust to mild obfuscation, but not
invariant to it. Individual transformations cause moderate degradation, with misleading
comments having the smallest effect and identifier renaming producing clearer drops.
The strongest static effect appears under combined obfuscation: gpt-5.4 remains at
91.3% approved repair attempts, while Qwen 3.6 falls to 78.0%. Dynamic debugging
changes this picture. In the agentic multistep condition, Qwen 3.6 improves to 94.8%
approval by using probes and debugger actions extensively, whereas gpt-5.4 changes
only slightly, from 91.3% to 91.7%, rarely using debugger-specific tools. This improve
ment comes with a substantial token-cost trade-off.
Overall, the thesis shows that obfuscation is a useful stress test for code-debugging
ability. Agentic debugging demonstrates that performance can be recovered when models
are allowed to gather runtime evidence.
| Date of Award | 2026 |
|---|---|
| Original language | English (American) |
| Awarding Institution |
|
| Supervisor | Josef Pichler (Supervisor) & Mario Winterer (Supervisor) |
Studyprogram
- Software Engineering
Cite this
- Standard