Von flachen Daten zu tiefem Verständnis: Wissensgraphen zur Optimierung der LLM-basierten Bedrohungsanalyse

Abstract

Modern companies increasingly rely on complex digital infrastructures that are essential for their operations and value creation. Vulnerabilities in these systems can cause
significant damage. At the same time, the constantly growing number of IT securityrelated news articles makes it difficult for organizations to maintain a clear overview of
their own threat landscape. To address this issue, the preliminary study RiskAlert was
conducted as part of the SIM master’s program at FH Hagenberg.
RiskAlert continuously monitors IT security-related news sources and uses a Large
Language Model (LLM) to analyze news articles by taking into account internal IT assets
(including software) to identify potential threats. In each analysis, all company data is
loaded entirely into a prompt provided to the LLM. However, due to the limited context
size of LLMs, scalability is restricted. Additionally, it was observed that even when
using a relatively small sample company, RiskAlert struggled to establish relationships
between company assets and technologies described in news articles.
The aim of this thesis is to investigate how corporate data can be transformed from
a flat structure into a knowledge graph (KG) and integrated into RiskAlert, as well
as to evaluate system performance before and after integration. The thesis follows the
design science approach and verifies several hypotheses in the form of research questions
through the development of multiple prototypes.
Initially, an ontology is modeled, defining the data model of the KG and incorporating additional contextual information. Based on this, an LLM-based assistant is
developed which identifies missing information (including additional contextual information) between the ontology and an initial corporate KG, automatically supplementing
it through expert interviews. Subsequently, RiskAlert is enhanced to utilize the KG for
threat analysis. Various techniques are employed to selectively reduce and extract relevant corporate data for the LLM, leveraging the advantages provided by using the KG
within a Neo4j database. The system is then evaluated, using a specially created ground
truth, and compared to the original version of RiskAlert.
All identified information gaps are successfully closed by the LLM-based assistant
based on the developed ontology and the initial KG. The evaluation demonstrates that
the enhanced RiskAlert system more precisely identifies threats for at least two out of
three tested LLMs. For both models, a statistically significant difference in the classification results can be demonstrated using McNemar’s test.

Date of Award	2025
Original language	German (Austria)
Supervisor	Eckehard Hermann (Supervisor)