Vergleich und Evaluierung von LLM-Agenten Frameworks für Prozessautomatisierung

Abstract

This master thesis explores and evaluates the utilization of large language models (LLMs) in agent systems, with a particular emphasis on process automation and data protection requirements. It presents a systematic analysis of current agent frameworks, including LangChain, LlamaIndex, CrewAI, OpenAI Agents SDK, PydanticAI, and n8n. This analysis is motivated by rapid technological advancements and the increasing significance of autonomous AI agents in business environments. The study aims to provide a practical evaluation of these frameworks in terms of functionality, documentation quality, integration capabilities, scalability, security, and legal compliance requirements. The thesis involves the development of a prototype LLM-based agent that is capable of autonomously analyzing and classifying incoming IT support tickets by integrating and leveraging a variety of models. This implementation provides a realistic foundation for the evaluation of the practical potential of these technologies. The frameworks are evaluated based on predefined criteria. The results show that CrewAI and n8n offer high flexibility, consistent outputs, and quick implementation, whereas frameworks with a higher level of abstraction tend to limit transparency and customization. The thesis also identifies key challenges such as hallucinations, data privacy concerns, cost management, and reliance on external service providers. It provides tailored recommendations for framework selection based on specific organizational needs. Finally, the study outlines its limitations, particularly with regard to its focus on a subset of LLMs and frameworks, and suggests potential areas for future research and expansion.

Date of Award	2025
Original language	German (Austria)
Supervisor	Gabriel Kronberger (Supervisor)