TY - GEN
T1 - Optimizing App Review Classification with Large Language Models
T2 - 4th International Conference on Electrical, Computer, Communications and Mechatronics Engineering, ICECCME 2024
AU - Palmetshofer, Miriam
AU - Schedl, David C.
AU - Stöckl, Andreas
N1 - Publisher Copyright:
© 2024 IEEE.
PY - 2024/11/6
Y1 - 2024/11/6
N2 - This paper explores the application of large language models in classifying app reviews into predefined cate-gories. The study leverages various sizes of Mistral models (tiny, small, medium, and large). It examines the effectiveness of dif-ferent prompting techniques, including zero-shot, one-shot, and few-shot prompting. The research utilizes a dataset of 6406 app reviews, previously categorized into bug/problem report, inquiry, and irrelevant, to evaluate the models' classification performance. Results indicate that few-shot prompting techniques significantly enhance the performance of all model sizes, suggesting that sophisticated prompting can almost offset the limitations of smaller models. The study also finds a diminishing return on investment with increasing model size when applying advanced prompting techniques. This highlights the importance of prompt design in optimizing large language models' performance and suggests that even smaller models can achieve competitive results with the right prompting strategy.
AB - This paper explores the application of large language models in classifying app reviews into predefined cate-gories. The study leverages various sizes of Mistral models (tiny, small, medium, and large). It examines the effectiveness of dif-ferent prompting techniques, including zero-shot, one-shot, and few-shot prompting. The research utilizes a dataset of 6406 app reviews, previously categorized into bug/problem report, inquiry, and irrelevant, to evaluate the models' classification performance. Results indicate that few-shot prompting techniques significantly enhance the performance of all model sizes, suggesting that sophisticated prompting can almost offset the limitations of smaller models. The study also finds a diminishing return on investment with increasing model size when applying advanced prompting techniques. This highlights the importance of prompt design in optimizing large language models' performance and suggests that even smaller models can achieve competitive results with the right prompting strategy.
KW - Mechatronics
KW - Reviews
KW - Computational modeling
KW - Large language models
KW - Investment
KW - app re-views
KW - classification
KW - deep learning
KW - large language models
KW - natural language processing
UR - http://www.scopus.com/inward/record.url?scp=85215942625&partnerID=8YFLogxK
U2 - 10.1109/ICECCME62383.2024.10796376
DO - 10.1109/ICECCME62383.2024.10796376
M3 - Conference contribution
SN - 979-8-3503-9119-0
T3 - International Conference on Electrical, Computer, Communications and Mechatronics Engineering, ICECCME 2024
SP - 1
EP - 6
BT - International Conference on Electrical, Computer, Communications and Mechatronics Engineering, ICECCME 2024
PB - IEEE
Y2 - 4 November 2024 through 6 November 2024
ER -