Optimizing App Review Classification with Large Language Models: A Comparative Study of Prompting Techniques

Miriam Palmetshofer, David C. Schedl, Andreas Stöckl

Research output: Chapter in Book/Report/Conference proceedingsConference contributionpeer-review

Abstract

This paper explores the application of large language models in classifying app reviews into predefined cate-gories. The study leverages various sizes of Mistral models (tiny, small, medium, and large). It examines the effectiveness of dif-ferent prompting techniques, including zero-shot, one-shot, and few-shot prompting. The research utilizes a dataset of 6406 app reviews, previously categorized into bug/problem report, inquiry, and irrelevant, to evaluate the models' classification performance. Results indicate that few-shot prompting techniques significantly enhance the performance of all model sizes, suggesting that sophisticated prompting can almost offset the limitations of smaller models. The study also finds a diminishing return on investment with increasing model size when applying advanced prompting techniques. This highlights the importance of prompt design in optimizing large language models' performance and suggests that even smaller models can achieve competitive results with the right prompting strategy.

Original languageEnglish
Title of host publicationInternational Conference on Electrical, Computer, Communications and Mechatronics Engineering, ICECCME 2024
PublisherIEEE
Pages1-6
Number of pages6
ISBN (Electronic)9798350391183
ISBN (Print)979-8-3503-9119-0
DOIs
Publication statusPublished - 6 Nov 2024
Event4th International Conference on Electrical, Computer, Communications and Mechatronics Engineering, ICECCME 2024 - Male, Maldives
Duration: 4 Nov 20246 Nov 2024

Publication series

NameInternational Conference on Electrical, Computer, Communications and Mechatronics Engineering, ICECCME 2024

Conference

Conference4th International Conference on Electrical, Computer, Communications and Mechatronics Engineering, ICECCME 2024
Country/TerritoryMaldives
CityMale
Period04.11.202406.11.2024

Keywords

  • app re-views
  • classification
  • deep learning
  • large language models
  • natural language processing

Fingerprint

Dive into the research topics of 'Optimizing App Review Classification with Large Language Models: A Comparative Study of Prompting Techniques'. Together they form a unique fingerprint.

Cite this