GOR - General Online Research 2023

Description

P 1.5: Postersession
Time: 21/Sept/2023: 2:15pm-3:30pm · Location: Atrium

AI-Enabled Data Analysis Quality: Addressing A Knowledge Gap

Daniela Wetzelhütter1, Dimitri Prandner2

1University of Applied Sciences Upper Austria, Austria; 2Johannes Kepler University Linz, Austria

Relevance & Research Question: During quantitative data analysis various errors can occur, e.g. use of inappropriate analysis methods, mishandling missing data or overfitting overly complex model. It can be assumed that, apart from deliberate deception, a lack of knowledge in particular increases the risk of erroneous results. And this is exacerbated by new AI-supported possibilities for data analysis. This poses a significant challenge, as one may now be enabled to conduct simple as well as complex data analysis, with only limited knowledge. Consequently, our research question is: What are the pitfalls of using AI-based data analysis?

Methods & Data: Our approach illustrates the potential for error in AI-based data analysis: descriptive statistics, factor and regression analysis. We use published replication datasets to do so and compare the published results with AI-based ones. We use AI-tools to generate the required syntax for SPSS, Stata and R, before executing the code and providing the AI with results to interpret. We use replication data available at AUSSDA (https://data.aussda.at/dataverse/AUSSDA/?q=replication), which provides us with both data and a correct syntax.

Results: While tools like Chat GPT can solve simple calculations and equations correctly, this is not the case when it comes to data analysis. (1) Descriptive data analysis tends to include mistakes, reporting such results may have severe consequences. (2) However, the use of a syntax generated by Chat-GPT (e.g., for conducting exploratory factor analysis) seems more promising - but it requires expertise (to adapt e.g. Missing value handling, criteria, extraction, rotation). Though, the information needed can be requested to overcome knowledge gaps. However, the more complex the procedure being considered for application, the higher the risk that the syntax will not work.

Added Value: Because AI-supported programming for data analysis offers valuable opportunities, we illustrate that such option need to treated with uttermost care. However, our results are limited due to the continuous, rapid development of AI-tools for data analysis.

Excerpt Literatur: Burger, B., Kanbach, D. K., Kraus, S., Breier, M., & Corvello, V. (2023). On the use of AI-based tools like ChatGPT to support management research. European Journal of Innovation Management, 26(7), 233-241.

Period	20 Sept 2023 → 22 Sept 2023
Event title	GOR - General Online Research 2023
Event type	Conference
Location	Kassel, GermanyShow on map
Degree of Recognition	International