Natural Language Interface for Data Visualization with Deep Learning Based Language Models

In this work we investigate the possibilities of integrating a Deep Learning language model for a Natural Language Interface (NLI) of an information visualisation software. For this purpose, we have developed a prototype web application that uses the deep learning model OpenAI Codex from the GPT3 family to create visualisations from text input. For comparison, we created a second prototype with a classical NLP approach based on NL4DV toolkit (with subtasks like part-of-speech (POS) tagging, entity recognition, and dependency parsing) and an almost identical interface. The two variants were subjected to a study with test persons, and the advantages and disadvantages of the two approaches and the suitability for the most common visualisation types were investigated. The Deep Learning approach offers greater expressiveness for describing the graphics, but also the danger of not always being entirely comprehensible. The participants were able to use it to create more complex visualisations, but also sometimes had problems finding the right text input to solve the tasks. In our preliminary usability study, the Deep Learning prototype performed slightly better than the comparison prototype and achieved a useful usability score.