Natural Language Interface for Data Visualization with Deep Learning Based Language Models

Research output: Contribution to conferencePaperpeer-review


In this work, we investigate the possibilities of integrating a deep learning language model for a natural language interface (NLI) of information visualization software. For this purpose, we develop a prototype web application that uses the deep learning model OpenAI Codex from the generative pretrained transformer 3 (GPT-3) family to create visualizations from text input. For comparison, we create a second prototype through a classical natural language processing (NLP) approach based on the Natural Language for Data Visualization (NL4DV) toolkit (with subtasks like part-of-speech (POS) tagging, entity recognition, and dependency parsing) and an almost identical interface. The two variants are subject to a study with test persons, and the advantages and disadvantages of the two approaches and their suitability for the most common visualization types are investigated.
The deep learning approach offers greater expressiveness in terms of describing the graphics but also carries the danger of not always being entirely comprehensible. Participants are able to use this approach to create more complex visualizations but, sometimes, face problems in terms of finding the appropriate text input to solve tasks. In our preliminary usability study, the deep learning prototype performs slightly better than does the comparison prototype and achieves a useful usability score.
Original languageGerman (Austria)
Number of pages7
Publication statusPublished - 2022
Event26th International Conference Information Visualisation (IV) - Technische Universität Wien, Wien, Austria
Duration: 19 Jul 202222 Jul 2022


Conference26th International Conference Information Visualisation (IV)
Internet address

Cite this