Skip to content

Talk UI: A Multi-Modal, Conversational Interface for UI Prototyping

Published: at 01:00 AM


Hongyi Zhang, Mengxin Cao, Ron Chew



UI designers have long been sketching to realize the interfaces they have in mind for prototyping. However, there is still much that can be done to make designers’ life easier. Current UI design tools are time-consuming and repetitive: drawing sets of graphics and widgets, then specifying their interactions and behaviors often requires a large effort in code and configuration for small results. The iteration process cannot catch up with the speed the designer’s minds are operating at.

In this project, we propose “Talk UI”, a rapid multi-modal prototyping tool that takes advantage of natural language to facilitate the design process and accelerate prototype iteration. With Talk UI, designers will be able to create graphical interfaces as well as specify interactive outcomes through talking and demonstration. No more sketching is needed, and the UI design process will be as natural as having a conversation.


Our Talk UI leverages natural language and voice input from users and supports the following features that we would like to highlight: