(left to right) James Davis, Assistant Professor of ECE and Yung-Hsiang Lu, Professor of ECEE

Artificial intelligence is becoming more powerful, but some of the hardest problems still involve making sense of many different kinds of information at once. For example, answering a question might require not just listening to someone’s words but also interpreting their tone or gestures. Or creating a movie clip might mean weaving together a script, a director’s style, and what an audience expects.

Yung-Hsiang Lu, professor in the Elmore Family School of Electrical and Computer Engineering, and James Davis, assistant professor of ECE, have received a grant from the National Science Foundation (NSF) to push forward a promising approach for tackling these challenges.

The research focuses on multi-stream architectures, a new kind of AI design that can process multiple types of data, or “modes,” at the same time. Traditional deep learning systems often struggle when they need to combine text, images, audio, or video, but multi-stream systems have shown stronger results.

Despite their potential, multi-stream methods have not yet been widely adopted. Lu and Davis aim to change that by building a library of models and software tools that researchers and developers everywhere can use. Their project has three main goals:

1. Create high-performance AI models that can handle tasks like detecting objects in images, matching text descriptions to images, and analyzing audio and video together.
2. Develop algorithms that make these systems faster and more efficient.
3. Provide new insights that will help guide the design of future multi-stream systems.

Importantly, the Purdue team plans to make their cyberinfrastructure openly available. This means other researchers and practitioners will be able to use, test, and build upon their work.

“Multi-stream architectures underlie an emerging class of neural networks that achieve state-of-the-art performance on discriminative and generative tasks in computer vision,” said Davis. “Although the potential of multi-stream architectures has been demonstrated on individual tasks, we do not have a systematic understanding of how to generalize this technology nor deploy it efficiently. This grant will develop cyberinfrastructure to facilitate broader engineering and research adoption of this technology.”

The project will test the new AI systems across a wide range of hardware, from small devices like smartphones to large high-performance computing systems. The researchers will also compare their results to current state-of-the-art methods to measure improvements in both speed and accuracy.

The NSF award recognizes the intellectual merit and broader impacts of the project, supporting both fundamental advances in deep learning and the development of resources that will benefit the wider scientific and engineering communities.