Teaching AI to See Faster

As artificial intelligence systems take on larger tasks, the amount of information they process has become a practical concern. More tokens can mean higher costs for users, slower responses and greater demand for computing infrastructure, including data centers. Video adds another layer to that issue because a few seconds can contain hundreds or thousands of visual details that a model must process before answering a question or making a decision.

Manan Suri, a Ph.D. student in the University of Maryland Department of Computer Science, is studying ways to help AI systems handle video with zero tokens. Advised by Distinguished University Professor Dinesh Manocha, Suri’s research focuses on improving AI agents by providing richer context, including methods that enable models to internalize information rather than repeatedly process the same data.

Suri’s project, Frames2LoRA, explores a new approach to vision-language models, which process visual and written information together. The method turns a video into a Low-Rank Adaptation, or LoRA, adapter in a single forward pass. That allows a frozen model, whose core parameters remain unchanged, to answer questions about the video without placing visual tokens in its context window at query time.

“My research tries to improve agents to perform better by giving them better context,” Suri said. “The end goal is to improve the context so that the agent can perform better on the tasks that it ends up doing, which is beneficial to everyone.”

Challenge of video

Video is difficult for AI systems because it contains information across time and space. A video is made up of frames, and each frame includes visual information across height and width. As videos get longer or sharper, token counts can rise quickly because the model must represent more frames and more visual detail in each frame.

That growth affects both performance and cost. More tokens require more computation, which can make large models more expensive to run and slower to respond, especially when the same video must be reviewed for multiple questions.

Suri said the issue is not only the amount of data but also how that data affects reasoning.

“When you give AI too much context, it’s not able to reason through it,” Suri said.

The problem is often described as “context bloat,” when information fills a model’s context window, making it harder for the system to identify what matters. Suri compared it to information overload.

The same challenge appears when AI systems repeatedly review video data.

“This essentially has two consequences,” Suri said. “For a long video, the model has too many visual tokens in its context window, and for a lot of tasks, parsing a huge number of tokens introduces latency that we do not need.”

A different path

Frames2LoRA addresses the problem by mapping video information into the model’s weights through an adapter. Instead of requiring the model to repeatedly view the video through visual tokens, the method creates a reusable representation of the video for later questions.

In the project, a perceiver hypernetwork reads intermediate representations generated as a frozen vision-language model encodes a video. It then predicts a LoRA adapter directly from the video in one forward pass, without the iterative updates used in standard LoRA fine-tuning.

The method was trained using SmolVLM2 models with 500 million and 2.2 billion parameters on video summarization and captioning tasks. According to Suri, Frames2LoRA enabled the same frozen model to answer questions using only the adapter, with no visual tokens in context at query time.

Removing visual tokens from the query-time context can reduce memory use, lower latency and make repeated questions about the same video more efficient. It also leaves more room in the context window for text instructions, prior dialogue or other information needed for the task.

Across the evaluation, the method reduced answer-time visual-token load by up to 1,500 times and improved time to first token by six to 80 times while preserving outputs tied to the video. The project also found that adapters generated from separate video segments could compose in rank space, suggesting a possible path for processing longer videos in chunks.

“What Frames2LoRA did for us is that it gave us a very strong proof of concept that you can map context in a single forward pass,” Suri said.

Toward real-world agents

Efficiency becomes especially important as AI systems move toward tasks that depend on continuous visual input. Robots in homes, factories or other settings may need to process a constant visual stream while responding quickly to changing conditions.

“Videos are very relevant when you look at real-world embodied tasks, such as a robot in your home or a robot in a factory,” Suri said. “Its input is a constant visual stream.”

Suri is now exploring whether the same idea can extend beyond perception. Frames2LoRA showed that video context can be internalized in one pass, but future work may examine whether that internalized context can help AI systems act on what they see.

For Suri, that next step is the central question.

“It’s important to know that we can use this to understand videos,” Suri said. “The next question is whether we can make that understanding actionable once the video has been internalized.”

—Story by Samuel Malede Zewdu, CS Communications