Where to Connect? Boosting MLLMs via Dynamic Gated Pathways across ALL ViT and LLM Layers
A central yet under-explored question in Multimodal Large Language Models (MLLMs) is where to connect---how visual information from a hierarchical vision encoder should be wired into the layer-wise semantics of a large language model. Mainstream MLLMs inject a fixed single-layer visual representation into the LLM and t...