Browsing by Subject "Multimodal fusion"

Now showing 1 - 1 of 1
  • Thumbnail Image
    Publication
    Open Access
    Saffe: multimodal model composition with semantic‑alignment fusion of frozen encoders
    (Springer, 2025-07-07) Kulasekara, Maithri; Inglés‑Romero, Juan F.; Imbernón, Baldomero; Abellán, José L.; Ingeniería y Tecnología de Computadores
    Transformer-based multimodal models often require expensive, full-model training on task-specific all-modality datasets to achieve high accuracy on targeted downstream tasks. To reduce this significant cost, we introduce SAFFE, a methodology for building accurate, task-specific multimodal models with minimal training, using only standard GPU hardware. SAFFE leverages off-the-shelf, pre-trained, frozen unimodal encoders for each input modality (e.g., text, image, or audio) and connects them through a lightweight, trainable component called the FusionAlign Module (FAM). FAM is a bottleneck mid-fusion neural network, trained on the target data set to align the outputs of the independently pre-trained unimodal encoders. This approach eliminates the need for end-to-end training while maintaining strong accuracy for the downstream task. As a proof of concept, we validate SAFFE on image retrieval and language understanding tasks. SAFFE-derived models outperform state-of-the-art multimodal systems on datasets such as CIFAR-10, ImageNet-100, and COCO, achieving competitive results with significantly fewer trainable parameters and training time.