Speech to Speech with unsupervised training — Project link Paper link Translatotron 3 is a novel approach for training speech-to-speech translation models in a fully unsupervised manner using only monolingual speech-text datasets. It incorporates a masked autoencoder, unsupervised embedding mapping, and back-translation. In tests, it outperformed traditional cascade systems in Spanish-English translation tasks, achieving a significant 18.14…

Stable Diffusion, Score Distillation, and Adversarial Loss — Project link Adversarial Diffusion Distillation (ADD) is a new training approach that samples large-scale foundational image diffusion models efficiently in 1–4 steps while maintaining high image quality. It combines score distillation, using existing image diffusion models as a teaching signal, with an adversarial loss to ensure fidelity in low-step sampling…

Stable Diffusion for image segmentation! — Paper link Project link Producing quality segmentation masks in computer vision is a key challenge, especially for zero-shot segmentation across various image styles without specific training. The new method utilizes self-attention layers in pre-trained stable diffusion models, which have learned to recognize objects, to address this challenge. This approach involves…

Diffusion models, now with DPO! — Paper link DiffusionDPO is a new method for aligning text-to-image diffusion models with human preferences, adapted from Direct Preference Optimization. Unlike traditional approaches that use high-quality images and captions for fine-tuning, DiffusionDPO optimizes models based on human comparison data. …

Smaller, better, stronger! — Paper link Project link Orca 1 improved upon conventional instruction-tuned models by learning from rich signals, such as explanation traces, and showed superior performance on challenging benchmarks like BigBench Hard and AGIEval. Orca 2 builds on this by focusing on enhancing the reasoning abilities of smaller language models (LMs). Unlike…

RAG, but this time with additional notes — Paper link Retrieval-augmented language models (RALMs) are enhanced by using external knowledge sources, but they face challenges such as retrieving irrelevant information and failing to use their inherent knowledge. They also struggle to determine when they have sufficient information for accurate answers, often not indicating when knowledge is lacking. To…

Forecasting weather at scale — Paper link Blogpost link MetNet-3 is a neural network designed for weather forecasting, capable of making predictions with high temporal (up to every 2 minutes) and spatial (1 km) resolution. Trained using atmospheric observations, it forecasts various conditions such as precipitation, wind, temperature, and dew point up to 24 hours…

End-to-end LLM for QA and speech continuation trained on spectrograms — Paper link Project link A novel approach adapts pre-trained LLMs for question answering and speech continuation by incorporating a pre-trained speech encoder. This enables the model to handle speech inputs and outputs. The model is trained end-to-end on spectrograms. The training objective jointly supervises speech recognition, text continuation, and speech…

Visual language foundation model with a visual expert module — Paper link Code link CogVLM is a new open-source visual language model noted for its deep integration of visual and language features. It surpasses conventional models that align image features superficially with language model inputs. This integration is achieved through a trainable module within the attention and FFN layers, allowing…