What do LLMs dream of? — Paper link Project link Code link DreamLLM is a new learning framework designed for Multimodal Large Language Models. It emphasizes the synergy between understanding and generating both text and images. Its two main principles are generative modeling in the raw multimodal space, avoiding the constraints of external feature extractors, and…

Free lunch in Generative Networks — Paper link Project link Code link In this paper, the authors explore the potential of diffusion U-Net for improved generation quality. While the U-Net's main structure aids in denoising, its skip connections add high-frequency features, sometimes overshadowing the main backbone's semantics. Based on this understanding, the authors introduce "FreeU", a…

Marrying LLMs with EA — what could possibly go wrong? — https://arxiv.org/abs/2309.08532 EvoPrompt is a new framework that uses evolutionary algorithms EAs to automate the process of creating prompts for LLMs. Traditional LLMs require manually crafted prompts which can be labor-intensive. By integrating EAs with LLMs, EvoPrompt can generate coherent and readable prompts without needing gradients or parameters. Starting with a…

One-shot segmentation with the required granularity — Paper link Large vision-language models like Stable Diffusion have made advances in tasks like image editing and 3D shape generation. A new method, SLiMe, has been proposed to use these models for image segmentation with as little as one annotated sample. The process involves extracting attention maps from the SD…

MLP — now for time series Paper link Code link Time-series datasets in real-world scenarios often contain multiple variables with intricate patterns. While deep learning models using recurrent or attention mechanisms are commonly used to handle this complexity, recent studies show that basic linear models can sometimes outperform them on…

Ungrokking and semi-grokking Paper link The concept of “grokking” refers to a phenomenon in neural networks where a network that initially memorizes the training data but performs poorly on new, unseen data eventually learns to generalize well after further training. The authors propose that this happens because there are two…

ViT + MAE + Contrastive Learning + PED = SOTA Paper link Contrastive Feature Masking Vision Transformer (CFM-ViT) is a new method for image-text pretraining that enhances open-vocabulary object detection. CFM-ViT integrates the masked autoencoder with contrastive learning, focusing on joint image-text embedding reconstruction instead of typical pixel space, which…

LLMs for recommendations — Paper link Recent advancements have enabled Large Language Models (LLMs) to use external tools and execute multi-step plans, enhancing their performance in various tasks. However, their proficiency in handling personalized queries, such as recommendations, hasn’t been deeply studied. To address this, an LLM-based recommender system named RecMind has been developed. …

Gotta Track ’Em All! — Paper link Code link Project link In the paper, a new video motion prediction method called CoTracker is introduced. Traditional methods either estimate motion for all points in a video frame simultaneously using optical flow or track each point’s motion separately throughout the video. …