Movie Gen, Were RNNs All We Needed? and Contextual Document Embeddings

In this issue: 3x industry news, 3x AI research papers

Sep 30th ~ Oct 6th

🗞️ Industry News in 1 Line

1. ♥ 6.2k Meta released MovieGen, a media foundation AI model which provides video personalization, text based video editing, and generate audio with text + video. A full 92 page research paper is available (which is covered in this issue!).

   

2. ♥ 1.7k Black Forest Labs, the startup with the key researcher behind Latent Diffusion and released Flux.1, has announced Flux1.1 Pro under API. Seems like SoTA to me.

   

3. ♥ 9.1k OpenAI just raised the largest venture capital round in history, with 6.6 billion raised at a valuation of 157 billion. On the other hand, ChatGPT announced Canvas, a new way of writing & coding projects with ChatGPT. It is hard to use with codes right now, but it has been perfect for writing in my opinion.

   

Were RNNs All We Needed?

Feng et al. [Universite de Montreal, Borealis AI]

♥ 721   RNN LM   bycloud’s pick  

Introduction to minGRU and minLSTM

Transformers are one of the most commonly used components in the field of AI but they suffer from scalability issues due to their quadratic computational complexity with respect to sequence length. This means if the input is 10 times longer, it takes 100 times more computing power to process. 

In recent weeks, we have seen new recurrent architectures like S4, and Mamba which use parallelization techniques such as the parallel prefix scan algorithm for efficient training. But this paper revisits much older recurrent models, LSTMs and GRUs, which were previously considered computationally expensive due to the need for backpropagation through time (BPTT).

The authors propose a key modification: removing hidden state dependencies from the input, forget, and update gates of these models. This seemingly simple change eliminates the need for BPTT and enables parallel training using the efficient parallel scan algorithm.

How Does minGRU and minLSTM Work?

minGRU and minLSTM are simplified and parallelizable versions of traditional GRU and LSTM recurrent neural networks. The core idea is to enable parallel training to address the computational bottleneck of sequential processing in traditional RNNs that rely on backpropagation through time.

Changes in minGRU Architecture

1. Removing Hidden State Dependencies: The key change is removing the dependence of the update gate (z_t) and candidate hidden state (h̃_t) on the previous hidden state (h_t-1). In a standard GRU, these components are calculated based on both the current input (x_t) and the previous hidden state. By making them solely dependent on the current input, each step's computation becomes independent of prior steps, enabling parallelization. This also eliminates the need for the reset gate (r_t), which controls the influence of the previous hidden state on the candidate state.

2. Removing Tanh: The hyperbolic tangent (tanh) function is used in standard GRUs to constrain the range of hidden states.  By removing it, this further reduces computational overhead without significant performance degradation, as the hidden state dependencies which were managed by tanh are already removed.

Changes in minLSTM Architecture

1. Removing Hidden State Dependencies: Similar to minGRU, the forget (f_t), input (i_t), and candidate cell state (c̃_t) calculations are made independent of the previous hidden state (h_t-1) which allows parallel computation.

2. Removing Tanh: The tanh activations, used in both the candidate cell state and hidden state calculations are removed.

3. Normalizing Gates and Removing Output Gate: The forget and input gates are normalized to ensure their sum equals 1. This ensures the cell state's scale remains consistent across time steps, simplifying training and mimicking the inherent time-independent scale of GRUs. Consequently, the output gate (o_t), which scaled the hidden state based on the cell state, becomes unnecessary and is removed. Furthermore, the cell state and hidden state become equivalent and are merged into a single hidden state.

Results and Real-World Implications of minGRU and minLSTM

Training Speed: minLSTM and minGRU are significantly faster to train than traditional LSTMs and GRUs. With a sequence length of 512, they achieve speedups of 175x and 235x, respectively, on a T4 GPU. This advantage grows even larger with longer sequences (over 1300x speedup for sequence length 4096). This massive speed improvement is mainly due to the removal of hidden state dependencies from the gates which enables parallel computation via the parallel scan algorithm.

Memory: While minLSTM and minGRU consume ~88% more memory than their traditional counterparts and Mamba uses ~56% more memory than minGRU due to the larger computational graph created by the parallel scan algorithm, the substantial gains in training speed outweigh the increased memory footprint. Training runtime is generally the bottleneck in RNN training, not memory.

Benefits of the Minimal Architectures:

• Parallelization: The most significant advantage is the ability to train these networks in parallel using the parallel scan algorithm. This leads to substantial speed improvements compared to the sequential BPTT training required by traditional LSTMs and GRUs.

• Reduced Parameters: minGRU and minLSTM use fewer parameters than their traditional counterparts, making them more memory-efficient. minGRU requires only about 13-33% of the parameters of a GRU, and minLSTM requires 15-38% of the parameters of an LSTM, depending on the relative sizes of the input and hidden state dimensions.

• Comparable Performance: Despite these simplifications, minGRU and minLSTM achieve performance comparable to state-of-the-art sequence models, including Transformers and recent recurrent architectures like Mamba, across various tasks.

Contextual Document Embeddings

Morris and Rush [Cornell University]

♥ 2.3k   LLM Embedding

Introduction to Contextual Document Embeddings

When we talk to computers, we are just storing bits of zeroes and ones in the memory. When you type “red”, the computer stores a string of text in its memory but it doesn’t understand the concept of red. Embeddings are a way to store the idea of red in the computer’s memory.

Currently, the embeddings don’t have the necessary contextual awareness in current neural document embedding models for retrieval tasks. Existing methods treat each document in isolation by encoding them independently without considering their relationship to other documents in the corpus. For example, the term "draft" might have different importance in the context of sports articles versus legal documents. 

This paper contains two complementary solutions to introduce Contextual Document Embeddings (CDE):

1. Contextual Contrastive Learning: This method modifies the training process by incorporating the notion of "neighboring documents." 

2. Contextual Encoder Architecture: This introduces a new architecture that explicitly injects information about neighboring documents into the embedding process. 

How do Contextual Document Embeddings Work?

The core idea of Contextual Document Embedding is to make document embeddings (representations of text documents as numerical vectors) more aware of their context. Traditionally, when we convert a document into a vector, we only look at that document in isolation. But this new approach considers other documents in the corpus when creating the embedding.

1. Two-Stage Process: The model uses a two-stage process to create these contextual embeddings: 

   1. First Stage: The model takes a subset of documents from the corpus and embeds them using a special embedding model. This creates a set of "context vectors" that represent the broader context of the corpus. 

   2. Second Stage: When embedding a specific document, the model combines the original document's text with the context vectors created in the first stage. It then uses another embedding model to create the final representation of the document. 

2. Handling Queries: For search queries, the model uses a similar approach. It combines the query text with the same context vectors used for documents. This allows the query representation to be aware of the corpus context too.

3. Flexibility and Efficiency: The model is designed to be flexible and efficient: 

   1. It can work without context by using a special "null" token instead of context vectors.

   2. During training, it shares context within batches of documents to save computation time.

   3. When indexing a new corpus, it can cache the first-stage context vectors to speed up processing.

4. Training Improvements: The researchers also introduce two key improvements to the training process: 

   1. Contextual Batching: Similar documents are grouped together instead of randomly sampling documents for training. This creates "mini-domains" within each batch which helps the model learn to handle different contexts more effectively. 

   2. Two-Stage Gradient Caching: This is a technical optimization that allows the model to work with larger batches and more context samples without running out of memory. 

By doing all this, this model tries to mimic how humans understand documents - not in isolation, but in the context of other related documents. 

Results and Evaluation of Contextual Document Embeddings

In the following table, we can see that the suggested technique improves performance compared to standard biencoder training. This paper also showed that creating more challenging training batches through contextual batching leads to more effective learning. We also saw that while large batch and cluster sizes are beneficial without filtering, smaller, harder clusters, achieved through filtering false negatives significantly improve performance. This suggests that focusing on challenging examples within a context is more crucial than simply increasing batch size.

Though they were designed for finding text, however this context-aware approach also helps with grouping similar things, categorizing things, and figuring out how similar words or sentences are. This paper showed that if you don't give the model information about the context of the words it's looking at, it does a worse job, especially at judging similarity. This proves that knowing the context of words is really important for creating good word representations.

Movie Gen: A Cast of Media Foundation Models

The Movie Gen team @ Meta

♥ 5.8k   Video Gen

Introduction to Movie Gen

Current AI models can generate pretty realistic images, but they struggle in creating high-quality, coherent, and customizable video content with synchronized audio from text prompts. The researchers behind MovieGen have developed a suite of AI models that aims to compete existing commercial systems like Runway Gen3, LumaLabs, and OpenAI Sora in video quality Check out the sample video below!

They've also introduced new capabilities for video personalization and precise editing, which they are missing from current commercial systems. Additionally, their audio generation model, MovieGen Audio, surpasses prior state-of-the-art systems for sound effect and music generation, as well as audio extension. 

Inner-Workings of Movie Gen

Movie Gen is an advanced model, and it uses a sophisticated approach to processing and encoding text prompts for video generation. In this section, we have tried to explain how it works but we left out many important parts. Please read the entire paper provided at the bottom of this post if you want to learn more. 

1. Text Input: The system starts with a text prompt provided by the user. This prompt describes the desired video content.

2. Multiple Text Encoders: Instead of using a single text encoder, this model uses three different pre-trained text encoders

   • UL2: This encoder is trained on a vast amount of text-only data and it provides strong text reasoning capabilities.

   • Long-prompt MetaCLIP: This is a modified version of the MetaCLIP text encoder, fine-tuned to handle longer text inputs (up to 256 tokens instead of the original 77). It provides text representations that are well-aligned with visual concepts.

   • ByT5: This is a character-level encoder, specifically used for encoding visual text - parts of the prompt that explicitly request text to appear in the generated video.

3. Text Processing: Each of these encoders processes the input text prompt in its own way

   • UL2 and Long-prompt MetaCLIP process the entire prompt, creating prompt-level embeddings.

   • ByT5 focuses on character-level encoding, particularly for parts of the prompt requesting visible text in the video.

4. Embedding Transformation: The outputs from each encoder are then processed

   • Each output goes through a separate linear projection layer to transform it.

   • The transformed outputs are then normalized using LayerNorm layers.

   • This process ensures all embeddings are in the same 6144-dimensional space, making them compatible for combination.

5. Embedding Concatenation: The processed embeddings from all three encoders are concatenated into a single, comprehensive text embedding.

6. Text Output: This combined embedding is used as the conditioning input for the video generation backbone. It contains a rich representation of the text prompt with semantic understanding, character-level details, and visual-textual alignments.

Temporal Autoencoder (TAE) in MovieGen Video Model

The MovieGen Video model uses a Temporal Autoencoder (TAE) for video handling. Here's an explanation of how it works:

1. Spatio-Temporal Compression: The TAE is designed to compress both the spatial and temporal dimensions of input videos and images. It takes RGB pixel-space videos (or images) and encodes them into a learned, compressed latent space. This compression reduces the input dimensions by a factor of 8 in each of the spatial and temporal dimensions. 

2. Architecture Inflation: The researchers started with an image autoencoder architecture and "inflated" it to handle video data. This inflation involves adding following temporal parameters to the existing spatial architecture:

   1. They added 1D temporal convolutions after each 2D spatial convolution.

   2. They added 1D temporal attention layers after each spatial attention layer.

   3. Temporal convolutions use symmetrical replicate padding to maintain temporal consistency.

3. Temporal Processing: Downsampling in the temporal dimension is achieved using strided convolutions with a stride of 2. This allows the model to handle videos of varying lengths, including single-frame "videos" (i.e., images).

4. Latent Space: The researchers found that increasing the number of channels in the latent space improved both reconstruction and generation performance. They settled on using 16 channels (C = 16) in the latent space.

5. Training Process: The spatial parameters of the TAE are initialized using a pre-trained image autoencoder and Temporal parameters are then added to "inflate" the model for video processing. The TAE is jointly trained on both images and videos, with a ratio of 1 batch of images to 3 batches of videos.

This approach allows the MovieGen Video model to efficiently process and generate videos by working in a compressed latent space, reducing computational requirements while maintaining the ability to produce high-quality, temporally coherent video outputs.

Transformer Backbone

Movie Gen’s Video Transformer backbone have model parallelism applied.

On the left side, it shows the Transformer backbone and they’ve color-coded different model parallelizations used to shard their 30B model.

On the right, there are more details about its feature dimensions in a number of key steps during the most expensive stage of Movie Gen Video training. It processes 768 px video inputs with a per-sample sequence length of 73K tokens.

Evaluation of Movie Gen

When researchers publicly release results, they often use the cherry-picked "best" samples. To address this potential bias and ensure a fairer comparison, the researchers took a methodical approach when comparing MovieGen Video to OpenAI's Sora model. For each prompt, they generated five videos using MovieGen Video and manually selected one. This allows them to copy the selection process of publicly released examples from other models. This approach aims to level the playing field and provide a more accurate comparison of capabilities.

Next, different models produce outputs in different resolutions and aspect ratios. To mitigate potential annotator bias stemming from these differences, the researchers downsampled MovieGen Video's outputs to match the resolution and aspect ratio of the comparison model for each evaluation. After all that, they calculated the net win rate as shown in the table below. 

Here we can see that MovieGen Video has strong performance in overall quality and outperforms Runway Gen3 as well as LumaLabs significantly, while showing moderate improvement over OpenAI Sora. The model excels in various quality aspects such as motion naturalness and frame consistency which indicates its ability to generate realistic videos that respect physics and maintain consistency. However, it still faces some challenges in motion completeness compared to Kling1.5.