2D SDS Experimentation Framework

(A better name TBD.)

Overview

This framework is designed for experimenting with Score Distillation Sampling (SDS) and its variations on 2D representations.

If you are not familiar with SDS, please refer to this paper.

Why Choose This?

SDS and similar optimization-based methods are typically implemented in 3D generation frameworks like Threestudio. However, conducting experiments in 3D can be:

• Time-consuming
• Complex, as results may be affected by factors like 3D representations or camera configurations.

Sometimes, the goal is simply to identify the best optimization-based diffusion sampling strategy without being tied to specific tasks, such as 3D, SVG, or 4D content generation.

This framework addresses that by implementing optimization-based methods on 2D representations (e.g., pixels, latents).

It provides a simpler, more flexible, and user-friendly way to experiment with various settings and techniques.

Example

This framework records and visualizes diverse intermediate results for your experiments.

Here are some examples:

Example: experiment SDS algorithm with different guidance scales.

Figure: SDS algorithm with different guidance scales.

Example: test different algorithms with different timestep strategies

We implement numerous timestep strategies. The following is visualization of Dreamtime strategy:

Figure: Visualization of Dreamtime strategy.

With results on SDS and VSD:

Figure: Results on SDS and VSD using Dreamtime strategy.

Example: test ISM with different hyperparameters

Figure: ISM tested with different hyperparameters.

Example: test DDS with different prompts

Figure: DDS tested with different prompts.

Example: visualization of the PDS gradient

Figure: PDS gradient visualization.

How to use it?

For installation, refer to INSTALL.md.

For basic usage, refer to USAGE.md.

If you are interested in developing new algorithms, refer to DEVELOPMENT.md.

Implemented Features

Algorithms

• Score Distillation Sampling (SDS)
• Variational Score Distillation (VSD)
• Interval Score Matching (ISM)
• Delta Denoising Score (DDS)
• Posterior Distillation Sampling (PDS)

2D representations:

• pixel
• latents
• gaussians 2D

Weight and timestep schedule:

• dreamfusion
• dreamtime
• hifa
• linear
• random decay
• etc.

Acknowledgement

Some code is referenced from Threestudio. Many thanks to their great work!

Also, some designs are inspired by prolific_dreamer_2d, which also implemented some nice features.