Skip to content

Latest commit

 

History

History
102 lines (80 loc) · 2.86 KB

README.md

File metadata and controls

102 lines (80 loc) · 2.86 KB
Raw

SD-T2I-360PanoImage

repository for Diffusion360: Seamless 360 Degree Panoramic Image Generation based on Diffusion Models

News!!!

  • 2024.5.20. I recommend to install 0.20.0<= diffusers <= 0.26.0. The higher diffusers version will get an over-saturated SR result.
  • 2024.5.17. A ComfyUI plugin of this repo is released! See https://github.com/ArcherFMY/Diffusion360_ComfyUI for more information

Text-to-360Panorama

a living room

the mountains

the times square

Single-Image-to-360Panorama

samples-i2p

Requirements

  • torch
  • torchvision
  • torchaudio
  • diffusers
  • accelerate
  • xformers
  • triton
  • transformers
  • realesrgan
  • py360convert

Installation

git clone https://github.com/ArcherFMY/SD-T2I-360PanoImage.git
cd SD-T2I-360PanoImage
pip install -r requirements.txt

Getting Started

Download Models

Download models from Baidu Disk. Unzip models.zip into the root directory of the project.

${ROOT}  
|-- data  
|   |-- a-living-room.png
|   |...
|-- models  
|   |-- sd-base
|   |-- sr-base
|   |-- sr-control
|   |-- RealESRGAN_x2plus.pth
|-- txt2panoimg
|-- img2panoimg
|...

For users who want the Single-Image-to-360Panorama models, please download the additional models from Baidu Disk, and unzip it into the 'models' directory. Or download the models from Hugging Face

Inference

Text-to-360Panorama

import torch
from txt2panoimage import Text2360PanoramaImagePipeline

prompt = 'The living room'
input = {'prompt': prompt, 'upscale': False}
model_id = './models'
txt2panoimg = Text2360PanoramaImagePipeline(model_id, torch_dtype=torch.float16)
output = txt2panoimg(input)

output.save('result.png')

see more in demo_t2p.py

Single-Image-to-360Panorama

import torch
from diffusers.utils import load_image
from img2panoimg import Image2360PanoramaImagePipeline

image = load_image("./data/i2p-image.jpg").resize((512, 512))
mask = load_image("./data/i2p-mask.jpg")
prompt = 'The office room'
input = {'prompt': prompt, 'image': image, 'mask': mask, 'upscale': False}
model_id = 'models'
img2panoimg = Image2360PanoramaImagePipeline(model_id, torch_dtype=torch.float16)
output = img2panoimg(input)

output.save('result.png')

see more in demo_i2p.py

Use Text-to-360Panorama in ModelScope

see here for more information.

License

This code is released under the Apache License 2.0 (refer to the LICENSE file for details).