Skip to content

Latest commit

 

History

History
95 lines (62 loc) · 4.33 KB

README.md

File metadata and controls

95 lines (62 loc) · 4.33 KB
Raw

Clip_matrix

3d obj mesh generator using text prompts

Open in Google Colab

[example of ClipMatrix 3d mesh output] Figure: rotated camera views showing an example ClipMatrix 3d mesh output

for text prompt "Mecha Warrior in Sci Fi Giger style"

open_clipmatrix : very short intro

An open source ClipMatrix AI tool for text_to_3d mesh mapping; fixed deformable mesh template without rigging. Have fun making funny 3d creatures with it.

Use provided jupyter notebook to define text prompts and use the tool.

I suggest to run in Colab, import and installs defined for this case.

With small modifications you could also get it to run locally.

In any case, use at least a 16GB GPU card for smooth performance.

Introduction to ClipMatrix: text controlled 3D mesh deformation and stylization.

A novel type of text controlled AI generative art: you give a text and get a 3D textured shape from it. This code is loosely based on Njetchev recent paper, but has many more tricks inside. Feel free to play with it, make art or even better: experiment with new ML techniques based on it. Please cite Njetchev twitter account @NJetchev or the paper when using this code, posting artwork from it or modifying it.

Inputs from you:

  • starting mesh template
  • text prompt to be interpreted by CLIP

Outputs:

videos of the resulting generated mesh. by default saved in clip_videos folder

Cool features:

  • check Njetchev account for artwork and inspiration @NJetchev
  • we have also created a library of clipmatrix GLB models that are animated using blender workflows and other rigging tools.
  • support for arbitrary template meshes for starting shapes
  • enhance starting templates by cloning
  • define regions of interest for fine part control
  • shadow mapping
  • normal mapping
  • Sobolev smoothing
  • neural shaders for advanced animation effects
  • you can export the 3D mesh file and print your work in a 3D printer service

Libraries and references used: quite many, but I am most thankful to

  • OpenAI's CLIP
  • Meta's Pytorch3D
  • the authors of this paper
  • the researchers of SMPL , the sample static model.obj is taken from their website for research purposes
  • @nonlethalcode for helpful suggestions and testing of the UI of this code

updates of Open_ClipMatrix:

  • this code is related to the recent ClipMatrix Creatures art project. If you appreciate it, holders of NFT Creatures will get earlier exclusive access to new versions of the ClipMatrix tool

  • currently there are no riggable models inside, just static templates. I may add them in the future, not certain yet. It can complicate the code and increase runtimes a lot.

  • In this version main focus is on a small and agile demo that is fun and fast to play with.

CHANGELOG:

  • 0.65 initial version

  • 0.651 few bug fixed, options to disable shaders and normal maps

  • 0.7 added graph convolutions, fixed library paths; default regularization strength settings favor stronger mesh deformation

  • 0.8 better shadows, fixed library paths, support multiple template clones

Support

If you create a fun image with this model, please show your result and message us on twitter at @jags111 or @NeuralismAI .

You can join the NEURALISM AI DISCORD or JAGS AI DISCORD Share your work created with this model. Exchange experiences and parameters. And see more interesting custom models.

Support us in Patreon for more future models and new versions of AI notebooks.

Thank you for being awesome!