CRM

CRM (Convolutional Reconstruction Model) is Tsinghua University's image-to-3D model. Two multi-view diffusion models generate six views of the object and their coordinate maps, a convolutional UNet turns them into a triplane, and FlexiCubes extracts a coloured mesh. LibreYOLO runs it in plain PyTorch through the LibreGen factory.

Tâches
3d generation
Installer
pip install "libreyolo[gen]"
Niveau de support
Niveau parallèle, depuis v1.8.0. Une interface produit distincte avec sa propre factory et son propre contrat.
Origine
CRM par Tsinghua University (TSAIL), CreativeML OpenRAIL++-M. Article, source
Licences
Code MIT and Apache-2.0, poids CreativeML OpenRAIL++-M. Usage commercial

Install

bash
pip install "libreyolo[gen]"

texture=True also needs the MIT-licensed xatlas package (pip install xatlas) for the UV unwrap.

Predict

Nothing is gated and no login is needed. Weights download on first use from LibreYOLO/LibreCRM at a pinned revision.

Python
from libreyolo import LibreGen, SAMPLE_IMAGE model = LibreGen("crm")             # weights download on first useresult = model(SAMPLE_IMAGE, seed=0) result.save("object.obj")           # vertex colours as `v x y z r g b`m = result.meshesprint(m.vertices[0].shape, m.extras["vertex_colors"][0].shape)
CLI
# Saves the object as .obj under runs/generate/predict.libreyolo generate source=https://raw.githubusercontent.com/LibreYOLO/libreyolo/release/libreyolo/assets/guggenheim-bilbao.jpg model=crm seed=0libreyolo generate source=https://raw.githubusercontent.com/LibreYOLO/libreyolo/release/libreyolo/assets/guggenheim-bilbao.jpg model=crm texture=true
Texture
# pip install xatlas   (MIT, for the UV unwrap)from libreyolo import LibreGen, SAMPLE_IMAGE model = LibreGen("crm")model(SAMPLE_IMAGE, texture=True).save("object.obj")# writes object.obj, object.mtl and object_texture.png

Two multi-view diffusion models generate six orthographic views of the object and then their canonical coordinate maps; a convolutional UNet turns them into a triplane, and FlexiCubes extracts a mesh coloured by the triplane. Upstream needs xformers, nvdiffrast (NVIDIA, non-commercial) and a GPL mesh library; LibreYOLO runs the same weights in plain PyTorch on NVIDIA GPUs, Apple silicon and CPU, in fp16 on the GPUs as upstream.

The input is padded to a square on grey, without a crop, so a photo works best with the object filling the frame. A seed gives the same starting noise on every device; on CUDA a repeat call gives the same views and triangles, with vertices that can move by about 1e-7. The library measured about 12 s and 12 GB per object on an RTX 3090.

result.meshes holds vertices (V, 3), z-up in [-0.5, 0.5]^3, faces (F, 3) and extras["vertex_colors"] (V, 3) RGB in [0, 1]. texture=True bakes a 1024 x 1024 UV texture, as upstream does.

Settings

SettingDefaultMeaning
steps50Sampling steps of both diffusion stages, 1 to 499.
guidance5.0Classifier-free guidance of both stages.
textureFalseBake a UV texture (.mtl and .png beside the .obj).

Variants

One checkpoint, 4.1B parameters: two 2.0B multi-view diffusion models (UNet, OpenCLIP ViT-H/14 and SD VAE each) and a 0.12B reconstruction UNet. About 12.8 GB, plus 0.4 GB for BiRefNet.

Export

TâcheONNXTorchScriptExecuTorchTensorRTOpenVINOPaddleMNNRKNNAxelerancnnTFLiteCoreMLCore AI
3D generation3D generation to ONNX : non pris en charge3D generation to TorchScript : non pris en charge3D generation to ExecuTorch : non pris en charge3D generation to TensorRT : non pris en charge3D generation to OpenVINO : non pris en charge3D generation to Paddle : non pris en charge3D generation to MNN : non pris en charge3D generation to RKNN : non pris en charge3D generation to Axelera : non pris en charge3D generation to ncnn : non pris en charge3D generation to TFLite : non pris en charge3D generation to CoreML : non pris en charge3D generation to Core AI : non pris en charge

Export is not implemented for LibreGen 3D families. export(), train() and val() raise NotImplementedError (ADR 0031 in the library).

Licensing

Vérifiez la licence dans le dépôt Hugging Face des poids précis que vous téléchargez. Chaque checkpoint de l<link>organisation LibreYOLO</link> en possède une, et elles ne sont pas toujours identiques au sein dune même famille. Ce dépôt fait autorité ; le résumé ci-dessous décrit les conditions applicables lors de la dernière vérification de cette page.

Ceci décrit les licences concernées et ne constitue pas un conseil juridique. Si la réponse a une importance commerciale, lisez vous-même les licences et consultez votre propre avocat.

Travail d'origine
CRM, Tsinghua University (TSAIL)
Licence du projet d'origine
CreativeML OpenRAIL++-M
Source du projet d'origine
github.com/thu-ml/CRM
Code de LibreYOLO
MIT
Poids
CreativeML OpenRAIL++-M, republiés sur huggingface.co/LibreYOLO
Interprétation
The CRM code is MIT and its model card declares MIT for the weights, but CRM's two diffusion models are fine-tuned from ImageDream and MVDream, themselves from Stable Diffusion 2.1, so LibreYOLO treats the weights as under the CreativeML OpenRAIL++-M License of that lineage. OpenRAIL++-M allows commercial use, but its use-based restrictions (no illegal, harmful, discriminatory or deceptive uses, among others) bind you and must be passed on to anyone you share the model or derivatives with. LibreYOLO's adapter is MIT; the code it adapts from CRM is MIT, the FlexiCubes meshing comes from NVIDIA's Apache-2.0 release, and ImageDream parts are Apache-2.0, noted in libreyolo/models/gen/crm/NOTICE. CRM's own FlexiCubes copies with NVIDIA's all-rights-reserved header, nvdiffrast (non-commercial) and the GPL-3.0 mesh tools upstream calls are not used. The Zhengyi/CRM weights are redistributed unchanged as LibreYOLO/LibreCRM, whose card carries the OpenRAIL++-M terms; Background removal runs on LibreYOLO's BiRefNet port with the ZhengPeng7/BiRefNet weights (MIT), from LibreYOLO/LibreTRELLISl. Both are pinned and not gated.

CRM is by Tsinghua University (TSAIL) (thu-ml/CRM). The code is MIT and the model card declares MIT for the weights.

The weights carry use restrictions. CRM's two diffusion models descend from ImageDream, MVDream and Stable Diffusion 2.1, so LibreYOLO treats them as under the CreativeML OpenRAIL++-M License of that lineage. Commercial use is allowed, but its use-based restrictions (no illegal, harmful, discriminatory or deceptive uses, among others) bind you and must be passed on to anyone you share the model or derivatives with.

The Zhengyi/CRM weights are redistributed unchanged as LibreYOLO/LibreCRM, whose card carries the OpenRAIL++-M terms. FlexiCubes comes from NVIDIA's Apache-2.0 release; nvdiffrast and the GPL-3.0 mesh tools upstream calls are not used.

Citation

@article{wang2024crm,
  title={CRM: Single Image to 3D Textured Mesh with Convolutional Reconstruction Model},
  author={Zhengyi Wang and Yikai Wang and Yifei Chen and Chendong Xiang and Shuo Chen and Dajiang Yu and Chongxuan Li and Hang Su and Jun Zhu},
  journal={arXiv preprint arXiv:2403.05034},
  year={2024}
}

Copié depuis le bloc de citation des auteurs sur github.com/thu-ml/CRM#citation.

Vérifié avec LibreYOLO v1.8.0. Les tableaux de support, les checkpoints et les chiffres de benchmark de cette page sont générés à partir de la bibliothèque publiée et des poids publiés, et non rédigés à la main.