LGM
LGM (Large Multi-View Gaussian Model) turns one picture of an object into 3D Gaussians in two stages: ImageDream samples four views around the object, then LGM reads them and returns the Gaussians. LibreYOLO runs both stages in plain PyTorch through the LibreGen factory.
- 작업
- 3d generation
- 설치
pip install "libreyolo[gen]"- 지원 티어
- 별도 티어, v1.8.0부터 지원. 자체 팩토리와 계약을 갖춘 별도의 제품 영역입니다.
- 라이선스
- 코드 MIT and Apache-2.0, 가중치 MIT and CreativeML OpenRAIL++-M. 상업적 사용
Install
pip install "libreyolo[gen]"The extra brings diffusers and transformers, which load ImageDream's VAE,
scheduler and CLIP encoders.
Predict
Nothing is gated and no login is needed. Weights download on first use at pinned revisions.
from libreyolo import LibreGen, SAMPLE_IMAGE model = LibreGen("lgm") # weights download on first useresult = model(SAMPLE_IMAGE, seed=0) result.save("object.ply") # standard 3DGS layoutprint(len(result.gaussians)) # those above opacity 0.005# Saves the object as .ply under runs/generate/predict.libreyolo generate source=https://raw.githubusercontent.com/LibreYOLO/libreyolo/release/libreyolo/assets/guggenheim-bilbao.jpg model=lgm seed=0from libreyolo import LibreGen, SAMPLE_IMAGE model = LibreGen("lgm")result = model(SAMPLE_IMAGE, elevation=-10) # picture taken from aboveresult = model(SAMPLE_IMAGE, steps=20, guidance=1.0) # no guidanceImageDream, a multi-view diffusion model fine-tuned from Stable Diffusion 2.1, samples four views around the object from the 256 px input. LGM, an asymmetric UNet with cross-view attention, reads the four views with their camera rays and returns one Gaussian per output pixel of each view: 4 x 128 x 128 = 65,536 Gaussians, of which those above opacity 0.005 are kept, as upstream saves them.
Upstream needs xformers on CUDA and renders with a non-commercial rasterizer. LibreYOLO runs attention through PyTorch's SDPA and does not render, so the model runs on NVIDIA GPUs, Apple silicon and CPU, in fp16 on GPUs and fp32 on CPU. On CUDA a seed reproduces upstream's object; other devices draw other noise for the same seed.
The output is result.gaussians, the same payload as
Speridlabs Idol, saved as .ply or
.splat.
Settings
| Setting | Default | Meaning |
|---|---|---|
steps | 30 | ImageDream's DDIM sampling steps. |
guidance | 5.0 | Classifier-free guidance; 1 turns it off. |
elevation | 0 | Camera height of the picture in degrees; negative is above the object. |
Variants
One size, 0.4b: the 0.42B LGM network (about 0.8 GB) plus the 1.3B
ImageDream multi-view stage and its two CLIP encoders (about 4.0 GB).
Export
| 작업 | ONNX | TorchScript | ExecuTorch | TensorRT | OpenVINO | Paddle | MNN | RKNN | Axelera | ncnn | TFLite | CoreML | Core AI |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 3D generation | 3D generation to ONNX: 지원하지 않음 | 3D generation to TorchScript: 지원하지 않음 | 3D generation to ExecuTorch: 지원하지 않음 | 3D generation to TensorRT: 지원하지 않음 | 3D generation to OpenVINO: 지원하지 않음 | 3D generation to Paddle: 지원하지 않음 | 3D generation to MNN: 지원하지 않음 | 3D generation to RKNN: 지원하지 않음 | 3D generation to Axelera: 지원하지 않음 | 3D generation to ncnn: 지원하지 않음 | 3D generation to TFLite: 지원하지 않음 | 3D generation to CoreML: 지원하지 않음 | 3D generation to Core AI: 지원하지 않음 |
Export is not implemented for LibreGen 3D families. export(), train() and
val() raise NotImplementedError (ADR 0031 in the library).
Licensing
내려받는 특정 가중치의 Hugging Face 저장소에서 라이선스를 확인하십시오. LibreYOLO 조직의 모든 체크포인트에는 라이선스가 있으며 한 계열 안에서도 항상 같지는 않습니다. 해당 저장소가 신뢰할 수 있는 기준입니다. 아래 요약은 이 페이지를 마지막으로 검증했을 때 적용된 내용을 설명합니다.
관련 라이선스에 관한 설명이며 법률 자문이 아닙니다. 상업적으로 중요한 사안이라면 라이선스를 직접 읽고 별도의 법률 자문을 받으십시오.
- 원작
- LGM, 3D Topia
- 업스트림 라이선스
- MIT and CreativeML OpenRAIL++-M
- 업스트림 소스
- github.com/3DTopia/LGM
- LibreYOLO 코드
- MIT
- 가중치
- MIT and CreativeML OpenRAIL++-M, huggingface.co/LibreYOLO에 다시 게시됨
- 해석
- LGM runs in two stages with different licenses. The LGM Gaussian model and its code are MIT, a permissive license that allows commercial and closed-source use with the copyright notice kept. The first stage, the ImageDream multi-view diffusion model (fine-tuned from MVDream and Stable Diffusion 2.1), is declared openrail upstream and LibreYOLO carries the CreativeML OpenRAIL++-M License of its Stable Diffusion 2.1 lineage: commercial use is allowed, but its use-based restrictions (no illegal, harmful, discriminatory or deceptive uses, among others) bind you and must be passed on to anyone you share the model or derivatives with. LibreYOLO's adapter is MIT; the code it adapts from LGM is MIT, with parts following ImageDream, the DINOv2 attention module and kiuikit under Apache-2.0, noted in libreyolo/models/gen/lgm/NOTICE. LibreYOLO leaves out upstream's renderer, which needs Inria's non-commercial Gaussian Splatting rasterizer. Weights are redistributed unchanged at pinned revisions: ashawkey/LGM with the BiRefNet weights (MIT) as LibreYOLO/LibreLGM0.4b, and ashawkey/imagedream-ipmv-diffusers as LibreYOLO/LibreImageDream; neither is gated.
LGM is by Jiaxiang Tang et al. at 3D Topia (3DTopia/LGM). The two stages carry different licences:
- The LGM Gaussian model and its code are MIT.
- ImageDream carries use restrictions. It is declared OpenRAIL upstream, and LibreYOLO applies the CreativeML OpenRAIL++-M License of its Stable Diffusion 2.1 lineage: commercial use is allowed, but its use-based restrictions (no illegal, harmful, discriminatory or deceptive uses, among others) bind you and must be passed on with the model or derivatives.
Weights are redistributed unchanged at pinned revisions: ashawkey/LGM with
BiRefNet (MIT) as LibreYOLO/LibreLGM0.4b, and
ashawkey/imagedream-ipmv-diffusers as
LibreYOLO/LibreImageDream.
Neither is gated. Upstream's renderer, which needs Inria's non-commercial
rasterizer, is left out.
Citation
@article{tang2024lgm,
title={LGM: Large Multi-View Gaussian Model for High-Resolution 3D Content Creation},
author={Tang, Jiaxiang and Chen, Zhaoxi and Chen, Xiaokang and Wang, Tengfei and Zeng, Gang and Liu, Ziwei},
journal={arXiv preprint arXiv:2402.05054},
year={2024}
}github.com/3DTopia/LGM#citation에 있는 저자의 인용 블록에서 복사했습니다.