LGM

LGM (Large Multi-View Gaussian Model) turns one picture of an object into 3D Gaussians in two stages: ImageDream samples four views around the object, then LGM reads them and returns the Gaussians. LibreYOLO runs both stages in plain PyTorch through the LibreGen factory.

Zadania
3d generation
Instalacja
pip install "libreyolo[gen]"
Poziom obsługi
Poziom siostrzany, od wersji v1.8.0. Osobna część produktu z własną fabryką i kontraktem.
Projekt źródłowy
LGM, autorzy: 3D Topia, licencja: MIT and CreativeML OpenRAIL++-M. Publikacja, kod źródłowy
Licencje
Kod: MIT and Apache-2.0, wagi: MIT and CreativeML OpenRAIL++-M. Użycie komercyjne

Install

bash
pip install "libreyolo[gen]"

The extra brings diffusers and transformers, which load ImageDream's VAE, scheduler and CLIP encoders.

Predict

Nothing is gated and no login is needed. Weights download on first use at pinned revisions.

Python
from libreyolo import LibreGen, SAMPLE_IMAGE model = LibreGen("lgm")             # weights download on first useresult = model(SAMPLE_IMAGE, seed=0) result.save("object.ply")           # standard 3DGS layoutprint(len(result.gaussians))        # those above opacity 0.005
CLI
# Saves the object as .ply under runs/generate/predict.libreyolo generate source=https://raw.githubusercontent.com/LibreYOLO/libreyolo/release/libreyolo/assets/guggenheim-bilbao.jpg model=lgm seed=0
Settings
from libreyolo import LibreGen, SAMPLE_IMAGE model = LibreGen("lgm")result = model(SAMPLE_IMAGE, elevation=-10)     # picture taken from aboveresult = model(SAMPLE_IMAGE, steps=20, guidance=1.0)   # no guidance

ImageDream, a multi-view diffusion model fine-tuned from Stable Diffusion 2.1, samples four views around the object from the 256 px input. LGM, an asymmetric UNet with cross-view attention, reads the four views with their camera rays and returns one Gaussian per output pixel of each view: 4 x 128 x 128 = 65,536 Gaussians, of which those above opacity 0.005 are kept, as upstream saves them.

Upstream needs xformers on CUDA and renders with a non-commercial rasterizer. LibreYOLO runs attention through PyTorch's SDPA and does not render, so the model runs on NVIDIA GPUs, Apple silicon and CPU, in fp16 on GPUs and fp32 on CPU. On CUDA a seed reproduces upstream's object; other devices draw other noise for the same seed.

The output is result.gaussians, the same payload as Speridlabs Idol, saved as .ply or .splat.

Settings

SettingDefaultMeaning
steps30ImageDream's DDIM sampling steps.
guidance5.0Classifier-free guidance; 1 turns it off.
elevation0Camera height of the picture in degrees; negative is above the object.

Variants

One size, 0.4b: the 0.42B LGM network (about 0.8 GB) plus the 1.3B ImageDream multi-view stage and its two CLIP encoders (about 4.0 GB).

Export

ZadanieONNXTorchScriptExecuTorchTensorRTOpenVINOPaddleMNNRKNNAxelerancnnTFLiteCoreMLCore AI
3D generation3D generation to ONNX: brak obsługi3D generation to TorchScript: brak obsługi3D generation to ExecuTorch: brak obsługi3D generation to TensorRT: brak obsługi3D generation to OpenVINO: brak obsługi3D generation to Paddle: brak obsługi3D generation to MNN: brak obsługi3D generation to RKNN: brak obsługi3D generation to Axelera: brak obsługi3D generation to ncnn: brak obsługi3D generation to TFLite: brak obsługi3D generation to CoreML: brak obsługi3D generation to Core AI: brak obsługi

Export is not implemented for LibreGen 3D families. export(), train() and val() raise NotImplementedError (ADR 0031 in the library).

Licensing

Sprawdź licencję w repozytorium konkretnych pobieranych wag na Hugging Face. Każdy checkpoint w organizacji LibreYOLO ma licencję, która nie zawsze jest taka sama w całej rodzinie. To repozytorium jest źródłem rozstrzygającym, a poniższe podsumowanie opisuje stan z chwili ostatniej weryfikacji tej strony.

To opis obowiązujących licencji, a nie porada prawna. Jeśli odpowiedź ma znaczenie komercyjne, przeczytaj licencje i zasięgnij własnej porady prawnej.

Oryginalna praca
LGM, 3D Topia
Licencja projektu źródłowego
MIT and CreativeML OpenRAIL++-M
Kod źródłowy projektu
github.com/3DTopia/LGM
Kod LibreYOLO
MIT
Wagi
MIT and CreativeML OpenRAIL++-M, ponownie opublikowane w huggingface.co/LibreYOLO
Interpretacja
LGM runs in two stages with different licenses. The LGM Gaussian model and its code are MIT, a permissive license that allows commercial and closed-source use with the copyright notice kept. The first stage, the ImageDream multi-view diffusion model (fine-tuned from MVDream and Stable Diffusion 2.1), is declared openrail upstream and LibreYOLO carries the CreativeML OpenRAIL++-M License of its Stable Diffusion 2.1 lineage: commercial use is allowed, but its use-based restrictions (no illegal, harmful, discriminatory or deceptive uses, among others) bind you and must be passed on to anyone you share the model or derivatives with. LibreYOLO's adapter is MIT; the code it adapts from LGM is MIT, with parts following ImageDream, the DINOv2 attention module and kiuikit under Apache-2.0, noted in libreyolo/models/gen/lgm/NOTICE. LibreYOLO leaves out upstream's renderer, which needs Inria's non-commercial Gaussian Splatting rasterizer. Weights are redistributed unchanged at pinned revisions: ashawkey/LGM with the BiRefNet weights (MIT) as LibreYOLO/LibreLGM0.4b, and ashawkey/imagedream-ipmv-diffusers as LibreYOLO/LibreImageDream; neither is gated.

LGM is by Jiaxiang Tang et al. at 3D Topia (3DTopia/LGM). The two stages carry different licences:

  • The LGM Gaussian model and its code are MIT.
  • ImageDream carries use restrictions. It is declared OpenRAIL upstream, and LibreYOLO applies the CreativeML OpenRAIL++-M License of its Stable Diffusion 2.1 lineage: commercial use is allowed, but its use-based restrictions (no illegal, harmful, discriminatory or deceptive uses, among others) bind you and must be passed on with the model or derivatives.

Weights are redistributed unchanged at pinned revisions: ashawkey/LGM with BiRefNet (MIT) as LibreYOLO/LibreLGM0.4b, and ashawkey/imagedream-ipmv-diffusers as LibreYOLO/LibreImageDream. Neither is gated. Upstream's renderer, which needs Inria's non-commercial rasterizer, is left out.

Citation

@article{tang2024lgm,
  title={LGM: Large Multi-View Gaussian Model for High-Resolution 3D Content Creation},
  author={Tang, Jiaxiang and Chen, Zhaoxi and Chen, Xiaokang and Wang, Tengfei and Zeng, Gang and Liu, Ziwei},
  journal={arXiv preprint arXiv:2402.05054},
  year={2024}
}

Skopiowano z bloku cytowania autorów w github.com/3DTopia/LGM#citation.

Zweryfikowano z LibreYOLO v1.8.0. Tabele obsługi, checkpointy i wyniki benchmarków na tej stronie są generowane na podstawie wydanej biblioteki i opublikowanych wag, a nie wpisywane ręcznie.