[AAAI'26] StreamSTGS: Streaming spatial and temporal gaussian grids for real-time free-viewpoint video
Zhihui Ke, Yuyang Liu, Xiaobo Zhou, Tie Qiu
StreamSTGS is a novel FVV representation designed for real-time streaming, which represents a dynamic scene using canonical 3D Gaussians, temporal features, and a deformation field. For high compression efficiency, StreamSTGS encodes canonical Gaussian attributes as 2D images and temporal features as a video. This design not only enables real-time streaming, but also naturally supports adaptive bitrate control based on network condition without any extra training. Moreover, we propose a sliding window scheme to aggregate adjacent temporal features to learn local motions, and then introduce a transformer-guided auxiliary training module to learn global motions. On diverse FVV benchmarks, StreamSTGS demonstrates competitive performance on all metrics compared to state-of-the-art methods. Notably, StreamSTGS increases the PSNR by an average of 1dB while reducing the average frame size to just 170KB.
We tested our code on a server with Ubuntu 20.04.6, cuda 11.6, pytorch 1.13.1
- Unzip files
cd thirdparty/gaussian_splatting/submodules
unzip diff-gaussian-mcmc-rasterization.zip
unzip diff-gaussian-rasterization.zip
unzip fast-gaussian-rasterization.zip
unzip simple-knn.zip
unzip fused-ssim.zip
unzip PLAS.zip # please attention!!! pytorch 2.0+ cannot install PLAS- Install conda environment
sudo apt install ffmpeg
conda create -n streamstgs python=3.10
conda activate streamstgs
pip install torch==1.13.1+cu116 torchvision==0.14.1+cu116 torchaudio==0.13.1 --extra-index-url https://download.pytorch.org/whl/cu116
pip uninstall numpy
pip install numpy==1.26.4
pip install -r requirements.txt
pip install thirdparty/gaussian_splatting/submodules/diff-gaussian-mcmc-rasterization
pip install thirdparty/gaussian_splatting/submodules/simple-knn
pip install thirdparty/gaussian_splatting/submodules/diff-gaussian-rasterization
pip install thirdparty/gaussian_splatting/submodules/fast-gaussian-rasterization
pip install thirdparty/gaussian_splatting/submodules/fused-ssim
pip install thirdparty/gaussian_splatting/submodules/PLAS- Download the multi-view video datasets:
N3DV: https://github.com/facebookresearch/Neural_3D_Video
MeetRoom: https://drive.google.com/drive/folders/1lNmQ6_ykyKjT6UKy-SnqWoSlI5yjh3l_?usp=share_link
- preprocess dataset
We follow 4DGS to preprocess multi-view dynamic scene datasets
python script/preprocess_dynerf.py --datadir /root/dataset/N3DV/cut_roasted_beef --type n3dv
python runcolmap.py --root_dir /root/dataset/N3DV/cut_roasted_beef --start 0 --end 1 --type n3dv
python script/calc_std.py --path /root/dataset/N3DV/cut_roasted_beef --type n3dv# train all scenes
sh train_n3dv.sh # You need to modify output_path and dataset_path
# or train one scene
python train_offset.py -s {dataset_path} --configs arguments/n3dv/{default.py} --compre_config gop60.yaml --checkpoint_path {output_path}/coarse --expname {output_path} --gop 60 --coarse
python train_offset.py -s {dataset_path} --configs arguments/n3dv/{default.py} --compre_config gop60.yaml --checkpoint_path {output_path}/coarse --expname {output_path} --compres_thre 28 --gopids 0 1 2 3 --gop 60 --postfix defaultpython test_offset.py -s /root/dataset/N3DV/cut_roasted_beef --configs arguments/n3dv/default.py --compre_config gop60.yaml --checkpoint_path {ckpt path} --all --qp 20If you have any questions, please feel free to contact me via kezhihui@tju.edu.cn.
If you find our work helpful, please consider citing:
@inproceedings{ke2025edgegaussian,
title={StreamSTGS: Streaming spatial and temporal gaussian grids for real-time free-viewpoint video},
author={Zhihui Ke, Yuyang Liu, Xiaobo Zhou, Tie Qiu},
booktitle={Proceedings of the AAAI Conference on Artificial Intelligence},
year={2026}
}- Please follow the LICENSE of 3DGS.
- We thank all authors from 3DGS, SOG, and TimeFormer for presenting such an excellent work.