Diffusers 远程推理Remote Inference实战用 Inference Endpoints 把 VAE 编码/解码卸载到云端【免费下载链接】diffusers Diffusers: State-of-the-art diffusion models for image, video, and audio generation in PyTorch.项目地址: https://gitcode.com/GitHub_Trending/di/diffusers远程推理Remote Inference是 Diffusers 提供的实验性能力它把大模型推理中最吃显存的 VAE 编码与解码环节卸载到 Hugging Face Inference Endpoints 上执行从而显著放松本地推理的内存要求。本文将以 overview.md 为主线结合 remote_utils.py、constants.py 与 tests/remote 下的测试源码完整讲解如何用remote_encode/remote_decode在 Stable Diffusion v1、SDXL、Flux 与 HunyuanVideo 上实现远程编码、远程解码与多请求排队并附上官方基准数据供选型参考。远程推理是什么把“最重的两步”搬到云端在本地跑 Stable Diffusion / Flux / HunyuanVideo 这类模型时显存压力通常来自三个部分文本编码器、去噪主干UNet/Transformer、以及 VAE。其中 VAE 的编码图像/视频 → 潜变量与解码潜变量 → 图像/视频在超高分辨率如 2048×2048下会瞬间拉满显存甚至直接 OOM。传统解法是模型 offload 或分块tiled编码/解码但这两种手段都会增加推理时间分块解码还会影响输出质量。远程推理的思路很直接让 VAE 编码/解码跑在远端 Inference Endpoint 上本地只保留文本编码器和去噪主干。通过 remote_utils.py 中提供的remote_encode与remote_decode两个函数本地把图像或潜变量张量序列化后 POST 到远端端点端点完成编码/解码后把结果传回本地显存占用因此只取决于去噪主干本身。[!NOTE] 该功能当前标记为实验特性experimental feature官方正在收集反馈使用前请确认所用端点仍处于部署状态。支持的模型与端点官方为以下四类模型部署了公开的远程端点支持能力各不相同HunyuanVideo 目前只支持 decode模型端点Endpoint关联检查点支持Stable Diffusion v1https://q1bj3bpq6kzilnsu.us-east-1.aws.endpoints.huggingface.cloudstabilityai/sd-vae-ft-mseencode/decodeStable Diffusion XLhttps://x2dmsqunjd6k9prw.us-east-1.aws.endpoints.huggingface.cloudmadebyollin/sdxl-vae-fp16-fixencode/decodeFluxhttps://whhx50ex1aryqvw6.us-east-1.aws.endpoints.huggingface.cloudblack-forest-labs/FLUX.1-schnellencode/decodeHunyuanVideohttps://o7ywnmrahorts457.us-east-1.aws.endpoints.huggingface.cloudhunyuanvideo-community/HunyuanVideodecode这些端点地址同时也被固化在仓库常量中便于测试与二次开发时直接引用DECODE_ENDPOINT_SD_V1/DECODE_ENDPOINT_SD_XL/DECODE_ENDPOINT_FLUX/DECODE_ENDPOINT_HUNYUAN_VIDEOENCODE_ENDPOINT_SD_V1/ENCODE_ENDPOINT_SD_XL/ENCODE_ENDPOINT_FLUX见 constants.py。其中 Flux 的 encode/decode 端点与文档示例共用同一地址而ENCODE_ENDPOINT_*系列常量提供了独立的编码端点地址实际接入时可按需选用。远程编码图像/视频 → 潜变量编码把图像或视频转换成潜变量表示。把一张 PIL 图像传给remote_encode即可模型对应的scaling_factor与shift_factor取值见后文“缩放因子速查表”。以 Flux 为例完整的编码流程如下先用FluxPipeline.from_pretrained加载除 VAE 之外的全部组件vaeNone再调用remote_encode把输入图编码成潜变量import torch from diffusers import FluxPipeline from diffusers.utils import load_image from diffusers.utils.remote_utils import remote_encode pipeline FluxPipeline.from_pretrained( black-forest-labs/FLUX.1-schnell, dtypetorch.float16, vaeNone, device_mapcuda # or mps, xpu, cpu ) init_image load_image( https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/diffusers/astronaut.jpg ) init_image init_image.resize((768, 512)) init_latent remote_encode( endpointhttps://whhx50ex1aryqvw6.us-east-1.aws.endpoints.huggingface.cloud, imageinit_image, scaling_factor0.3611, shift_factor0.1159 )从实现上看remote_utils.py 中remote_encode的核心逻辑是prepare_encode把PIL.Image序列化为 PNG 字节流若传入torch.Tensor则走 safetensors 二进制序列化随后requests.post(endpoint, ...)上传远端完成编码后postprocess_encode依据响应头中的shape与dtype用torch.frombuffer还原出torch.Tensor。因此返回值就是可直接喂给去噪循环的潜变量张量。在 test_remote_encode.py 中可以看到编码张量的形状约定[1, channels, height // 8, width // 8]其中 SD v1/SDXL 为 4 通道Flux 为 16 通道测试还覆盖了从 320×320 到 2048×2048 的多分辨率编码-解码往返。需要注意该测试文件中的ENCODE_ENDPOINT_*用例当前以xfail标记注释说明这些编码端点曾返回404 NOT_FOUND端点被下线因此接入生产环境前务必自行验证端点可用性。远程解码潜变量 → 图像/视频解码把潜变量还原成图像或视频。做法是pipeline 的output_typelatent、vaeNone把得到的潜变量交给remote_decode。对于 Flux由于潜变量经过 packed 处理形状为[1, 4096, 64]而非[1, 16, 128, 128]还必须额外传入height与width——这与 check_inputs_decode 中对三维张量强制要求height/width的校验逻辑一致。Flux 图像解码from diffusers import FluxPipeline pipeline FluxPipeline.from_pretrained( black-forest-labs/FLUX.1-schnell, dtypetorch.bfloat16, vaeNone, device_mapcuda # or mps, xpu, cpu ) prompt A photorealistic Apollo-era photograph of a cat in a small astronaut suit with a bubble helmet, standing on the Moon and holding a flagpole planted in the dusty lunar soil. The flag shows a colorful paw-print emblem. Earth glows in the black sky above the stark gray surface, with sharp shadows and high-contrast lighting like vintage NASA photos. latent pipeline( promptprompt, guidance_scale0.0, num_inference_steps4, output_typelatent, ).images image remote_decode( endpointhttps://whhx50ex1aryqvw6.us-east-1.aws.endpoints.huggingface.cloud/, tensorlatent, height1024, width1024, scaling_factor0.3611, shift_factor0.1159, ) image.save(image.jpg)HunyuanVideo 视频解码视频模型的解码类似只是pipeline(...).frames拿到的是视频潜变量remote_decode以output_typemp4直接返回 MP4 字节流import torch from diffusers import HunyuanVideoPipeline, HunyuanVideoTransformer3DModel transformer HunyuanVideoTransformer3DModel.from_pretrained( hunyuanvideo-community/HunyuanVideo, subfoldertransformer, dtypetorch.bfloat16 ) pipeline HunyuanVideoPipeline.from_pretrained( model_id, transformertransformer, vaeNone, dtypetorch.float16, device_mapcuda # or mps, xpu, cpu ) latent pipeline( promptA cat walks on the grass, realistic, height320, width512, num_frames61, num_inference_steps30, output_typelatent, ).frames video remote_decode( endpointhttps://o7ywnmrahorts457.us-east-1.aws.endpoints.huggingface.cloud/, tensorlatent, output_typemp4, ) if isinstance(video, bytes): with open(video.mp4, wb) as f: f.write(video)remote_decode 的输出控制参数结合 remote_utils.py 的函数签名与 docstringremote_decode除endpoint/tensor外还有一组影响传输量与返回形态的关键参数scaling_factor/shift_factor解码时在远端对潜变量做缩放/平移等价于本地latents / vae.config.scaling_factor与latents vae.config.shift_factor。SD v1 为0.18215SDXL 为0.13025Flux 为0.3611/0.1159不传则要求输入已自带缩放。output_type端点输出类型pil、pt或mp4视频模型。ptpartial_postprocessTrue是“全质量下最小传输量”的推荐组合ptpartial_postprocessFalse与第三方代码兼容性最好pilimage_formatjpg整体传输量最小。return_type函数返回类型pil返回PIL.Image.Imageoutput_typept时会做一次后处理转换pt返回torch.Tensormp4返回bytes。partial_postprocessoutput_typept时False返回未反归一化的float16/bfloat16张量True返回已反归一化的uint8图像张量。image_formatjpg或png仅配合output_typepil使用。processorVaeImageProcessor或VideoProcessor实例在output_typept且return_typepil、未开partial_postprocess时必填check_inputs_decode 会强制校验。height/widthpacked 潜变量如 Flux 的三维张量必填。这些行为在 test_remote_decode.py 中都有逐一验证test_output_type_pt、test_output_type_pil、test_output_type_pil_image_format、test_output_type_pt_partial_postprocess、test_output_type_pt_return_type_pt以及 HunyuanVideo 的test_output_type_mp4同时用参考切片reference slice校验了输出张量的数值稳定性并覆盖 320 至 2048 的多分辨率场景。缩放因子速查表各模型的scaling_factor/shift_factor与潜变量形状整理如下数值来源remote_utils.py 的 docstring 与 test_remote_decode.py 中各测试类的类属性模型scaling_factorshift_factor潜变量通道说明Stable Diffusion v10.18215无4[1, 4, H/8, W/8]Stable Diffusion XL0.13025无4[1, 4, H/8, W/8]Flux0.36110.115916packed 后为[1, 4096, 64]解码需传height/widthHunyuanVideo0.476986无165D 视频潜变量[1, 16, T, H/8, W/8]详细的函数级 API 说明含[[autodoc]]自动生成的签名见 api_reference.md。排队解码流水线式处理多个生成请求远程推理天然适合“生成与解码并行”当前潜变量正在远端解码时本地可以继续生成下一个潜变量从而隐藏解码延迟。官方示例用queue.Queue 守护线程实现了解码工作线程并对 SDXL 启用了torch.compile加速去噪主干import queue import threading from IPython.display import display from diffusers import StableDiffusionXLPipeline def decode_worker(q: queue.Queue): while True: item q.get() if item is None: break image remote_decode( endpointhttps://q1bj3bpq6kzilnsu.us-east-1.aws.endpoints.huggingface.cloud/, tensoritem, scaling_factor0.13025, ) display(image) q.task_done() q queue.Queue() thread threading.Thread(targetdecode_worker, args(q,), daemonTrue) thread.start() def decode(latent: torch.Tensor): q.put(latent) prompts [ A grainy Apollo-era style photograph of a cat in a snug astronaut suit with a bubble helmet, standing on the lunar surface and gripping a flag with a paw-print emblem. The gray Moon landscape stretches behind it, Earth glowing vividly in the black sky, shadows crisp and high-contrast., A vintage 1960s sci-fi pulp magazine cover illustration of a heroic cat astronaut planting a flag on the Moon. Bold, saturated colors, exaggerated space gear, playful typography floating in the background, Earth painted in bright blues and greens., A hyper-detailed cinematic shot of a cat astronaut on the Moon holding a fluttering flag, fur visible through the helmet glass, lunar dust scattering under its feet. The vastness of space and Earth in the distance create an epic, awe-inspiring tone., A colorful cartoon drawing of a happy cat wearing a chunky, oversized spacesuit, proudly holding a flag with a big paw print on it. The Moon’s surface is simplified with craters drawn like doodles, and Earth in the sky has a smiling face., A monochrome 1969-style press photo of a “first cat on the Moon” moment. The cat, in a tiny astronaut suit, stands by a planted flag, with grainy textures, scratches, and a blurred Earth in the background, mimicking old archival space photos. ] pipeline StableDiffusionXLPipeline.from_pretrained( https://huggingface.co/stabilityai/stable-diffusion-xl-base-1.0, dtypetorch.float16, vaeNone, device_mapcuda # or mps, xpu, cpu ) pipeline.unet pipeline.unet.to(memory_formattorch.channels_last) pipeline.unet torch.compile(pipe.unet, modereduce-overhead, fullgraphTrue) _ pipeline( promptprompts[0], output_typelatent, ) for prompt in prompts: latent pipeline( promptprompt, output_typelatent, ).images decode(latent) q.put(None) thread.join()要点拆解decode_worker线程从队列取潜变量并调用remote_decode取到哨兵值None后退出主线程依次生成 5 个潜变量并q.put提交无需等待上一次解码完成最后q.put(None)通知工作线程结束thread.join()等待全部解码完成pipeline.unet.to(memory_formattorch.channels_last)与torch.compile进一步压榨本地去噪主干的吞吐让“生成速度”跟得上“远程解码速度”。基准测试远程编码/解码的显存与耗时下表展示了官方在多种 NVIDIA GPU 上对 Stable Diffusion v1.5 与 SDXL 进行远程编码/解码的实测数据时间单位秒内存为该 GPU 显存占用百分比。对于大多数显卡而言显存占用直接决定了文本编码器、UNet/Transformer 是否需要 offload以及是否需要分块tiled编码——后两种手段都会增加推理时间并影响质量而远程推理可以规避这些取舍。Encoding - Stable Diffusion v1.5GPUResolutionTime (seconds)Memory (%)Tiled Time (secs)Tiled Memory (%)NVIDIA GeForce RTX 4090512x5120.0153.519010.0153.51901NVIDIA GeForce RTX 4090256x2560.0041.31540.0051.3154NVIDIA GeForce RTX 40902048x20480.40247.18520.4963.51901NVIDIA GeForce RTX 40901024x10240.07812.26580.0943.51901NVIDIA GeForce RTX 4080 SUPER512x5120.0235.301050.0235.30105NVIDIA GeForce RTX 4080 SUPER256x2560.0061.981520.0061.98152NVIDIA GeForce RTX 4080 SUPER2048x20480.57471.080.6565.30105NVIDIA GeForce RTX 4080 SUPER1024x10240.11118.47720.145.30105NVIDIA GeForce RTX 3090512x5120.0323.527820.0323.52782NVIDIA GeForce RTX 3090256x2560.011.318690.0091.31869NVIDIA GeForce RTX 30902048x20480.74247.30330.9543.52782NVIDIA GeForce RTX 30901024x10240.13612.29650.2073.52782NVIDIA GeForce RTX 3080512x5120.0368.517610.0368.51761NVIDIA GeForce RTX 3080256x2560.013.183870.013.18387NVIDIA GeForce RTX 30802048x20480.86386.74241.1918.51761NVIDIA GeForce RTX 30801024x10240.15729.68880.2278.51761NVIDIA GeForce RTX 3070512x5120.05110.69410.05110.6941NVIDIA GeForce RTX 3070256x2560.0153.997430.0153.99743NVIDIA GeForce RTX 30702048x20481.21796.0541.48210.6941NVIDIA GeForce RTX 30701024x10240.22337.27510.32710.6941Encoding SDXLGPUResolutionTime (seconds)Memory Consumed (%)Tiled Time (seconds)Tiled Memory (%)NVIDIA GeForce RTX 4090512x5120.0294.957070.0294.95707NVIDIA GeForce RTX 4090256x2560.0072.296660.0072.29666NVIDIA GeForce RTX 40902048x20480.87366.34520.86315.5649NVIDIA GeForce RTX 40901024x10240.14215.54790.14315.5479NVIDIA GeForce RTX 4080 SUPER512x5120.0447.467350.0447.46735NVIDIA GeForce RTX 4080 SUPER256x2560.013.45970.013.4597NVIDIA GeForce RTX 4080 SUPER2048x20481.31787.16151.29123.447NVIDIA GeForce RTX 4080 SUPER1024x10240.21323.42150.21423.4215NVIDIA GeForce RTX 3090512x5120.0585.656380.0585.65638NVIDIA GeForce RTX 3090256x2560.0162.450810.0162.45081NVIDIA GeForce RTX 30902048x20481.75577.82391.61418.4193NVIDIA GeForce RTX 30901024x10240.26518.40230.26518.4023NVIDIA GeForce RTX 3080512x5120.06413.65680.06413.6568NVIDIA GeForce RTX 3080256x2560.0185.917280.0185.91728NVIDIA GeForce RTX 30802048x2048OOMOOM1.86644.4717NVIDIA GeForce RTX 30801024x10240.30244.43080.30244.4308NVIDIA GeForce RTX 3070512x5120.09317.14650.09317.1465NVIDIA GeForce RTX 3070256x2560.0257.429310.0267.42931NVIDIA GeForce RTX 30702048x2048OOMOOM2.67455.8355NVIDIA GeForce RTX 30701024x10240.44355.78410.44355.7841Decoding - Stable Diffusion v1.5GPUResolutionTime (seconds)Memory (%)Tiled Time (secs)Tiled Memory (%)NVIDIA GeForce RTX 4090512x5120.0315.60%0.031 (0%)5.60%NVIDIA GeForce RTX 40901024x10240.14820.00%0.301 (103%)5.60%NVIDIA GeForce RTX 4080512x5120.058.40%0.050 (0%)8.40%NVIDIA GeForce RTX 40801024x10240.22430.00%0.356 (59%)8.40%NVIDIA GeForce RTX 4070 Ti512x5120.06611.30%0.066 (0%)11.30%NVIDIA GeForce RTX 4070 Ti1024x10240.28440.50%0.454 (60%)11.40%NVIDIA GeForce RTX 3090512x5120.0625.20%0.062 (0%)5.20%NVIDIA GeForce RTX 30901024x10240.25318.50%0.464 (83%)5.20%NVIDIA GeForce RTX 3080512x5120.0712.80%0.070 (0%)12.80%NVIDIA GeForce RTX 30801024x10240.28645.30%0.466 (63%)12.90%NVIDIA GeForce RTX 3070512x5120.10215.90%0.102 (0%)15.90%NVIDIA GeForce RTX 30701024x10240.42156.30%0.746 (77%)16.00%Decoding SDXLGPUResolutionTime (seconds)Memory Consumed (%)Tiled Time (seconds)Tiled Memory (%)NVIDIA GeForce RTX 4090512x5120.05710.00%0.057 (0%)10.00%NVIDIA GeForce RTX 40901024x10240.25635.50%0.257 (0.4%)35.50%NVIDIA GeForce RTX 4080512x5120.09215.00%0.092 (0%)15.00%NVIDIA GeForce RTX 40801024x10240.40653.30%0.406 (0%)53.30%NVIDIA GeForce RTX 4070 Ti512x5120.12120.20%0.120 (-0.8%)20.20%NVIDIA GeForce RTX 4070 Ti1024x10240.51972.00%0.519 (0%)72.00%NVIDIA GeForce RTX 3090512x5120.10710.50%0.107 (0%)10.50%NVIDIA GeForce RTX 30901024x10240.45938.00%0.460 (0.2%)38.00%NVIDIA GeForce RTX 3080512x5120.12125.60%0.121 (0%)25.60%NVIDIA GeForce RTX 30801024x10240.52493.00%0.524 (0%)93.00%NVIDIA GeForce RTX 3070512x5120.18331.80%0.183 (0%)31.80%NVIDIA GeForce RTX 30701024x10240.79496.40%0.794 (0%)96.40%观察结论编码RTX 4090 在 2048×2048 下仍只占 47% 显存而 RTX 3070/3080 直接 OOM启用 tiled 编码后显存降到个位数百分比但耗时普遍增加 20%40%。解码1024×1024 是分水岭——RTX 3080 占用达 93%、RTX 3070 达 96.4%此时分块解码的显存收益有限SDXL 场景下分块几乎不省显存远程解码的价值尤其明显。上述数据仅供选型参考实际表现会随批次、dtype、端点负载与网络延迟波动。在仓库中如何进一步研究该功能实现核心remote_utils.pyremote_encode见 L382remote_decode见 L190输入校验check_inputs_decode见 L60请求构造prepare_decode见 L147响应还原postprocess_decode见 L94。端点常量constants.py。解码测试test_remote_decode.py各模型 scaling_factor、形状与参考切片含多分辨率慢测试。编码测试test_remote_encode.py编码-解码往返验证当前编码端点用例标记为 xfail。API 文档api_reference.md。总结远程推理把 VAE 编码/解码从本地 GPU 上摘除直接落到 Inference Endpoints让 SD v1、SDXL、Flux 与 HunyuanVideo 在低显存设备上也能跑高分辨率生成同时免去了 offload 与 tiled 编码带来的额外耗时和质量损耗。接入路径非常清晰vaeNone加载 pipeline →output_typelatent出潜变量 →remote_decode或remote_encode做图像到潜变量→ 配合队列实现“边生成边解码”的流水线吞吐。由于该功能仍属实验阶段且部分编码端点可能下线实际落地前请先验证目标端点可用并以 api_reference.md 中的最新签名为准。【免费下载链接】diffusers Diffusers: State-of-the-art diffusion models for image, video, and audio generation in PyTorch.项目地址: https://gitcode.com/GitHub_Trending/di/diffusers创作声明：本文部分内容由AI辅助生成（AIGC），仅供参考