Outlines 视觉语言模型结构化输出实战使用 Pixtral-12B 构建图像多级标注流水线【免费下载链接】outlinesStructured Outputs项目地址: https://gitcode.com/GitHub_Trending/ou/outlines导读本文以 docs/guide/vlm.md 为主干讲解如何用 Outlines 驱动视觉语言模型Vision-Language Model, VLM让模型在看图说话的同时直接产出符合预定义 JSON Schema 的结构化输出。文中将以 Mistral 的 Pixtral-12B 为例完整演示从模型初始化、Schema 定义、Prompt 构造到图像结构化生成的端到端流程并结合仓库源码src/outlines/models/transformers.py、src/outlines/inputs.py 等剖析底层实现原理。读完本文你将能够为图像分类、视觉问答、内容标签化等场景搭建图像 → 结构化元数据的生产级流水线。背景Outlines 如何支持多模态模型传统上使用 VLM 做图像理解只能得到自由文本无法直接对接数据库、业务规则或下游 API。Outlines 的核心能力是在解码阶段通过 logits 处理器约束每一步生成的 token使输出天然满足指定的 Pydantic 模型、枚举、正则或 JSON Schema。对本地多模态模型Outlines 通过from_transformers函数将transformers模型包装为统一的 Outlines 模型接口。其关键实现位于 src/outlines/models/transformers.pyfrom_transformers 会根据传入的第二个参数类型自动分派若传入的是PreTrainedTokenizer/PreTrainedTokenizerFast返回纯文本的Transformers模型若传入的是ProcessorMixin如AutoProcessor生成的实例则返回TransformersMultiModal多模态模型。TransformersMultiModal 是围绕transformers模型与 processor 的薄封装构造时会将processor.padding_side设为left、pad_token设为[PAD]以支持批处理并通过TransformersMultiModalTypeAdapter负责把用户输入与输出类型翻译成模型可执行的参数。也就是说传入一个 model 一个 processor这一动作正是让 Outlines 识别并激活视觉多模态能力的开关。环境准备安装 Outlines 及其依赖pip install outlines transformers torch pillowoutlines结构化生成框架本体transformers加载与运行 HF 模型Pixtral 的视觉编码器与语言解码器都依赖它torch模型推理的深度学习框架pillow图像加载、格式转换outlines.inputs.Image在构造时需要读取图像格式并做 Base64 编码见 src/outlines/inputs.py因此 pillow 是硬依赖。初始化视觉多模态模型使用outlines.from_transformers初始化模型。关键点必须同时传入模型实例和能处理文本图像的 processor 实例Outlines 才能判定这是多模态模型并启用对应适配器。import outlines import torch from transformers import ( AutoProcessor, LlavaForConditionalGeneration ) model_name mistral-community/pixtral-12b # 原版 magnet 模型可直接加载 model_class LlavaForConditionalGeneration processor_class AutoProcessor def get_vision_model(model_name: str, model_class, processor_class): model_kwargs { torch_dtype: torch.bfloat16, attn_implementation: flash_attention_2, device_map: auto, } processor_kwargs { device: cuda, } model outlines.from_transformers( model_class.from_pretrained(model_name, **model_kwargs), processor_class.from_pretrained(model_name, **processor_kwargs), ) return model model get_vision_model(model_name, model_class, processor_class)参数说明参数取值作用torch_dtypetorch.bfloat16以 BF16 精度加载权重显著降低显存占用视觉编码与解码均适用attn_implementationflash_attention_2启用 Flash Attention 2 加速注意力计算需对应环境支持device_mapauto自动将模型分布到可用 GPU 显存processor_kwargs.devicecuda将 processor 的输入张量放置到 GPU底层细节from_transformers内部会检查第二个参数是否为ProcessorMixin实例src/outlines/models/transformers.py命中后创建TransformersMultiModal随后TransformersMultiModal.__init__会取processor.tokenizer复用Transformers基类的初始化逻辑src/outlines/models/transformers.py。因此model processor 的组合方式与官方多模态文档 docs/features/models/transformers_multimodal.md 中的做法AutoModelForImageTextToTextAutoProcessor等价。定义输出 Schema下一步是为模型输出定义结构。Outlines 会把 Pydantic 模型编译成解码时的约束即 logits 处理器确保模型只能生成合法 JSON。这里我们定义一个图像标注任务所需的 Schema标签列表含类别与置信度、短标题与密集描述。from enum import Enum from pydantic import BaseModel, Field, confloat, constr from pydantic.types import StringConstraints, PositiveFloat from typing import List from typing_extensions import Annotated class TagType(Enum): ENTITY Entity RELATIONSHIP Relationship STYLE Style ATTRIBUTE Attribute COMPOSITION Composition CONTEXTUAL Contextual TECHNICAL Technical SEMANTIC Semantic class ImageTag(BaseModel): tag: Annotated[ constr(min_length1, max_length30), Field( description( Descriptive keyword or phrase representing the tag. ) ) ] category: TagType confidence: Annotated[ confloat(le1.0), Field( description( Confidence score for the tag, between 0 (exclusive) and 1 (inclusive). ) ) ] class ImageData(BaseModel): tags_list: List[ImageTag] Field(..., min_items8, max_items20) short_caption: Annotated[str, StringConstraints(min_length10, max_length150)] dense_caption: Annotated[str, StringConstraints(min_length100, max_length2048)] image_data_generator outlines.Generator(model, ImageData)要点解读TagTypeEnum约束标签类别只能取 8 个枚举值之一Entity/Relationship/Style/Attribute/Composition/Contextual/Technical/Semantic模型无法输出枚举之外的类别天然保证数据可枚举、可检索。constr/StringConstraints为字符串字段施加长度约束如tag1~30 字符、dense_caption100~2048 字符。confloat(le1.0)置信度上限为 1.0文档语义为开区间 0 到闭区间 1。min_items8, max_items20标签数量下限 8、上限 20保证输出信息密度。outlines.Generator(model, ImageData)Generator工厂函数src/outlines/generator.py会为可引导steerable的本地模型创建SteerableGenerator其内部把ImageData编译为 logits 处理器并挂到模型解码循环上output_type与processor两个参数互斥只能传其一。仓库中 tests/models/test_transformers_multimodal.py 验证了多模态模型下各类输出类型的约束能力Pydantic JSONtest_transformers_multimodal_json、正则Regex(r[0-9])test_transformers_multimodal_regex、枚举选择test_transformers_multimodal_choice说明同一套结构化机制在 VLM 场景下完整可用。构造 Prompt视觉模型的 prompt 需要包含占位标签如image并给出足够详细的指令让模型按 Schema 语义逐项产出。本示例为多阶段原子标注multistage atomic caption任务先做标签生成作为视觉锚定再做短标题与密集描述。pixtral_instruction s[INST] TaskYou are a structured image analysis agent. Generate comprehensive tag list, caption, and dense caption for an image classification system./Task TagCategories requirementYou should generate a minimum of 1 tag for each category. confidenceConfidence score for the tag, between 0 (exclusive) and 1 (inclusive). - Entity : The content of the image, including the objects, people, and other elements. - Relationship : The relationships between the entities in the image. - Style : The style of the image, including the color, lighting, and other stylistic elements. - Attribute : The most important attributes of the entities and relationships in the image. - Composition : The composition of the image, including the arrangement of elements. - Contextual : The contextual elements of the image, including the background, foreground, and other elements. - Technical : The technical elements of the image, including the camera angle, lighting, and other technical details. - Semantic : The semantic elements of the image, including the meaning of the image, the symbols, and other semantic details. Examples noteThese show the expected format as an abstraction. { tags_list: [ { tag: subject 1, category: Entity, confidence: 0.98 }, { tag: subject 2, category: Entity, confidence: 0.95 }, { tag: subject 1 runs from subject 2, category: Relationship, confidence: 0.90 }, } /Examples /TagCategories ShortCaption noteThe short caption should be a concise single sentence caption of the image content with a maximum length of 100 characters. DenseCaption noteThe dense caption should be a descriptive but grounded narrative paragraph of the image content with high quality narrative prose. It should incorporate elements from each of the tag categories to provide a broad dense caption [IMG]image[/INST] .strip()提示词设计要点分阶段指令排序先标签、后标题、再密集描述。标签生成充当对图像的视觉锚定使后续描述更贴合图像内容减少人工后处理。image标签必不可少必须放在模型期望插入图像的位置。从实现层面看多模态输入会被TransformersMultiModalTypeAdapter.format_list_input解析为{text: prompt, images: [...]}交给 HF processorsrc/outlines/models/transformers.pyprocessor 再根据image标签把图像张量注入对应位置若 prompt 中image数量与图像数量不匹配会触发校验错误对应测试 tests/models/test_transformers_multimodal.py 中的test_transformers_multimodal_wrong_number_image。详见官方多模态文档 docs/features/models/transformers_multimodal.md 中的相关警告。生成结构化输出准备图像并调用生成器。示例从 Wikimedia 加载阿波罗 11 号宇航员的著名照片from io import BytesIO from urllib.request import urlopen from PIL import Image def img_from_url(url): img_byte_stream BytesIO(urlopen(url).read()) return Image.open(img_byte_stream).convert(RGB) image_url https://upload.wikimedia.org/wikipedia/commons/9/98/Aldrin_Apollo_11_original.jpg image img_from_url(image_url) result image_data_generator({ text: pixtral_instruction, images: image }) print(result)这里的关键点是输入格式调用生成器时传入一个字典包含text指令 prompt与images图像。这与TransformersMultiModalTypeAdapter的内部解析逻辑一一对应——列表/字典输入会被标准化为{text: ..., images: [...]}再交给 HF processorsrc/outlines/models/transformers.py。图像对象需要是 PILImage且outlines.inputs.Image在构造时会要求图像带 format无 format 会抛TypeError见 src/outlines/inputs.py因此上面的convert(RGB)与保存格式的中间步骤对某些图片来源如无 format 的内存图是必要的。运行后得到形如以下的结构化结果节选{tags_list: [ { tag: astronaut, category: TagType.ENTITY: Entity, confidence: 0.99 }, {tag: moon, category: TagType.ENTITY: Entity, confidence: 0.98}, { tag: space suit, category: TagType.ATTRIBUTE: Attribute, confidence: 0.97 }, { tag: lunar module, category: TagType.ENTITY: Entity, confidence: 0.95 }, { tag: shadow of astronaut, category: TagType.COMPOSITION: Composition, confidence: 0.95 }, { tag: footprints in moon dust, category: TagType.CONTEXTUAL: Contextual, confidence: 0.93 }, { tag: low angle shot, category: TagType.TECHNICAL: Technical, confidence: 0.92 }, { tag: human first steps on the moon, category: TagType.SEMANTIC: Semantic, confidence: 0.95 }], short_caption: First man on the Moon, dense_caption: The figure clad in a pristine white space suit, emblazoned with the American flag, stands powerfully on the moons desolate and rocky surface. The lunar module, a workhorse of space engineering, looms in the background, its metallic legs sinking slightly into the dust where footprints and tracks from the missions journey are clearly visible. The photograph captures the astronaut from a low angle, emphasizing his imposing presence against the desolate lunar backdrop. The stark contrast between the blacks and whiteslicks of lost light and shadow adds dramatic depth to this seminal moment in human achievement. }结果中枚举字段会以TagType.ENTITY这类枚举成员形式返回可以直接接入后续管线如序列化为元数据入库。进阶Chat 接口与批处理除text images 字典输入外Outlines 的多模态模型还支持更便捷的Chat多模态对话接口src/outlines/inputs.py 中的Chat类。TransformersMultiModalTypeAdapter.format_chat_input会调用tokenizer.apply_chat_template自动完成 chat 模板与占位标签注入src/outlines/models/transformers.py无需手动写image标签。消息内容支持三种形态纯字符串、文本资源列表如[Describe..., Image(...)]、以及带显式 type 的字典列表如{type: text, ...}、{type: image, image: Image(...)}仅限 HF transformers 模型。完整的 Chat 与批处理示例可参考 docs/features/models/transformers_multimodal.md 以及批处理测试 tests/models/test_transformers_multimodal.py。批处理方面TransformersMultiModal支持通过batch方法传入多个 prompt 并行生成底层会把各输入的text合并、资源展平后统一交给 processorpaddingTrue补齐配合构造时设置的padding_sideleft保证左侧填充不影响解码src/outlines/models/transformers.py。多图输入、多序列采样num_return_sequencesnum_beams均有测试覆盖。实际应用场景本指南展示的技术可直接用于构建以下系统内容管理系统Content Management Systems自动为视觉内容打标签与分类产出的结构化元数据可直接写入数据库支撑强大的搜索与筛选能力无障碍工具Accessibility Tools为图片生成丰富的结构化描述可适配不同场景——从简短的 alt 文本到面向读屏软件的详细场景描述质量保障流水线Quality Assurance Pipelines通过抽取结构化属性并对照业务规则校验对视觉内容是否符合特定标准进行自动化质检。总结通过本文的完整流程——from_transformers初始化多模态模型、Pydantic Schema 定义输出结构、分阶段指令 Prompt 与image占位标签、outlines.Generator约束解码——你可以让任意兼容 HF 的视觉语言模型输出可直接入库的结构化 JSON。仓库源码与测试src/outlines/models/transformers.py、src/outlines/inputs.py、tests/models/test_transformers_multimodal.py验证了这一机制在 JSON、正则、枚举、批处理等多模态场景下的稳定性与通用性。若需进一步了解纯文本模型接口的异同可对照阅读 docs/features/models/transformers.mdchat 模板相关的底层机制可参考 docs/guide/chat_templating.md。【免费下载链接】outlinesStructured Outputs项目地址: https://gitcode.com/GitHub_Trending/ou/outlines创作声明：本文部分内容由AI辅助生成（AIGC），仅供参考