使用开源的权重进行vLLM推理与online demo的结果不一样

[MarginGitHub]

如题，请问开源的模型权重和online demo的权重一样吗？还是调用配置有差异？我用vLLM调用方式跟开源的代码一致

[MarginGitHub]

online demo的输出结果是：内漢試作何好の？(210,25),(685,175)ミルの比率を何度も×2試作(156,317),(819,511)白さとり・きぶさとり・ひんとり・和さんほん・(158,537),(865,666)せんとり・色んなさとりの中じれがパパ外(147,710),(866,830)あすゅ？(144,883),(285,1000)
vLLM本地调用输出结果是：内漢試作何好の？(210,23),(685,175)ミルの比率を何度も×2試作(156,316),(819,512)白さとり・さとり・ひとり・和さんほん・(158,537),(865,666)せんとり・色んなさとりの中じれがパパ外(147,710),(865,830)あすゅ？(144,883),(285,1000)

[memorywxy]

您好，使用的模型权重是一样的，预测结果存在差异应该是推理过程存在diff。可以把完整的推理脚本发出来，我们来定位一下原因。

[MarginGitHub]

您好，使用的模型权重是一样的，预测结果存在差异应该是推理过程存在diff。可以把完整的推理脚本发出来，我们来定位一下原因。

import os
import math
import re
from vllm import LLM, SamplingParams
from PIL import Image
from modelscope import AutoProcessor
import numpy as np
import cv2
import base64


max_tokens = 1024

model_path = ""

img_base_dir = ""

def encode_image(img: np.ndarray) -> str:
    return base64.b64encode(img.tobytes()).decode('utf-8')


def run_ocr_show_single():

    llm = LLM(model=model_path, 
          tensor_parallel_size=1,
          max_model_len=16384 * 2,
          gpu_memory_utilization=0.9,
          )
    processor = AutoProcessor.from_pretrained(model_path)
    sampling_params = SamplingParams(temperature=0, 
                                     max_tokens=max_tokens, 
                                     top_k=1,
                                     repetition_penalty=1.,
                                     top_p=0.95,
                                     seed=1234,
                                    )
    

    while True:
        name = input("请输入名字：")
        if name == "exit":
            break
        file_path = os.path.join(img_base_dir, name + ".png")
        cv_img = cv2.imread(file_path, -1)
        cv_img = cv2.cvtColor(cv_img, cv2.COLOR_GRAY2RGB)
        pil_img = Image.fromarray(cv_img)
        # pil_img = Image.open(file_path)
        messages = [
                {"role": "user", "content": [
                    {
                                "type": "image_url",
                                "image_url": {
                                    "url": f"data:image/png;base64,{encode_image(cv_img)}"
                                }
                            },
                    {"type": "text", "text": "检测并识别图片中的文字，将文本坐标格式化输出。"},
                ]}
        ]

        # messages = [
        #     {"role": "system", "content": ""},
        #     {"role": "user", "content": [
        #         {"type": "image", "image": file_path},
        #         {"type": "text", "text": "检测并识别图片中的文字，将文本坐标格式化输出。"}
        #     ]}
        # ]
        prompt = processor.apply_chat_template(messages, 
                                               tokenize=False, 
                                               add_generation_prompt=True)
        
        
        inputs = {"prompt": prompt, "multi_modal_data": {"image": [pil_img]}}
        output = llm.generate([inputs], sampling_params)[0]
        text = output.outputs[0].text
        print(text)

run_ocr_show_single()

[memorywxy]

传入的messages需要增加system prompt
{"role": "system", "content": ""}