基于MLFlow的Model Evaluation

1. MLFlow基于单个trace/单次对话的模型评估

• MLFlow.genai提供了多个scorer作为模型评估的标准，比如‘Completeness’, 'Correctness', 'Safety', 'Equivalence'等适用于大部分模型的，其中正确性有关的需要在测试数据中提供‘expecrations’. 此外还可使用Guidelines自定义score. 这些scorer可直接在Databricks界面中添加到现有experiments中，或通过脚本注册。也可进行一次性评估，如下：

#测试数据实例
single_turn_data = [
    {
        "inputs" : {
            "messages": [
                {"role": "user", "content": "What is LLM"}
             ]
        },
        "expectations": {
            "expected_response": "..."
        }
    },
    ...
]

# Scorers示例
scorers = [
    Correctness(),
    RelevanceToQuery(),
    Safety(),
    Guidelines(
        name = "concise"
        guidelines = "回答简洁，不可超过300字“
    ),
]

#定义如何调用模型
def predict_fn(messages):
    client = WorkspaceClient().serving_endpoints.get_open_ai_client()
#从serving endpoint调用模型进行评估

    response = client.chat.completions.create(
        model = endpoint_name,
        messages = messages,
    )
    return {"response": response.choices[0].message.content}

#单伦评估示例
mlflow.set_experiment({})

with mlflow.start_run(run_name={}):
    single_turn_results = mlflow.genai.evaluate(
        data = single_turn_data,
        predict_fn = predict_fn,
        scorers = scorers,
    )

• 此外还提供了基于文档（RAG）的模型的相关scorers，比如Retrieval Groundedness, RetrievalRelevance, RetrievalSufficiency

2. MLFlow也可提供基于某次完整对话session的整体评估，比如conversation_completness， user_frustration, conversation_tool_call_efficiency, 等，都是基于多轮完整对话的评估：

mlflow.set_experiment(EXPERIMENT_NAME）

scorers = [
    ConversationCompletness(),
    ConversationToolcallEfficiency(),
    UserFrustration(),
    ConversationalSafety(),
]

#Register scores and start apply them in the experiment
for s in scores:
    r = s.register(s.name)
r = r.start()

# Test Sample
TEST_SESSIONS = [
    { "name": "happy_path",
        "turns": [
            "Hello, I'm xixi",
            "What is LLM",
            "What's my name",
        ],
    }
    ...
]

# Session evaluation
@mlflow.trace(name={run_name}, span_type = "AGENT")
def run_turn(messages, session_id, user_id):
    # Set session_id and user_id for current trace, so that mlflow can collect traces into same session
    mlflow.update_current_trace(
        metadata = {
            "mlflow.trace.session": session_id,
            "mlflow.trace.user": user_id
        }
    response = client.predict...

# 评估
for sess in TEST_SESSIONS:
    session_id = ...
    user_id = ...
    for user_msg in session["turns"]:
        history.append({"role": "user", "content": user_msg})
        assistant_reply = run_turn(history, session_id, user_id)
        history.append("role": "assistant", "content": assistant_reply})
    ...

# 对已有traces进行评估

traces = mlflow.search_traces(experiment_id = ...,
filter_string = ...)

mlflow.genai.evaluate(data=traces, scorers=scorers)