Portable test suites for evaluating Agent orchestrations (GroupChatOrchestration, etc.) — EvalPort interop?

[adhabnr-ux]

Hi SK community,

Looking through python/samples/getting_started_with_agents/multi_agent_orchestration/step3_group_chat.py, the orchestration pattern is clean: build a GroupChatOrchestration from a list of ChatCompletionAgents and a RoundRobinGroupChatManager, runtime.start(), await group_chat_orchestration.invoke(task=..., runtime=runtime), then await orchestration_result.get() for the final value. That's a nice, uniform shape — any orchestration (group chat, sequential, or your other managers) takes a task string in and produces a result out.

That uniformity made me wonder about testing it at scale: has anyone here wanted a portable way to define "given this task, the orchestration's output should satisfy X" test suites — the kind you'd want to run against an SK agent orchestration, and separately also run against a LangChain/AutoGen/raw-API pipeline doing the same job, to compare apples to apples?

I maintain EvalPort (https://github.com/adhabnr-ux/evalport), a small open spec (JSON Schemas under spec/schemas/ for TestSuite/TestCase/ResultSet/Grader, plus Python/TS SDKs) aimed at exactly that — not another eval runner, just a portable format so a test suite isn't locked to one framework's dataset shape.

Rough sketch of what an adapter over GroupChatOrchestration could look like:

async def run_suite(suite: evalport.TestSuite, orchestration, runtime) -> evalport.ResultSet:
    results = []
    for tc in suite.test_cases:
        res = await orchestration.invoke(task=tc.input, runtime=runtime)
        output = await res.get()
        results.append(evalport.Result(test_case_id=tc.id, actual_output=str(output)))
    return evalport.ResultSet(suite_id=suite.id, results=results)

Given your CONTRIBUTING guidance that plugins/connectors are best hosted outside the SK repo, I'd build and host this as its own small package rather than proposing it land in semantic_kernel/. Posting here first (per COMMUNITY.md) to check whether this is actually a gap for anyone, before spending time on it. Spec for reference: https://github.com/adhabnr-ux/evalport/blob/main/SPEC.md

No pressure either way — genuinely just gauging interest.