Skip to content

Redistribute SMB transforms with side inputs - #5991

Draft
yolgun wants to merge 1 commit into
spotify:mainfrom
yolgun:fix/smb-side-input-rebalance
Draft

Redistribute SMB transforms with side inputs#5991
yolgun wants to merge 1 commit into
spotify:mainfrom
yolgun:fix/smb-side-input-rebalance

Conversation

@yolgun

@yolgun yolgun commented Aug 6, 2026

Copy link
Copy Markdown

Summary

  • redistribute SMB bucket offsets with Reshuffle.viaRandomKey when TransformWithSides is used
  • keep TransformWithNoSides topology unchanged
  • add pipeline topology assertions for both paths

This prevents runner forward chaining from pinning side-input SMB work to workers available when processing starts, allowing workers that join later to receive work.

Tests

  • scio-smb/test: 173 passed, 0 failed
  • scio-smb/Compile/javafmtCheck: passed
  • scio-smb/Test/javafmtCheck: passed
  • git diff --check: passed
  • repository-wide sbt test: SMB and other applicable suites passed, but the aggregate command exited non-zero for unrelated local-environment failures:
    • Annoy tests load an x86_64 native library on this arm64 machine
    • scio-snowflake/update expects a missing local Maven artifact at org/bouncycastle/bcprov-jdk15on/1.70

The PR remains in draft pending CI and Flink validation.

Co-Authored-By: Codex <noreply@openai.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

1 participant