A fusion design for two-stream RGB and depth segmentation networks that deliberately discards cross-modal attention terms instead of adding more of them. Query, key and value are computed from the concatenated RGB and depth feature maps after a 1x1 pointwise convolution halves the channel dimension, and because the query and key retain their modality block structure, the product of query and transposed key collapses exactly to the sum of the RGB-only and depth-only attention maps. The softmax over that sum therefore strengthens spatial relationships inside each modality at lower cost than inter-modal fusion, a shortcut that is valid precisely because RGB and depth are pixel-aligned so simple element-wise addition already carries the cross-modal information. A channel-wise dynamic fusion stage then global-average-pools the fused feature and passes it through a sigmoid to produce a weight vector that scales the RGB stream and its complement that scales the depth stream, giving learned per-channel modality balance rather than equal weighting. The module drops between corresponding layers of two-stream detection-and-segmentation transformers, and teaches the general lesson that adding modality-specific refinement can beat adding cross-modal interaction while costing less computation.
A computer-vision benchmark-and-method paper that attacks the shortage of instance-level RGB-D segmentation data and then proposes a fusion module for it. Problem: RGB-D segmentation research has co…