Shaheer U. Saeed, Shiqi Huang, João Ramalhinho, Iani J.M.B. Gayo, Nina Montaña-Brown, Ester Bonmati, Stephen P. Pereira, Brian Davidson, Dean C. Barratt, Matthew J. Clarkson and Yipeng Hu 2025. Competing for Pixels: A Self-Play Algorithm for Weakly-Supervised Semantic Segmentation. IEEE Transactions on Pattern Analysis and Machine Intelligence. 47 (2), pp. 825-839. https://doi.org/10.1109/TPAMI.2024.3474094
| Title | Competing for Pixels: A Self-Play Algorithm for Weakly-Supervised Semantic Segmentation |
|---|---|
| Type | Journal article |
| Authors | Shaheer U. Saeed, Shiqi Huang, João Ramalhinho, Iani J.M.B. Gayo, Nina Montaña-Brown, Ester Bonmati, Stephen P. Pereira, Brian Davidson, Dean C. Barratt, Matthew J. Clarkson and Yipeng Hu |
| Abstract | Weakly-supervised semantic segmentation (WSSS) methods, reliant on image-level labels indicating object presence, lack explicit correspondence between labels and regions of interest (ROIs), posing a significant challenge. Despite this, WSSS methods have attracted attention due to their much lower annotation costs compared to fully-supervised segmentation. Leveraging reinforcement learning (RL) self-play, we propose a novel WSSS method that gamifies image segmentation of a ROI. We formulate segmentation as a competition between two agents that compete to select ROI-containing patches until exhaustion of all such patches. The score at each time-step, used to compute the reward for agent training, represents likelihood of object presence within the selection, determined by an object presence detector pre-trained using only image-level binary classification labels of object presence. Additionally, we propose a game termination condition that can be called by either side upon exhaustion of all ROI-containing patches, followed by the selection of a final patch from each. Upon termination, the agent is incentivised if ROI-containing patches are exhausted or disincentivised if a ROI-containing patch is found by the competitor. This competitive setup ensures minimisation of over- or under-segmentation, a common problem with WSSS methods. Extensive experimentation across four datasets demonstrates significant performance improvements over recent state-of-the-art methods. |
| Keywords | self-play, weak supervision, segmentation |
| Journal | IEEE Transactions on Pattern Analysis and Machine Intelligence |
| Journal citation | 47 (2), pp. 825-839 |
| ISSN | 1939-3539 |
| 2160-9292 | |
| 0162-8828 | |
| Year | 2025 |
| Publisher | IEEE |
| Accepted author manuscript | License CC BY 4.0 File Access Level Open (open metadata and files) |
| Digital Object Identifier (DOI) | https://doi.org/10.1109/TPAMI.2024.3474094 |
| Publication dates | |
| Published | 03 Oct 2024 |
| Published in print | Feb 2025 |