A Self-Evolving Semantic Partition Learning Framework for Adaptive Image Segmentation Using Multi-Level Region Correlation and Intelligent Pixel Relationship Modeling

Abstract

Static segmentation architectures make a static partitioning decision for all images to accommodate a particular amount of partitioning and cannot adjust to amount or type of change in partitioning needs that may arise for different parts of an image or in different scenes.  propose Self-Evolving Semantic Partition (SESP), an adaptive segmentation method which iteratively evolves the segmentation structure according to the reliability of the different levels of partition granularity in the image by three co-located components: (1) Intelligent Pixel Relationship Modeling (IPRM) that learns a number of pairwise pixel relationships jointly, both in appearance space and in learned semantic embedding space, rather than using any fixed similarity function; (2) Multi-Level Region Correlation (MLRC), which computes statistical correlation between candidate regions with different granularity of partition (pixel, superpixel and object-part) and uses agreements/ disagreements among the different levels as feedback signals to guide the process of partition self-evolution, allocating more refinement steps to less reliable levels and pruning the process earlier to more reliable levels; and (3) a self-evolution controller that integrates the three interactive components and iteratively evolves the partition. SESP is a segmentation method which does not have an automatic configuration number of segmentation steps nor a fixed number of levels of refinement steps; in other words, these can be considered as image-adaptive values. demonstrate steady gains across each of five benchmarks (BSDS500, PASCAL VOC 2012, Cityscapes, ADE20K and COCO-Stuff) in terms of mean Intersection-over-Union, Boundary F-score and Variation of Information compared with static-depth baselines, and analyze the learned per-image refinement depth, finding that it is correlated with independent measures of scene complexity. Ablations show that both IPRM and MLRC offers complementary enhancements, and that the self-evolution controller offers even further gains than just a network with a deeper fixed depth.