This homework covers standard binary (object/backgroud) segmentation methods using s/t graph cut or max-flow algorithms.
In general, energies combining unary data (e.g. color) terms and pair-wise boundary regularization terms (see below) are common in many computer vision problems. Such energies integrate different cues in a coherent fashion.
Similarly to the first assignemnt, you can start from an EZi-based
Graph cut project implementing an interface that allows to load images, enter seeds or boxes, play with parameters, switch between EDGE, COLOR_F, and COLOR_E segmentation modes, and save your results as images. EDGE mode is binary segmentation based on seeds (hard constraints) and edge alignemnt cues (see Topic 6 slide 5). It is fully implemented using an included max-flow library. COLOR_F is a mode with additional data term using fixed color models/histograms (slides 43-49), as in energy
Mode COLOR_E estimates color histograms treating them as optimization variables jointly with segmnetation S (slides 70-74), as in "GrabCut". These two COLOR modes should be implemented. You can work with function compute_mincut in file Graph2D.cpp. You can also use MATLAB or other software, as long as you write your own code constructing and manipulating the graphs. MATLAB wrapper for the same max-flow library can be found here (see Max-flow/min-cut).
Implement COLOR_F mode using fixed color appearance probability distributions (histograms in RGB space) for object and background. Use 3D histograms in RGB space with bins of width 32 in each channel (8 bins within range 0-255 in each channel). This gives RGB histogram with 512 = 8^3 bins. Your segmentation should be optimal for the energy above incorporating color appearance term and the same boundary smoothness term used in EDGE mode. Note that the interface allows you to choose the parameter "r" (relating weight of the data term in the energy, member variable m_r in class Graph2D). Use red and blue seeds both as hard constraints and as sample color points for computing fixed histograms. Compare your segmentation results with the EDGE mode and present ypour findings in the report.
Implement COLOR_E mode where color histograms are treated as variables. Optimize the energy combining model parameters and segmentation S using block-coordinate descent, as discussed in class. Compare all modes: EDGE, COLOR-F, COLOR-E and present your findings in the report.
(Grad students required, BONUS for undegrads): use a box-based interface (an option in BRUSHes drop-down list) to set initial color models based on pixels inside and outside the user-defined box. You can also set hard "background" constraints for points outside the box. Seeds can be edded later to correct the results.
(BONUS for all) implement background subtraction as follows: use the difference image and appropriate
fixed models for object and background pixels (as discussed in class). Compare two results: one using thresholding for two models and the other using regularization energy (as above) with
fixed models likelihoods. You should take your own pairs of images (with and without an object) using a tripod to prevent camera motion.
(SUPER BONUS for background subraction above): instead of fixed models for object and background in your "difference image", treat the corresponding "appearance" models as variables jointly with segmentation in the regularization energy above. In particular, use a Gaussian distribution model at each segment (object and background) where both "mean" and "variance" parameters are unknown. This corresponds to an "elliptic" version of K-means (K=2) with regularization (also "elliptic" Chan-Vese). This approach avoids artificial fixed models. For simplicity, use a grey-scale version of the difference image. NOTE: correct implementation of elliptic version of K-means might be tricky, talk to me after class if you are interested to try this.
What to submit
Submit a pdf file with your report (no more than 1.5 pages of text, but any number of images representing your results) summarizing your efforts in achieving the goals described above.
In general, images of your resulst are highly encourages as they are an ideal way to represent your experiemnts in image analysis. The report should be submitted electronically via OWL. You should also submit your code (in a .zip file). The project report and code should be your independent effort.