The convex hull is a way of characterizing the shape of an image by determining which pixels are adjacent to other pixels of the same intensity. This is a good way to find holes and convex features in an image.

The function returns a binary image matrix that contains the convex hull of M, with foreground pixels set to value 1 and background to 0.

The output is binarized with values of 1 inside the convex hull and 0 outside. The hull is found by choosing P1 as the leftmost and topmost point of the set of pixels in M and L1 as the horizontal line through P1. Then it rotates L1 about P1 until it hits the value fg in the set of pixels. Calling the resulting rotated line L2 and letting P2 be the point farthest from P1 along L2, the algorithm then repeats, until Pn = P1. The union of {P1, P2, ... , Pn-1} are the vertices of the convex hull.

The algorithm used is described in Digital Picture Processing by A. Rosenfeld and A. C. Kak, page 269, 1982.