Joining holes in a 2D matrix :

Question

My question is similar to this question about finding holes.

However, I want to also join the holes to make a single, consecutive area. The "strech" of the "bridge" that joins two holes should be minimal. What is the most robust and fastest algorithm?

My attempt to solve is the following :

for each row in matrix :
  for each element in row :
     e = element
     d = a large number
     for each row in matrix :
       for each element in row :
         f = element 
         if ( f != e && f != background && e != background) :
            d_temp = calculate distance between e & f
            if (d_temp < d) :
               d = dtemp;
               mark location of e & f
join the marked locations

As it is clear, the algorithm is of complexity O(N²M²), given a matrix of size N×M.

Is there a better way? Thank you.

Please note, I want the algorithm to be robust first (i.e. should never fail or degenerate) - and when that condition is satisfied, I want it to be fast.

PS: I am using D, if that is relevant.