In this paper, an algorithm is proposed to integrate the unsupervised learning with the optimization of the Finite Mixture Models (FMM). While learning parameters of the FMM the proposed algorithm minimizes the mutual information among components of the FMM provided that the reduction in the likelihood of the FMM to fit the input data is minimized. The performance of the proposed algorithm is compared with the performances of other algorithms in the literature. Results show the superiority of the proposed algorithm over the other algorithms especially with data sets that are sparsely distributed or generated from overlapped clusters.