In this paper, we discuss a system for finding and segmenting a human hand, which is holding an object. This is a difficult computer vision problem, because of the many degrees of freedom, the flexibility, and the clutter problem of the hand. In order to find the correspondences, we have used a lateral similarity function. In the lateral excitation we have not only compare-ed different features at one point but also the features at that point and its neighbors ( e.g. Gobar feature, and skin color feature). The system is an extension of the Elastic Graph Matching (EGM) algorithm [1]. EGM has been shown to be successful in numerous object vision tasks, its best performance being in face recognition.