Document examination is a vital mission for revealing illegal modifications that assist in the detection and resolution of criminal acts. Addition and alteration are more frequently used in handwritten documents. However, most of the documents have been modified with similar inks, and it is tough to detect or observe them with human eyes. As a result, there is a need for methods to automatically detect handwriting forgery to reach an accurate detection efficiently. In this paper, a novel and efficient method is proposed for automatically detecting altered handwritten documents and locating the fake part. Therefore, DE-Net is proposed to identify the forged document using a digitally scanned version of the document. Unlike the existing methods, a further localization schema is applied to locate the forged parts in the candidate forged document accurately. Where each forged document is segmented into objects. Color histograms of R, G, and B channels are used to generate a fused feature vector for each object. Then a structural similarity index (SSIM) is applied to detect the lower similarity parts as forged. The experimental results demonstrate that the proposed method can identify and localize foreign ink in handwritten documents with high performance.