Background Dense breast tissue significantly increases breast cancer (BC) risk. However, current mammographic methods for classifying BC are often subjective and unreliable, which complicates the task of accurate evaluation. Purpose This study introduces a deep learning method with a local context attention module (LCAM), using dual mammogram views aligned with BI-RADS to enhance grading consistency and accuracy in BC classification across four groups by leveraging local context around masses. Methods Specific regions of interest (ROIs) containing dense tissue around breast masses are identified from dual mammogram views, providing additional insights for predicting BC BI-RADS categories. These ROIs are then input into a convolutional neural network (CNN)-based model, which is crucial for selecting and differentiating radiomic features associated with BI-RADS. To enhance our model's ability to distinguish salient radiomic features associated with mass malignancy, the LCAM sequentially infers attention maps along two separate dimensions: channel and spatial. These attention maps are subsequently multiplied with the input feature map for adaptive feature refinement. Results Examining 3020 patients across four BI-RADS categories while leveraging dual mammogram views demonstrates the robust performance of the proposed framework, achieving a sensitivity of 82.46% and a specificity of 91.42% in identifying BI-RADS grading relevant to breast masses. Conclusions We introduced a novel CNN-based framework that utilizes dual mammogram views for the BC classification. It utilizes LCAM, which further understands the local characteristics surrounding breast masses, aiming to enhance the accuracy and consistency of classification outcomes.