Aggregation‑based partitioning algorithm for traffic congestion in MapReduce

Faculty Engineering Year: 2025
Type of Publication: ZU Hosted Pages:
Authors:
Journal: Journal of Big Data Springer Open Volume:
Keywords : Aggregation‑based partitioning algorithm for traffic congestion in MapReduce    
Abstract:
The current era has witnessed a remarkable transformation in scientific frontiers, largely driven by advancements in the digital domain. This has resulted in an unprecedented explosion of data known as big data. Among the platforms capable of effectively handling massive data volumes cost-effectively, MapReduce stands out. While previous research has focused on enhancing MapReduce’s overall performance by selecting and scheduling mappers, little focus has been given to optimizing the shuffle phase’s impact on performance. MapReduce operates through multiple phases, with the shuffle phase generating substantial data traffic in the case of heavy jobs. Optimizing or encapsulating aspects, e.g., catalyst, can significantly accelerate the platform’s performance. This work introduces an aggregation-based partitioning algorithm (ABPA) that addresses the limitations of existing approaches commonly adopted to reduce traffic congestion during the intermediate, i.e., shuffle, phase of MapReduce. The proposed ABPA algorithm is evaluated through several experiments involving different data sets of varying types and lengths, employing different numbers of partitions, each containing a specified number of mappers and an aggregator. The experimental results demonstrated a significant reduction in network traffic costs and improved total MapReduce job execution time when using the ABPA scheme. Specifically, the proposed algorithm achieved 73% and 56% network traffic cost improvement over basic hash and conventional aggregation schemes, respectively. Additionally, it reduced total job completion time by 60% and 46% compared to the same schemes.
   
     
 
       

Author Related Publications

  • Rania Ahmed Elsayed Ahmed Mansour Khalifa, "Speech Compression Techniques based on Gaussian Mixture Model and Wavelet Analysis", Ain Shams University, Faculty of Engineering, 2009 More
  • Rania Ahmed Elsayed Ahmed Mansour Khalifa, "Hybrid Method based on Multi-Feature Descriptor for Static Sign Language Recognition", (Institute of Electrical and Electronics Engineers (IEEE, 2017 More
  • Rania Ahmed Elsayed Ahmed Mansour Khalifa, "Skin-based Adaptive Background Subtraction for Hand Gesture Segmentation", (Institute of Electrical and Electronics Engineers (IEEE, 2015 More
  • Rania Ahmed Elsayed Ahmed Mansour Khalifa, "Hand Gesture Recognition Based on Dimensionality Reduction of Histogram of Oriented Gradients", (Institute of Electrical and Electronics Engineers (IEEE, 2017 More
  • Rania Ahmed Elsayed Ahmed Mansour Khalifa, "Performance Enhancement of Indoor Cellular Visible Light Communication through Cell Size and Wavelength Reuse Pattern", Springer Nature, 2023 More

Department Related Publications

  • Mohammed Ayesh Muhammad Hanafi, "Compressed sensing for reliable body area propagation with efficient signal reconstruction", IEEE, 2018 More
  • Saleh Ibrahiem Saied Saleh, "Rate Splitting Multiple Access Scheme for Cognitive Radio Network", The Egyptian International Journal of Engineering Sciences and Technology, 2021 More
  • Saleh Ibrahiem Saied Saleh, "Performance Evaluation of 5G Modulation Techniques", Springer US, 2021 More
  • Nabila Alsawy Elsayed Elsawy, "Mode Skipping for Screen Content Coding Based On Neural Network Classifier", Springer, 2021 More
  • Nabila Alsawy Elsayed Elsawy, "Efficient Coding Unit Classifier for HEVC Screen Content Coding Based on Machine Learning", Springer, 2022 More
Tweet