Aggregation‑based partitioning algorithm for traffic congestion in MapReduce

Faculty Engineering Year: 2025
Type of Publication: ZU Hosted Pages:
Authors:
Journal: Journal of Big Data Springer Open Volume:
Keywords : Aggregation‑based partitioning algorithm for traffic congestion in MapReduce    
Abstract:
The current era has witnessed a remarkable transformation in scientific frontiers, largely driven by advancements in the digital domain. This has resulted in an unprecedented explosion of data known as big data. Among the platforms capable of effectively handling massive data volumes cost-effectively, MapReduce stands out. While previous research has focused on enhancing MapReduce’s overall performance by selecting and scheduling mappers, little focus has been given to optimizing the shuffle phase’s impact on performance. MapReduce operates through multiple phases, with the shuffle phase generating substantial data traffic in the case of heavy jobs. Optimizing or encapsulating aspects, e.g., catalyst, can significantly accelerate the platform’s performance. This work introduces an aggregation-based partitioning algorithm (ABPA) that addresses the limitations of existing approaches commonly adopted to reduce traffic congestion during the intermediate, i.e., shuffle, phase of MapReduce. The proposed ABPA algorithm is evaluated through several experiments involving different data sets of varying types and lengths, employing different numbers of partitions, each containing a specified number of mappers and an aggregator. The experimental results demonstrated a significant reduction in network traffic costs and improved total MapReduce job execution time when using the ABPA scheme. Specifically, the proposed algorithm achieved 73% and 56% network traffic cost improvement over basic hash and conventional aggregation schemes, respectively. Additionally, it reduced total job completion time by 60% and 46% compared to the same schemes.
   
     
 
       

Author Related Publications

  • Rania Ahmed Elsayed Ahmed Mansour Khalifa, "Speech Compression Techniques based on Gaussian Mixture Model and Wavelet Analysis", Ain Shams University, Faculty of Engineering, 2009 More
  • Rania Ahmed Elsayed Ahmed Mansour Khalifa, "Hybrid Method based on Multi-Feature Descriptor for Static Sign Language Recognition", (Institute of Electrical and Electronics Engineers (IEEE, 2017 More
  • Rania Ahmed Elsayed Ahmed Mansour Khalifa, "Skin-based Adaptive Background Subtraction for Hand Gesture Segmentation", (Institute of Electrical and Electronics Engineers (IEEE, 2015 More
  • Rania Ahmed Elsayed Ahmed Mansour Khalifa, "Hand Gesture Recognition Based on Dimensionality Reduction of Histogram of Oriented Gradients", (Institute of Electrical and Electronics Engineers (IEEE, 2017 More
  • Rania Ahmed Elsayed Ahmed Mansour Khalifa, "Performance Enhancement of Indoor Cellular Visible Light Communication through Cell Size and Wavelength Reuse Pattern", Springer Nature, 2023 More

Department Related Publications

  • Hanaa Shaker Abdelbaset Ali, "Entropy with Local Binary Patterns for Efficient Iris Liveness Detection", Springer US, 2017 More
  • Hanaa Shaker Abdelbaset Ali, "Rakeness with block sparse Bayesian learning for efficient ZigBee‐based EEG telemonitoring", Wiley, 2019 More
  • Rania Ahmed Elsayed Ahmed Mansour Khalifa, "Dual-Band Millimeter-Wave Circularly Polarized Antenna for Mobile Communications", IEEE, 2022 More
  • Rania Ahmed Elsayed Ahmed Mansour Khalifa, "Internet of Medical Things and Healthcare 4.0: Trends, Requirements, Challenges, and Research Directions", MDPI, 2023 More
  • Rania Ahmed Elsayed Ahmed Mansour Khalifa, "Securing IoT and SDN systems using deep-learning based automatic intrusion detection", Elsevier, 2023 More
Tweet