Map Reduce Optimizer with an Intermediary Cache Manager

Big data is a term used to address the data set whose size is beyond the ability of traditional software technologies to capture, store, manage and process within a tolerable elapsed time. In the world of big data, we observe daily based data which is generated daily e.g. Google, Facebook, and Amazon etc. This large volume of data is un reliable to store, manage and analyze which runs on commodity hardware. As data is in large size it takes more time to execute. The MapReduce framework generates a large amount of intermediate data. These data thrown away after the tasks finish. MapReduce is unable to utilize these data. To improve the efficiency of MapReduce functionality by reducing repeated jobs in data nodes, we proposed an Intermediary cache management system inside the MapReduce framework. In which, tasks submit their intermediate results to the cache manager. Before executing the actual computing work, task queries the cache manager. In an Intermediary cache Management, cache request and cache reply mechanisms are designed. It detects the occurrence of repeated job in the incremental data process.