Exploiting Hybrid Parallelism in Web Search Engines

Abstract:

With the emergence of multicore processors, it is essential to develop techniques that capitalize on multicore's advantages to speed up very demanding applications of parallel computing such as Web search engines. In particular, for this application and given the huge amount of computational resources deployed at data centers, it is of paramount importance to come out with strategies able to get the best performance from hardware. This is specially critical when we consider how we organize hardware to cope with sustained periods of very high traffic of user queries. In this paper, we propose an hybrid technique based on MPI and OpenMP which has been devised to take advantage of the multithreading facilities provided by multicore nodes for search engines under high query traffic.