Indexat a

Llicència i ús

Citacions

Cited 6 times in

Cited 5 times in

Altmetrics

Grant support

We appreciate all anonymous reviewers at Middleware'22, who provided insightful feedback that makes this paper much stronger. This work has been partially supported by EU (No. 825184) and Spanish Government (No. PID2019-106774RB-C22). Marc SanchezArtigas is a Serra Hunter Fellow.

Anàlisi d'autories institucional

Compartir

A Seer Knows Best: Optimized Object Storage Shuffling for Serverless Analytics

Publicat a:Proceedings Of The Twenty-Third Acm/Ifip International Middleware Conference, Middleware 2022. 148-160 - 2022-01-01 (), DOI: 10.1145/3528535.3565241

Autors: Sanchez-Artigas, Marc; Eizaguirre, German T

Afiliacions

Resum

Serverless platforms offer high resource elasticity and pay-as-you-go billing, making them a compelling choice for data analytics. To craft a "pure" serverless solution, the common practice is to transfer intermediate data between serverless functions via serverless object storage (IBM COS; AWS S3). However, prior works have led to inconclusive results about the performance of object storage, since they have left large margin for optimization. To verify that object storage has been underrated, we design a novel shuffle manager for serverless data analytics termed Seer. Specifically, Seer dynamically chooses between two shuffle algorithms to maximize performance. The algorithm choice is based on some predictive models, and very importantly, without users having to specify intermediate data sizes at the time of the job submission. We integrate Seer with PyWren-IBM [31], a serverless analytics framework, and evaluate it against both serverful (e.g., Spark) and serverless systems (e.g., Google BigQuery). Our results certify that our new shuffle manager can deliver performance improvements over them.

Paraules clau