PyWren: Pushing Microservices to Teraflops

Much of cloud computing infrastructure remains hard to use, in spite of decades of both academic research and commercialization. Fortunately, recent technologies developed for web services and internet startups can be repurposed to enable a much lower-friction scalable cloud experience. Our goal is making the power, elasticity, and dynamism of commercial cloud services like Amazon's EC2 accessible to busy applied physicists, electrical engineers, and data scientists, as well as a compelling new capability over Matlab, hopefully encouraging migration. We built PyWren, a transparent distributed execution engine on top of AWS Lambda, which hopefully simplifies many scale-out use cases for data science and computational imaging. We will demo applications built on our framework and seek user input into next directions. Joint work with Shivaram Venkataraman, Qifan Pu, Vaishaal Shankar, Allan Peng, Ion Stoica, and Ben Recht.

Readings for this lecture
  1. Occupy the Cloud: Distributed Computing for the 99%
  2. Microservices and teraflops [blog]
  3. Microservices and terabits [blog]
Note that for students with an Amazon Web Services account, instructions for installing pywren and trying it out can be found here. There exist a large number of examples available as Jupyter notebooks in the PyWren Examples repostiory and an excellent step-by-step tutorial as part of the RISECamp Pywren section.

Eric Jonas

Eric Jonas is currently a postdoc in computer science at UC Berkeley working with Ben Recht on machine learning for scientific data acquisition. He earned his PhD in Computational Neuroscience, M. Eng in Electrical Engineering, BS in Electrical Engineering and Computer Science, and BS in Neurobiology, all from MIT. Prior to his return to academia, he was founder and CEO of Prior Knowledge, a predictive database company which was acquired in 2012 by Salesforce.com, where he was Chief Predictive Scientist until 2014. In 2015 he was named one of the top rising stars in bioengineering by the Defense Department’s Advanced Research Projects Agency (DARPA). His work has been featured in The Economist, Ars Technica, and The Atlantic.