2020-07-17 –, Warsaw Meetup, [Sessions start: Friday 17.07 6 pm (Friday 17.07 9am PDT)]
Being a pioneer for the past 25 years, SONY PlayStation has played a vital role in the Interactive Gaming Industry. Over 100+ million monthly active users, 100+ million PS-4 console sales along with thousands of game development partners across the globe, big-data problem is quite inevitable. This presentation talks about how we scaled Airflow horizontally which has helped us building a stable, scalable and optimal data processing infrastructure powered by Apache Spark, AWS ECS, EC2 and Docker.
PlayStation being the most popular gaming console across the globe with over 100+ million monthly active users, data is being generated by all means. Due to the demand for processing large volumes of data and also to meet the growing Organization’s data analytics and usage demands, data team here at PlayStation took an initiative to build an open source big data processing infrastructure where Apache Spark in Python as the core ETL engine. Apache Airflow is the core workflow management tool for the entire eco system. We started with an Airflow application running on a single AWS EC2 instance to support parallelism of 16 with 1 scheduler and 1 worker and eventually scaled it to a bigger scheduler along with 4 workers to support a parallelism of 96, DAG concurrency of 96 and a worker task concurrency of 24. Containerized all the services on AWS ECS which gave us an ability to scale Airflow horizontally.
Naresh has about 12+ years of hands-on experience in big data and data warehousing technologies. Currently working as a Lead Data Engineer at PlayStation based of Los Angeles, CA. Prior to that, he had opportunities to work for various organizations such as GRUBHUB, Comcast, Dell and AT&T.
He holds a Bachelors degree in Electrical Engineering and a Masters Degree in Power Electronics.
Doing DIY home projects on weekends, reading tech blogs, crawling through LinkedIn network are few of his hobbies
God Of War is his favorite 1st party PlayStation game.
