07-14, 11:00–11:45 (US/Pacific), NYC Meetup,[Sessions start: Tuesday 14.07 12pm (Tuesday 14.07 9am PDT)]
BigQuery is GCP's serverless, highly scalable and cost-effective cloud data warehouse that can analyze petabytes of data at super fast speeds. Amazon S3 is one of the oldest and most popular cloud storage offerings. Folks with data in S3 often want to use BigQuery to gain insights into their data. Using Apache Airflow, they can build pipelines to seamlessly orchestrate that connection. In this talk, Emily and Leah will walk through how they created an easily configurable pipeline to extract data
When a team at work mentioned wanting to set up a repeatable process for migrating data stored in S3 to BigQuery, Leah knew using Cloud Composer (GCP-hosted Airflow) was the right tool for the job, but she didn't have much experience with the proprietary file types the data used. Luckily, one of her colleagues did have experience with that proprietary file type, though they hadn't worked with Airflow. Leah and her colleague teamed up to build a reusable, easily configurable solution for the team. She will walk you through their problem, the solution, and the process they took for coming to that solution, highlighting resources that were especially useful to a first-time Airflow user.
Leah Cole is a developer programs engineer at Google, working on Composer, Google Cloud’s hosted version of Apache Airflow. Previously, she worked at GE for on multiple projects in the industrial IoT space. Leah is a graduate of Carleton College, where she studied computer science and also took enough German to have a semi-accidental minor. Outside of work, Leah likes playing piano, traveling, and crocheting.