Python Conference APAC 2024

Audience Learning Outcomes:

1. Enable BI and ML on a modern two-tier data architecture to support a business continuity plan, including a live demonstration. (with Demo)
2. Improve real-time analysis for a financial service application, including a live demonstration. (with Demo)
3. Create a centralized BI dashboard for organizational performance forecasting, including a live demonstration. (with Demo)
4. Create an automated ETL workflow for cross-functional collaboration, including a live demonstration. (with Demo)
5. Create a data-intelligent service layer for rapid development through experience sharing. (experience sharing)

Stories:

• A financial service company with 10+ applications for customers, including bonds, ETFs/mutual funds, annuities, cryptocurrencies, collectibles (NFT), precious metals, and alternative assets.
• Each customer (or individual investor) has at least 5+ wallets on different applications, and customer data is stored on different applications.

Challenges:

1. Adopting new database technology and data architecture to create data intelligence to fulfill the business continuity plan.
2. Transforming financial processes and ensuring cross-functional collaboration between different applications.
3. Establishing a single source of truth for financial reporting that is GAAP (Generally Accepted Accounting Principles) compliant and includes business logic and accounting logic checks.

Problems:

1. Lack of centralized data storage for low-cost, easy maintenance.
2. Absence of a unified data format for rapid development.
3. Inability to handle streaming data (velocity) for real-time dashboards.
4. Lack of transactional financial reports for customers.
5. Insufficient operational analysis and forecasting for business intelligence.

Solutions:

Build a brand-new data-intelligent service layer with a modern data architecture to start the business continuity plan.

1. Create a modern data architecture and central data storage strategy using dbt Core and Databricks.
2. Implement a data transformation workflow from different applications to deliver high-quality, trusted, and reliable data using dbt Core.
3. Develop real-time analytics, ML, and applications using Databricks Data Streaming.
4. Ensure ACID transactions for customer reports using Delta Lake.
5. Leverage Business Intelligence (BI) and Machine Learning (ML) for operational analysis using the Databricks Lakehouse.

Use Cases:

• Pricing recommendations: Calculating optimal ticket prices.
• Conversion optimization: Uncovering opportunities in the customer journey.
• Marketing: Identifying channels leading to purchases.
• Predictive analytics: Forecasting.
• Fraud detection: Reducing fraud.
• Customer experience: Enhancing fan experience.

Architectures:

1. dbt Core (data orchestration, data catalog, SQL Python-based transformations, reusable code with macros).
2. Databricks Delta Lake (ACID transactions on Spark, scalable metadata, streaming and batch processing).
3. Databricks Lakehouse (scale of data lakes).

Results:

• Simplified data architecture reduces reliance on data engineers.
• Improved data velocity for real-time analysis.
• Confident deployment through an analytics engineering workflow.
• Consistent metrics among different applications for data consistency.
• Automated ETL process for data ingestion and transformation.
• Easy debugging and observability through automated alerts and pipeline visibility.