2026-09-11 –, Main stage
Pretraining foundation models on tabular and sequential data presents challenges that differ fundamentally from NLP or vision. This talk covers the key design decisions involved in building a payments foundation model trained on billions of transactions and trillions tokens: tokenisation of heterogeneous features, sequence construction, masking strategies and pretraining objectives for sequential tabular data, and the architectural trade-offs between hierarchical and language-model framings. Attendees will leave with transferable techniques for adapting self-supervised pretraining to non-text sequential data.
Building a foundation model for payments data requires rethinking assumptions borrowed from NLP and vision. This talk walks through why traditional ML hits a ceiling on tabular financial data, the architectural decisions, choosing between a hierarchical payment encoder and a language-model framing, and the engineering required to pretrain at the scale of trillions of tokens.
We cover:
- Why payments need a foundation model
- Architecture trade-offs: hierarchical encoder vs. treating features as tokens, payments as sentences, sequences as documents; concrete pros and cons of each (information bottleneck, context window blowup)
- Tokenization: representing mixed numeric, categorical, and high-cardinality ID features as a single vocabulary; handling missing fields and temporal structure
- How to handle sequences ranging from 1k to 100M payments per entity and extract meaningful signal
- Pretraining objectives: masked feature modeling, next-event prediction, and contrastive tasks
- Comparing standard attention, Flash Attention, windowed attention, and Mamba SSMs across memory, compute, and wall-clock time
- Inference constraints (P50 30ms SLA, 3000 tx/s peaks, 99.9999% uptime)
- Results and impact on various ML models (fraud detection, entity resolution, and more)
Attendees will leave knowing how to apply state-of-the-art techniques for adapting self-supervised pretraining to non-text sequential data: how to tokenize heterogeneous tabular features, which pretraining objectives work, architectural trade-offs, and how to serve a large model under strict latency constraints.
Timing (25 min + 5 min Q&A):
- Problem framing and motivation - 5 min
- Data: sequences and tokenization - 5 min
- Architecture and pretraining objectives - 5 min
- Engineering challenges - 5 min
- Results and lessons learned - 5 min
- Q&A - 5 min
Target audience: Intermediate-level; familiarity with Transformers and self-supervised learning is assumed.
Hanna is an AI research engineer at Adyen - a payment processor processing over 40B transactions a year, where she focuses on deep learning research, training infrastructure, and building foundational models. She is passionate about solving large-scale engineering challenges and deploying AI to optimize complex systems and drive real-world impact.
I work in Applied Machine Learning at Adyen, where I focus on deep learning research and training infrastructure. I am passionate about bridging SOTA deep learning with the scale and complexity of the financial industry.