Trillion-Token Pretraining: Building a Foundational Model for payment data
Pretraining foundation models on tabular and sequential data presents challenges that differ fundamentally from NLP or vision. This talk covers the key design decisions involved in building a payments foundation model trained on billions of transactions and trillions tokens: tokenisation of heterogeneous features, sequence construction, masking strategies and pretraining objectives for sequential tabular data, and the architectural trade-offs between hierarchical and language-model framings. Attendees will leave with transferable techniques for adapting self-supervised pretraining to non-text sequential data.
Building a foundation model for payments data requires rethinking assumptions borrowed from NLP and vision. This talk walks through why traditional ML hits a ceiling on tabular financial data, the architectural decisions, choosing between a hierarchical payment encoder and a language-model framing, and the engineering required to pretrain at the scale of trillions of tokens.
We cover:
- Why payments need a foundation model
- Architecture trade-offs: hierarchical encoder vs. treating features as tokens, payments as sentences, sequences as documents; concrete pros and cons of each (information bottleneck, context window blowup)
- Tokenization: representing mixed numeric, categorical, and high-cardinality ID features as a single vocabulary; handling missing fields and temporal structure
- How to handle sequences ranging from 1k to 100M payments per entity and extract meaningful signal
- Pretraining objectives: masked feature modeling, next-event prediction, and contrastive tasks
- Comparing standard attention, Flash Attention, windowed attention, and Mamba SSMs across memory, compute, and wall-clock time
- Inference constraints (P50 30ms SLA, 3000 tx/s peaks, 99.9999% uptime)
- Results and impact on various ML models (fraud detection, entity resolution, and more)
Attendees will leave knowing how to apply state-of-the-art techniques for adapting self-supervised pretraining to non-text sequential data: how to tokenize heterogeneous tabular features, which pretraining objectives work, architectural trade-offs, and how to serve a large model under strict latency constraints.
Timing (25 min + 5 min Q&A):
- Problem framing and motivation - 5 min
- Data: sequences and tokenization - 5 min
- Architecture and pretraining objectives - 5 min
- Engineering challenges - 5 min
- Results and lessons learned - 5 min
- Q&A - 5 min
Target audience: Intermediate-level; familiarity with Transformers and self-supervised learning is assumed.
I work in Applied Machine Learning at Adyen, where I focus on deep learning research and training infrastructure. I am passionate about bridging SOTA deep learning with the scale and complexity of the financial industry.
Raúl Soutelo Quintela works as an AI Research Engineer at Adyen since 2025.