Trillion-Token Pretraining: Building a Foundational Model for payment data
Martin Iglesias Goyanes, Raúl Soutelo Quintela
Pretraining foundation models on tabular and sequential data presents challenges that differ fundamentally from NLP or vision. This talk covers the key design decisions involved in building a payments foundation model trained on billions of transactions and trillions tokens: tokenisation of heterogeneous features, sequence construction, masking strategies and pretraining objectives for sequential tabular data, and the architectural trade-offs between hierarchical and language-model framings. Attendees will leave with transferable techniques for adapting self-supervised pretraining to non-text sequential data.
