, Ferrum [2nd Floor]
Teaching an LLM to Tune GBDT — and Beyond
Hyperparameter optimization for gradient boosted tree models is a repetitive yet cognitively demanding task. Practitioners must combine statistical intuition with detailed, library-specific knowledge—often buried across hundreds of pages of documentation for tools such as XGBoost, LightGBM, or CatBoost. As models and configurations grow in complexity, traditional approaches like grid search, random search, or even Bayesian optimization struggle to incorporate semantic understanding of model behavior.
Using LGBM as a concrete case study, I demonstrate how MCP and skills-powered agents, orchestrated in a structured workflow, can analyze model behavior and propose targeted hyperparameter adjustments grounded in both theory and library-specific constraints.
Why This Problem Matters in Practice
Hyperparameter tuning consumes a disproportionate amount of experimentation time, yet most tuning failures stem from recurring structural issues — not random chance. Experienced practitioners can spot these patterns, but automated optimizers only see scalar objective values.
What Is New or Different
This work reframes hyperparameter optimization as an iterative reasoning process rather than a pure search problem. Intermediate diagnostic artifacts (parameter importance, generalization gaps, plateau signals) become first-class inputs that guide subsequent decisions. Encoding this reasoning via agents enables systematic reuse of expert heuristics that are otherwise applied informally.
Scope and Limitations
The case study uses LightGBM as the sample demo, but the architecture is generic and can be applied to any ML model. The talk explicitly discusses scenarios where agent-based optimization adds limited value or introduces unnecessary complexity.
Audience Takeaways
Attendees will gain:
- A blueprint for putting an LLM in any decision loop with guardrails
- If you do ML: a new way to think about HPO
- If you don't: a reusable pattern for agent-driven automation
I am a machine learning practitioner and former founder working across predictive modeling, computer vision, MLOps, and autonomous systems. After studying mechanical engineering, I worked in the electric vehicle development sector at Hyundai Motor Group, contributing to large-scale, safety-critical automotive systems.
I later founded and scaled an agtech startup from zero to a six-figure ARR business. This experience shaped my focus on building technology that delivers measurable, real-world value rather than chasing technical hype. After exiting, I transitioned into the e-commerce domain, applying machine learning to large-scale experimentation and operational optimization.
My background includes graduate research in robotics, published work in applied machine learning, and hands-on experience deploying end-to-end ML systems. I am particularly interested in explainability-driven optimization, agent-based workflows, and cross-disciplinary system design. I believe polymath practitioners—those who can bridge domains—will be especially valuable in the era of AI.