Airline satisfaction prediction
A Kaggle competition with 700,000 airline passengers: predict who comes away satisfied. The leaderboard is crowded and the margins are tiny, so the work was less about picking a model and more about finding structure in the data that the models couldn't see on their own.
- Public AUC
- 0.9604
- Gain from features
- +0.0021
- Leaderboard
- #577
- Training rows
- 700k
The problem
Each row is a passenger: age, customer type, class, purpose of travel, flight distance, delays, and 13 service ratings from wifi to cleanliness. The target is whether they were satisfied, and entries are scored by ROC AUC, so what matters is ranking passengers correctly rather than the exact probabilities.
The data is synthetic, generated from a public airline survey dataset of about 130,000 passengers. That detail turned out to matter.
Baseline
I started with LightGBM, XGBoost and CatBoost on the same five cross-validation folds, plus a few row-level summaries such as the average rating and total delay, and blended the three by ranking their predictions. That scored 0.9589 in cross-validation and 0.9583 on the public leaderboard. The two numbers agreeing so closely was the most useful result: it meant I could trust cross-validation and stop spending submissions to test ideas.
What actually moved the score
- Flight distance is a route. It has only about 3,500 distinct values, each shared by hundreds of passengers. That makes it behave like a route identifier, not a number. For every route I added its size, its average ratings and delays, its mix of business and economy travellers, and how far each passenger's ratings sit from their route's average. A passenger rating the seats 3 on a route where everyone gives 4 tells you something a raw 3 doesn't.
- Target encoding without leakage. Satisfaction rates per route, per age, and per route combined with class and travel type. The trap is that these use the labels, so I computed them inside each cross-validation fold, with a further split for the training rows, so the validation score never gets to see its own answers.
- The original survey as a teacher. Other competitors had measured that adding the 130,000 original passengers as extra training rows hurts the score. Instead I trained a separate model on the original data alone and gave its prediction to the main models as a feature. That kept the original survey's signal without distorting the training set.
Results
| Version | Cross-validation AUC | Public leaderboard |
|---|---|---|
| Baseline blend | 0.95885 | 0.95834 (#925) |
| + route profile and target encoding | 0.96043 | not submitted |
| + original-data teacher | 0.96093 | 0.96042 (#577) |
Cross-validation and the leaderboard both moved by the same 0.0021, so the gain is real rather than tuned to the public test set. Ranks are from 10 October 2026, out of roughly 1,450 teams. The route and target-encoding ideas came from the competition's discussion forum, and the code is my own implementation.
One thing I chose not to do: many of the top public notebooks blend dozens of other people's published predictions. It moves you up the public leaderboard, but it teaches nothing and tends to fall apart when the private leaderboard is revealed.
What I'd do next
- Add a different kind of model. The three boosted models now score within 0.0001 of each other, so blending them adds almost nothing. A neural network or a tabular foundation model would make different mistakes, which is what a blend needs.
- Measure each feature block on its own, switching it off to see what it is really worth on the full data.
- Tune the boosted models once the feature set stops changing.