Using XGBoost to construct a vertically aligned skill difficulty scale

Abstract:

From the abstract: We sought to construct a vertical difficulty scale for the skill library in Freckle, a widely used online mathematics practice system. The scale would span kindergarten through grade 8 and permit direct comparisons of skill difficulty across grade levels. We fitted an Extreme Gradient Boosting (XGBoost) prediction model to 3 million practice sessions covering over 1,800 mathematics skills. We applied the model as a counterfactual, fitting it to the full population of students to generate a population percent-correct score for each skill in adjacent grades. We linked adjacent grades using Thurstone’s scaling method and validated the resulting scale against independently derived Rasch-based skill difficulties from Star Math. The two scales were strongly associated (Pearson r = 0.80), supporting the use of large-scale practice data to estimate comparable skill difficulties across grades.

Citation: Bielinski, J. (2026, October 5–7). Using XGBoost to construct a vertically aligned skill difficulty scale [Conference presentation]. Artificial Intelligence in Measurement and Education Conference (AIME-Con 2026), Pittsburgh, PA, United States.

Publication Date:
10/05/2026