The Vaegirs migrated over the mountains into the Taiga and settled there, and the Khergits are steppe nomads who have a similar story I believe.

The Nords were promised land by the Emperor in exchange for their sailors but were unsatisfied, and formed their own Kingdom on Calradia amidst the fall of the Empire. IIRC the Sarranid land was never even part of the Empire. For starters, we can eliminate the Nords, Vaegirs, Khergit and Sarranids. Basically, looking to the lore of warband Calradia and then the information we have in the bannerlord era helps us answer this question.

This question has been asked sooooo many times on this sub lol.