Transformers like BERT and Transformer-XL are now top choices for language modeling, surpassing LSTM models. In a study focusing on Finnish, BERT achieved a perplexity score of 14.5, a first in this field. Transformer-XL outperformed even more, reaching a score of 73.58, which is 27% better than the previous LSTM model.