Handling Class Imbalance
Real-world text datasets almost always come imbalanced. Spam detection is 99% normal, 1% spam. Hate speech detection on most platforms is overwhelmingly non-toxic. Some intents in a customer support classifier appear in less than 1% of tickets.
You have three families of fixes:
Sampling. Undersample the majority class, oversample the minority class, or use SMOTE on word embeddings. (Regular SMOTE doesn't work on raw text — you'd be interpolating between strings. Apply it to dense embedding vectors instead.)
Loss function modifications. Weight the cross-entropy loss inversely proportional to class frequency. Square-root inverse frequency is a popular middle ground that doesn't over-correct on extreme imbalances.
Architectural approaches. Two-stage training: first train on a balanced subset to learn the task, then fine-tune on the full distribution. Ensembles: train multiple models on balanced subsets and combine predictions via weighted voting.
Pick Your Evaluation Metric First
Accuracy is useless for imbalanced datasets — a model that always predicts the majority class gets 99% accuracy on a 99/1 split. Use precision, recall, F1, or AUC-ROC depending on the relative cost of false positives vs. false negatives for your application. Decide this before you train.