Quick answer: An XLNet text-classification pipeline needs a stable label mapping, compatible tokenizer and model configuration, leakage-resistant splits, a suitable classification head, and metrics that reflect the cost of each class. Save the full preprocessing and inference contract with the model.
XLNet is a transformer language model introduced in the 2019 paper XLNet: Generalized Autoregressive Pretraining for Language Understanding. For text classification, the usual workflow is to tokenize text, pass the tokens through an XLNet sequence-classification model, read the logits, and map the highest-scoring class back to a label.
XLNet is not the newest model family, but it remains useful for understanding transformer classification pipelines and for maintaining projects that already use XLNet checkpoints. In new systems, compare it against current encoder and instruction-tuned models before choosing it for production.
For classification tasks, focus on the full pipeline rather than the model name alone. Good results depend on clean labels, representative examples, consistent preprocessing, stable train and test splits, and a metric that matches the business cost of mistakes. A stronger architecture cannot fix unclear labels or data that does not match real user text.
XLNet also needs the matching tokenizer and checkpoint. Mixing a tokenizer from one model family with a checkpoint from another can produce poor results or shape errors. Keep tokenizer, model, label mapping, and preprocessing notes together so later predictions can be reproduced.
Primary references include the XLNet paper on arXiv, Hugging Face XLNet documentation, Transformers sequence classification guide, scikit-learn classification report documentation, and PyTorch no_grad documentation.
Create Label Mappings
Start by defining the class labels clearly. The model works with numeric class IDs, while the application usually needs readable labels.
labels = ["bug", "feature", "question"]
label_to_id = {label: index for index, label in enumerate(labels)}
id_to_label = {index: label for label, index in label_to_id.items()}
print(label_to_id)
print(id_to_label)
Keep these mappings with the trained model. If label order changes between training and prediction, evaluation results become misleading.
Tokenize Text For XLNet
Hugging Face Transformers provides tokenizers that prepare text in the format expected by each model checkpoint. Use padding and truncation so a batch has consistent shapes.
from transformers import AutoTokenizer
texts = [
"The login page crashes after submitting the form.",
"Please add CSV export to the dashboard.",
"How do I reset my password?",
]
tokenizer = AutoTokenizer.from_pretrained("xlnet-base-cased")
batch = tokenizer(texts, padding=True, truncation=True, max_length=128, return_tensors="pt")
print(batch.keys())
The tokenizer output usually includes token IDs and attention masks. Those tensors become the model input during training or prediction. Tokenization is the first boundary between raw text and XLNet inputs; Tokenize Strings in Python for NLP compares split(), regex, shlex, CSV, and NLP tokenizers.
Keep max_length tied to your data. Short support tickets may need a much smaller limit than legal documents or long reviews. Longer inputs cost more memory and time, so measure the length distribution before choosing a limit.
Load A Classification Model
For classification, use a sequence-classification head rather than the base language model alone. The head produces one score per label.
from transformers import AutoModelForSequenceClassification
model = AutoModelForSequenceClassification.from_pretrained(
"xlnet-base-cased",
num_labels=len(labels),
id2label=id_to_label,
label2id=label_to_id,
)
For real projects, fine-tune the model on labeled examples from your own task. A base checkpoint without task-specific training will not understand your custom labels.
Wrap Examples In A Dataset
Training code is easier to maintain when tokenized examples and labels are exposed through a dataset object.
import torch
class TextDataset(torch.utils.data.Dataset):
def __init__(self, encodings, target_ids):
self.encodings = encodings
self.target_ids = target_ids
def __len__(self):
return len(self.target_ids)
def __getitem__(self, index):
item = {key: value[index] for key, value in self.encodings.items()}
item["labels"] = torch.tensor(self.target_ids[index])
return item
This keeps training and evaluation code separate from data preparation. It also makes batching easier with PyTorch data loaders.
Predict Labels
During inference, turn off gradient tracking and convert the highest logit score into a label.
import torch
model.eval()
with torch.no_grad():
outputs = model(**batch)
predicted_ids = outputs.logits.argmax(dim=-1).tolist()
predicted_labels = [id_to_label[index] for index in predicted_ids]
print(predicted_labels)
Prediction code should keep preprocessing, model inference, and label mapping close together so the output is easy to audit.
Evaluate Predictions
After prediction, compare model labels with true labels. Precision, recall, and F1 are more informative than a single accuracy number for uneven class distributions.
from sklearn.metrics import classification_report
true_labels = ["bug", "feature", "question"]
predicted_labels = ["bug", "question", "question"]
print(classification_report(true_labels, predicted_labels, zero_division=0))
Review errors manually before changing the model. Mislabels may come from ambiguous text, incomplete labels, poor training data, or a mismatch between the label set and the real task.
Always inspect examples where the model is confident but wrong. Those cases often reveal overlapping labels, missing context, or text patterns that never appeared in training. Fixing those issues improves future experiments regardless of which model family you use.
Practical Checklist
Use XLNet for text classification when you have a reason to use that checkpoint family, such as an existing model, a comparison study, or a legacy pipeline. For new projects, benchmark it against current alternatives using the same train, validation, and test splits.
Keep label mappings stable, tokenize with the matching tokenizer, fine-tune on task-specific examples, and evaluate with metrics that show per-class behavior. Store the tokenizer, model, label mapping, and evaluation notes together so future predictions are reproducible.
If the model performs poorly, inspect the data before blaming XLNet. Classification quality often improves more from cleaner labels and better examples than from switching architectures.
Define Labels First
Give each class a stable integer ID and document its meaning. Keep the mapping with the trained artifact so a changed label order cannot silently turn correct predictions into the wrong business action.
Tokenize Consistently
Use the tokenizer associated with the selected XLNet checkpoint, define truncation and padding, and measure how often important text is cut off. Training and inference must share the same preprocessing rules.
Prevent Text Leakage
Duplicate documents, users, threads, and near-identical records can cross a random split. Group or time-based splitting may be required when the deployment unit is not an independent row.
Fine-Tune Deliberately
Control learning rate, sequence length, batch size, epochs, warmup, class weights, and checkpoint selection. Track the configuration and seed, but do not treat a seed as a substitute for robust validation.
Choose Useful Metrics
Accuracy can hide minority-class failures. Review per-class precision and recall, macro or weighted F1, confusion matrices, calibration where probabilities drive decisions, and examples of high-impact errors.
Package Inference
Persist tokenizer files, model configuration, label mappings, preprocessing, library versions, and threshold policy. Test loading the saved artifact on representative, empty, long, multilingual, and adversarial inputs.
Use the official Hugging Face Transformers documentation for model and tokenizer workflows, and the scikit-learn model-evaluation guide for classification metrics. Related Python Pool references include testing and array preparation.
For related NLP workflows, compare inference tests, tensor preparation, and label mappings before fine-tuning XLNet.
Frequently Asked Questions
What is XLNet used for?
XLNet is a Transformer language model that can be adapted to downstream NLP tasks such as text classification through a compatible tokenizer and classification head.
How should I split a text-classification dataset?
Create leakage-resistant training, validation, and test splits, especially when multiple rows come from the same user, document, or conversation.
Which metrics should I use for classification?
Choose metrics such as accuracy, precision, recall, F1, and a confusion matrix according to class balance and the cost of false positives and false negatives.
How do I make XLNet inference reproducible?
Persist the tokenizer, label mapping, model configuration, preprocessing rules, and library versions, then test representative inputs after loading the saved model.