XLNet is a transformer language model introduced in the 2019 paper XLNet: Generalized Autoregressive Pretraining for Language Understanding. For text classification, the usual workflow is to tokenize text, pass the tokens through an XLNet sequence-classification model, read the logits, and map the highest-scoring class back to a label.
XLNet is not the newest model family, but it remains useful for understanding transformer classification pipelines and for maintaining projects that already use XLNet checkpoints. In new systems, compare it against current encoder and instruction-tuned models before choosing it for production.
For classification tasks, focus on the full pipeline rather than the model name alone. Good results depend on clean labels, representative examples, consistent preprocessing, stable train and test splits, and a metric that matches the business cost of mistakes. A stronger architecture cannot fix unclear labels or data that does not match real user text.
XLNet also needs the matching tokenizer and checkpoint. Mixing a tokenizer from one model family with a checkpoint from another can produce poor results or shape errors. Keep tokenizer, model, label mapping, and preprocessing notes together so later predictions can be reproduced.
Primary references include the XLNet paper on arXiv, Hugging Face XLNet documentation, Transformers sequence classification guide, scikit-learn classification report documentation, and PyTorch no_grad documentation. Related PythonPool guides cover invalid tokens in Python, requests JSON, Python key-value pairs, Python syntax checking, and LLM chains.
Create Label Mappings
Start by defining the class labels clearly. The model works with numeric class IDs, while the application usually needs readable labels.
labels = ["bug", "feature", "question"]
label_to_id = {label: index for index, label in enumerate(labels)}
id_to_label = {index: label for label, index in label_to_id.items()}
print(label_to_id)
print(id_to_label)
Keep these mappings with the trained model. If label order changes between training and prediction, evaluation results become misleading.
Tokenize Text For XLNet
Hugging Face Transformers provides tokenizers that prepare text in the format expected by each model checkpoint. Use padding and truncation so a batch has consistent shapes.
from transformers import AutoTokenizer
texts = [
"The login page crashes after submitting the form.",
"Please add CSV export to the dashboard.",
"How do I reset my password?",
]
tokenizer = AutoTokenizer.from_pretrained("xlnet-base-cased")
batch = tokenizer(texts, padding=True, truncation=True, max_length=128, return_tensors="pt")
print(batch.keys())
The tokenizer output usually includes token IDs and attention masks. Those tensors become the model input during training or prediction.
Keep max_length tied to your data. Short support tickets may need a much smaller limit than legal documents or long reviews. Longer inputs cost more memory and time, so measure the length distribution before choosing a limit.
Load A Classification Model
For classification, use a sequence-classification head rather than the base language model alone. The head produces one score per label.
from transformers import AutoModelForSequenceClassification
model = AutoModelForSequenceClassification.from_pretrained(
"xlnet-base-cased",
num_labels=len(labels),
id2label=id_to_label,
label2id=label_to_id,
)
For real projects, fine-tune the model on labeled examples from your own task. A base checkpoint without task-specific training will not understand your custom labels.
Wrap Examples In A Dataset
Training code is easier to maintain when tokenized examples and labels are exposed through a dataset object.
import torch
class TextDataset(torch.utils.data.Dataset):
def __init__(self, encodings, target_ids):
self.encodings = encodings
self.target_ids = target_ids
def __len__(self):
return len(self.target_ids)
def __getitem__(self, index):
item = {key: value[index] for key, value in self.encodings.items()}
item["labels"] = torch.tensor(self.target_ids[index])
return item
This keeps training and evaluation code separate from data preparation. It also makes batching easier with PyTorch data loaders.
Predict Labels
During inference, turn off gradient tracking and convert the highest logit score into a label.
import torch
model.eval()
with torch.no_grad():
outputs = model(**batch)
predicted_ids = outputs.logits.argmax(dim=-1).tolist()
predicted_labels = [id_to_label[index] for index in predicted_ids]
print(predicted_labels)
Prediction code should keep preprocessing, model inference, and label mapping close together so the output is easy to audit.
Evaluate Predictions
After prediction, compare model labels with true labels. Precision, recall, and F1 are more informative than a single accuracy number for uneven class distributions.
from sklearn.metrics import classification_report
true_labels = ["bug", "feature", "question"]
predicted_labels = ["bug", "question", "question"]
print(classification_report(true_labels, predicted_labels, zero_division=0))
Review errors manually before changing the model. Mislabels may come from ambiguous text, incomplete labels, poor training data, or a mismatch between the label set and the real task.
Always inspect examples where the model is confident but wrong. Those cases often reveal overlapping labels, missing context, or text patterns that never appeared in training. Fixing those issues improves future experiments regardless of which model family you use.
Practical Checklist
Use XLNet for text classification when you have a reason to use that checkpoint family, such as an existing model, a comparison study, or a legacy pipeline. For new projects, benchmark it against current alternatives using the same train, validation, and test splits.
Keep label mappings stable, tokenize with the matching tokenizer, fine-tune on task-specific examples, and evaluate with metrics that show per-class behavior. Store the tokenizer, model, label mapping, and evaluation notes together so future predictions are reproducible.
If the model performs poorly, inspect the data before blaming XLNet. Classification quality often improves more from cleaner labels and better examples than from switching architectures.