Introduction: Explainable AI with SHAP and LIME in Python

Q: What is the difference between SHAP and LIME?

SHAP (SHapley Additive exPlanations) is based on game theory and gives consistent, additive feature-importance values for both global model behaviour and individual predictions. LIME (Local Interpretable Model-agnostic Explanations) approximates the model locally around one prediction with a simpler, interpretable model. SHAP is generally more theoretically grounded and better for global explanations; LIME is faster and works with any model type for local explanations.

Q: How do I install SHAP and LIME in Python?

Run pip install shap lime , then install the supporting libraries with pip install numpy pandas scikit-learn matplotlib seaborn 'xgboost<3' . Verify the install by importing both packages and printing shap.__version__ and importlib.metadata.version('lime') (lime does not expose a top-level __version__ ). Note that shap 0.49 is not yet compatible with xgboost 3.x, hence the version pin.

Q: Which SHAP explainer should I use for my model?

Use shap.TreeExplainer for tree-based models (Random Forest, XGBoost), shap.LinearExplainer for linear models such as Logistic Regression, and shap.KernelExplainer as a model-agnostic fallback for anything else (e.g. neural networks) — it is by far the slowest of the three. In modern SHAP (0.42+), calling the explainer directly, e.g. explainer(X) , returns an Explanation object; for scikit-learn binary classifiers the values have shape (n_samples, n_features, n_classes) , so select the positive class with explanation[..., 1] .

Q: Can I use LIME with any machine learning model?

Yes. LIME is model-agnostic — it only needs access to the model’s predict_proba (or predict ) method, so it works the same way whether the underlying model is a Random Forest, an SVM, or a neural network.

Black-box machine learning models can be difficult to trust and debug, especially in domains where decisions have significant consequences such as healthcare, finance, and criminal justice. A model that quietly denies someone a loan without anyone being able to say why is not just a technical problem — it is a governance problem.

Explainable AI (XAI) addresses this challenge by providing methods and tools to understand how AI models make decisions. Two of the most powerful and widely used libraries for model interpretability are SHAP (SHapley Additive exPlanations) and LIME (Local Interpretable Model-agnostic Explanations).

In this tutorial, I explore how to use SHAP and LIME in Python to make machine learning models transparent and interpretable. We will cover both global and local explanations, understand feature importance, and learn how to communicate model decisions effectively. Every code block below was executed and verified against shap 0.49.1, lime 0.2, scikit-learn 1.7.2, and xgboost 2.1.4 — worth mentioning because the SHAP API changed substantially around version 0.42, and a lot of older tutorials (including an earlier draft of this one) silently produce wrong results on current versions. I will point out those pitfalls as we go.

If you are new to the broader question of whether explainability is achievable at all, see my earlier post Explainable AI is possible for the conceptual groundwork this tutorial builds on.

Understanding Explainable AI

Why Explainability Matters

Explainable AI is crucial for several reasons:

Trust and Adoption: Users are more likely to trust and adopt AI systems they can understand
Regulatory Compliance: Many industries require explanations for AI decisions (the EU AI Act and GDPR’s provisions on automated decision-making being prominent examples)
Debugging and Improvement: Understanding model behaviour helps identify and fix issues, such as a model latching onto a leaky feature
Bias Detection: Explanations can reveal unfair biases in models before they reach production
Stakeholder Communication: Clear explanations help communicate with non-technical stakeholders

Types of Model Explanations

Global Explanations

Explain the overall behaviour of the model
Show which features are most important across all predictions
Help understand model behaviour at a high level

Local Explanations

Explain individual predictions
Show why a specific prediction was made
Help understand model behaviour for specific cases

SHAP covers both: aggregate its per-prediction values and you get a global view. LIME is local by design, though we will later see SubmodularPick, its attempt at a global summary.

Setting Up the Environment

Installing Required Libraries

# Install SHAP and LIME
pip install shap lime

# Install additional dependencies
# Note: shap 0.49 is not yet compatible with xgboost 3.x
# (TreeExplainer fails to parse the new base_score format),
# so pin xgboost below 3.
pip install numpy pandas scikit-learn matplotlib seaborn "xgboost<3"

Verifying Installation

import shap
import lime
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
import seaborn as sns
from importlib.metadata import version

print(f"SHAP version: {shap.__version__}")
# lime does not expose __version__ at the top level
print(f"LIME version: {version('lime')}")

# Set up plotting style
plt.style.use('seaborn-v0_8')
sns.set_palette("husl")

Understanding SHAP (SHapley Additive exPlanations)

SHAP is based on a concept from cooperative game theory. Imagine a team project where the payout must be split fairly among team members according to their actual contribution — including how well each person works alongside every possible combination of colleagues. The Shapley value, introduced by Lloyd Shapley in 1953, is the unique way of doing this split that satisfies a set of fairness axioms. SHAP treats each feature as a “player” and the model’s prediction as the “payout”.

Worth pausing on that lineage, because it is a strange one. Shapley’s 1953 paper had nothing to do with machine learning — it solved a fair cost-allocation problem in cooperative game theory, work for which he later shared the 2012 Nobel Memorial Prize in Economic Sciences. The idea sat outside ML for over half a century until Štrumbelj and Kononenko first proposed using Shapley values to explain individual classifier predictions, in 2010 and then with a Monte Carlo approximation in 2014 — and even then it barely caught on. It took Lundberg and Lee’s 2017 NeurIPS paper, A Unified Approach to Interpreting Model Predictions, to make it the default choice: they showed that LIME, DeepLIFT, and several other explanation methods are all special cases of one family of additive feature-attribution methods, and proposed SHAP values as the theoretically unique solution within that family. That paper — not the 1953 one — is why “SHAP” is now shorthand for the whole approach.

The Maths, Briefly

The Shapley value of feature i is its marginal contribution to the prediction, averaged over every possible subset S of the remaining features:

\[\phi_i = \sum_{S \subseteq F \setminus \{i\}} \frac{|S|! \, (|F| - |S| - 1)!}{|F|!} \big( f(S \cup \{i\}) - f(S) \big)\]

where F is the set of all features and f(S) is the model’s expected prediction when only the features in S are “present”. Two properties make this practically useful:

Additivity (local accuracy): the SHAP values for one prediction sum exactly to the difference between that prediction and the baseline: f(x) = base_value + Σ φ_i. We will verify this numerically below — it is a handy sanity check that you are reading the values correctly.
Consistency: if a model changes so that a feature contributes more, its SHAP value never decreases. LIME offers no such guarantee.

The catch is cost: the formula sums over 2^|F| subsets, which is hopeless beyond a handful of features. This is why SHAP ships specialised explainers. TreeSHAP exploits tree structure to compute exact values in polynomial time — O(T · L · D²) per instance, where T is the number of trees, L the maximum leaves, and D the maximum depth. KernelSHAP approximates the values for arbitrary models by sampling feature coalitions, which is why it is orders of magnitude slower.

Training a Model to Explain

import shap
import numpy as np
import pandas as pd
from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import train_test_split

# Generate sample data
np.random.seed(42)
n_samples = 1000
n_features = 5

# Create synthetic dataset
feature_names = [f'Feature_{i+1}' for i in range(n_features)]
X = pd.DataFrame(np.random.randn(n_samples, n_features), columns=feature_names)
y = (X['Feature_1'] + X['Feature_2'] * 2 + X['Feature_3'] * 0.5
     + np.random.randn(n_samples) * 0.1 > 0).astype(int)

# Split the data
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42
)

# Train a model
model = RandomForestClassifier(n_estimators=100, random_state=42)
model.fit(X_train, y_train)

print(f"Model accuracy: {model.score(X_test, y_test):.4f}")

A note on preprocessing: I have deliberately not standardised the features. Tree-based models are invariant to monotonic feature transformations, so scaling buys you nothing here — and it actively hurts explainability, because your stakeholders end up staring at rules like Blood_Pressure > 0.53 in standardised units rather than Blood_Pressure > 140 in mmHg. If your model genuinely needs scaling (linear models, neural networks), wrap it in a scikit-learn Pipeline and explain the pipeline’s predict_proba, keeping the raw features for display.

Basic SHAP Analysis with the Explanation API

Since SHAP 0.42 or so, the recommended pattern is to call the explainer directly and work with the returned shap.Explanation object, which bundles values, base values, the underlying data, and feature names:

# Create SHAP explainer
explainer = shap.TreeExplainer(model)

# Calling the explainer returns a shap.Explanation object
explanation = explainer(X_test)

# For scikit-learn binary classifiers the values are 3-dimensional:
# (n_samples, n_features, n_classes)
print(f"SHAP values shape: {explanation.values.shape}")  # (200, 5, 2)

# Select the positive class with an ellipsis slice
explanation_pos = explanation[..., 1]
print(f"Positive-class values shape: {explanation_pos.values.shape}")  # (200, 5)

# Verify the additivity property: base value + sum of SHAP values
# must equal the model's predicted probability, for every sample
predicted = model.predict_proba(X_test)[:, 1]
reconstructed = explanation_pos.values.sum(axis=1) + explanation_pos.base_values
print(f"Additivity holds: {np.allclose(reconstructed, predicted)}")  # True

A pitfall worth a paragraph. Older tutorials use explainer.shap_values(X_test) and then select the positive class with shap_values[1]. That worked when SHAP returned a Python list with one array per class. Modern SHAP returns a single 3D NumPy array instead, so shap_values[1] now silently selects the second sample rather than the second class — no error, plausible-looking plots, wrong analysis. If you maintain legacy code, the corrected indexing is shap_values[:, :, 1]; if you are writing new code, use the Explanation API as above. (Adding to the fun, XGBoost binary classifiers return 2D values in log-odds space — one column per feature, no class axis — because XGBoost’s trees output a single margin. Check values.ndim before slicing.)

SHAP Beeswarm (Summary) Plot

The beeswarm plot shows the distribution of SHAP values for each feature, with colour encoding the feature’s value:

# Global view: each dot is one sample's SHAP value for one feature
shap.plots.beeswarm(explanation_pos, show=False)
plt.title("SHAP Beeswarm Plot")
plt.tight_layout()
plt.show()

# Mean absolute SHAP value per feature — a cleaner bar-chart view
shap.plots.bar(explanation_pos, show=False)
plt.title("Mean |SHAP| per Feature")
plt.tight_layout()
plt.show()

Reading the beeswarm: features are ordered by overall importance; a long tail of red (high feature value) dots to the right means high values of that feature push predictions up. In our synthetic data, Feature_2 should dominate — we gave it double the weight when constructing y.

SHAP Force Plot for a Single Prediction

Force plots visualise a single prediction as a tug of war between features pushing the probability up and down. The default renderer is interactive JavaScript for notebooks; pass matplotlib=True to get a static figure in scripts:

shap.plots.force(explanation_pos[0], matplotlib=True, show=False)
plt.title("SHAP Force Plot for a Single Prediction")
plt.tight_layout()
plt.show()

SHAP Dependence (Scatter) Plots

Scatter plots show how a specific feature’s value relates to its SHAP value across the dataset:

# How does Feature_1 affect predictions?
shap.plots.scatter(explanation_pos[:, "Feature_1"], show=False)
plt.title("SHAP Scatter Plot for Feature_1")
plt.tight_layout()
plt.show()

# Colour by Feature_2 to inspect interactions
shap.plots.scatter(
    explanation_pos[:, "Feature_1"],
    color=explanation_pos[:, "Feature_2"],
    show=False
)
plt.title("SHAP Interaction: Feature_1 coloured by Feature_2")
plt.tight_layout()
plt.show()

If the vertical spread at a given Feature_1 value is explained by the colour gradient, the two features interact; if the colouring looks random, they do not.

Understanding LIME (Local Interpretable Model-agnostic Explanations)

LIME provides local explanations by approximating the model’s behaviour around a specific prediction using a simpler, interpretable model. Ribeiro, Singh, and Guestrin published it a year before SHAP existed, out of the University of Washington, and their KDD 2016 paper title — “Why Should I Trust You?” — became something of a rallying cry for the field. The recipe, from that original paper, is refreshingly direct:

Perturb the instance: generate a neighbourhood of synthetic samples (5,000 by default for tabular data) by sampling each feature from its training distribution.
Weight each synthetic sample by proximity to the original instance, using an exponential kernel — exp(−d²/kernel_width²), where kernel_width defaults to 0.75 × sqrt(n_features).
Fit a weighted sparse linear model (ridge regression by default) on the black-box model’s predictions for those samples, keeping only the top num_features features.

The linear model’s coefficients are the explanation. Because steps 1–3 involve random sampling, LIME explanations are not deterministic: run it twice and you may get somewhat different weights, particularly for weak features. Always set random_state for reproducibility, and treat small weights with suspicion.

Basic LIME Analysis

from lime.lime_tabular import LimeTabularExplainer

# Create LIME explainer — note it wants raw NumPy arrays
lime_explainer = LimeTabularExplainer(
    X_train.values,
    feature_names=feature_names,
    class_names=['Class 0', 'Class 1'],
    mode='classification',
    random_state=42
)

# Explain a single prediction
exp = lime_explainer.explain_instance(
    X_test.values[0],
    model.predict_proba,
    num_features=5
)

# Get explanation as a list of (rule, weight) pairs
print("LIME Explanation:")
for feature, weight in exp.as_list():
    print(f"{feature}: {weight:.4f}")

# Visualise: as_pyplot_figure() works everywhere;
# exp.show_in_notebook() only works inside Jupyter
fig = exp.as_pyplot_figure()
plt.tight_layout()
plt.show()

Notice that LIME reports rules rather than feature names — Feature_2 > 0.68 rather than just Feature_2. By default LimeTabularExplainer discretises continuous features into quartile bins, which makes explanations far easier for humans to parse (and is another argument for keeping features in their raw units).

LIME for Different Model Types

LIME is model-agnostic: it only needs predict_proba. Let’s compare explanations across four very different models:

from sklearn.linear_model import LogisticRegression
from sklearn.svm import SVC
import xgboost as xgb

# Train different models
models = {
    'Random Forest': RandomForestClassifier(n_estimators=100, random_state=42),
    'Logistic Regression': LogisticRegression(max_iter=1000, random_state=42),
    'SVM': SVC(probability=True, random_state=42),
    'XGBoost': xgb.XGBClassifier(random_state=42)
}

# Train all models
for name, clf in models.items():
    clf.fit(X_train, y_train)
    print(f"{name} accuracy: {clf.score(X_test, y_test):.4f}")

# Compare LIME explanations across models for the same instance
test_instance = X_test.values[0]
print(f"Test instance: {dict(zip(feature_names, np.round(test_instance, 3)))}")

for name, clf in models.items():
    print(f"\n{name} LIME Explanation:")
    exp = lime_explainer.explain_instance(
        test_instance,
        clf.predict_proba,
        num_features=3
    )
    for feature, weight in exp.as_list():
        print(f"  {feature}: {weight:.4f}")

Two practical notes. First, SVC(probability=True) calibrates probabilities via internal five-fold cross-validation (Platt scaling), which makes training noticeably slower — that is the price of giving LIME the predict_proba it needs. Second, if the four models broadly agree on which features matter, that convergence is itself evidence the signal is real rather than a modelling artefact.

Advanced SHAP Analysis

SHAP for Different Model Types

SHAP provides specialised explainers, each exploiting model structure for speed and exactness:

TreeExplainer — exact, fast, for tree ensembles (Random Forest, XGBoost, LightGBM). By default it uses the interventional perturbation mode against a background dataset, which has a cleaner causal interpretation; feature_perturbation='tree_path_dependent' uses the tree’s own cover statistics instead and needs no background data.
LinearExplainer — exact and essentially free for linear models; values are in the model’s margin (log-odds) space.
KernelExplainer — model-agnostic sampling approximation; works on anything with a prediction function but is the slowest by far. Always summarise the background data (e.g. shap.sample or shap.kmeans) — passing thousands of background rows multiplies runtime for no benefit.

A sizing note on interventional TreeExplainer: every value it computes involves swapping features against rows from the background dataset, so runtime scales roughly linearly with background size — a background of a few thousand rows will noticeably slow down explaining a large test set. A background that is too small or unrepresentative distorts the values instead, so the trade-off is real in both directions. A few hundred representative rows (or shap.sample(X_train, 100), as used later in this section) is usually enough; tree_path_dependent mode sidesteps the whole question by not using a background dataset at all.

from sklearn.neural_network import MLPClassifier

models['Neural Network'] = MLPClassifier(
    hidden_layer_sizes=(50, 25), max_iter=1000, random_state=42
)
models['Neural Network'].fit(X_train, y_train)

def positive_class_shap_values(name, clf, X_background, X_eval):
    """Compute SHAP values for the positive class, per model type."""
    if name in ('Random Forest', 'XGBoost'):
        values = shap.TreeExplainer(clf)(X_eval).values
    elif name == 'Logistic Regression':
        values = shap.LinearExplainer(clf, X_background)(X_eval).values
    else:  # model-agnostic fallback, e.g. Neural Network
        background = shap.sample(X_background, 100, random_state=42)
        values = shap.KernelExplainer(
            clf.predict_proba, background
        ).shap_values(X_eval, nsamples=200)
    # sklearn probability models give (n, features, classes);
    # XGBoost and linear margins give (n, features)
    if values.ndim == 3:
        values = values[..., 1]
    return values

X_eval = X_test.iloc[:50]
compare = {'Random Forest', 'Logistic Regression', 'Neural Network', 'XGBoost'}
mean_abs_shap = {
    name: np.abs(positive_class_shap_values(name, clf, X_train, X_eval)).mean(axis=0)
    for name, clf in models.items() if name in compare
}

# Grouped bar chart of mean |SHAP| per feature and model.
# summary_plot manages its own figure, so it cannot be placed
# in subplots — build the comparison manually instead.
importance_df = pd.DataFrame(mean_abs_shap, index=feature_names)
importance_df.plot.bar(figsize=(12, 6))
plt.ylabel("Mean |SHAP value|")
plt.title("Feature Importance Across Models")
plt.tight_layout()
plt.show()

One caveat on this comparison: the Random Forest values live in probability space, while the XGBoost and Logistic Regression values live in log-odds space. Compare feature rankings across models, not raw magnitudes.

SHAP Interaction Values

For tree models, SHAP can decompose predictions further into per-pair interaction effects. The output for a scikit-learn binary classifier has shape (n_samples, n_features, n_features, n_classes), so slice the class axis before plotting. Be warned that computation scales with the square of the feature count — budget accordingly on wide datasets:

rf_explainer = shap.TreeExplainer(models['Random Forest'])
interaction_values = rf_explainer.shap_interaction_values(X_test.iloc[:100])
print(f"Interaction values shape: {interaction_values.shape}")  # (100, 5, 5, 2)

# Plot interaction summary for the positive class
shap.summary_plot(
    interaction_values[:, :, :, 1],
    X_test.iloc[:100],
    show=False
)
plt.title("SHAP Interaction Values")
plt.tight_layout()
plt.show()

The diagonal entries are each feature’s main effect; off-diagonal entries capture pairwise interactions, and each row of interaction values still sums to the feature’s ordinary SHAP value — additivity all the way down.

SHAP Waterfall Plots

Waterfall plots show how each feature moves a single prediction away from the baseline, one step at a time — my preferred plot for explaining an individual decision to a non-technical audience:

rf_explanation = shap.TreeExplainer(models['Random Forest'])(X_test)

# Waterfall for the first test instance, positive class
shap.plots.waterfall(rf_explanation[..., 1][0], show=False)
plt.title("SHAP Waterfall Plot")
plt.tight_layout()
plt.show()

Advanced LIME Analysis

Discretisation and Multiple Instances

LIME’s discretiser controls how continuous features become human-readable rules:

from lime.lime_tabular import LimeTabularExplainer
from lime import submodular_pick

# Explainer with explicit discretisation settings
lime_explainer = LimeTabularExplainer(
    X_train.values,
    feature_names=feature_names,
    class_names=['Class 0', 'Class 1'],
    mode='classification',
    discretize_continuous=True,
    discretizer='quartile',   # also: 'decile', 'entropy'
    random_state=42
)

# Explain multiple instances
for i, instance in enumerate(X_test.values[:5]):
    exp = lime_explainer.explain_instance(
        instance,
        models['Random Forest'].predict_proba,
        num_features=5
    )
    print(f"Instance {i+1} explanation:")
    for feature, weight in exp.as_list():
        print(f"  {feature}: {weight:.4f}")

Submodular Pick: a Global View from Local Explanations

SubmodularPick selects a small, diverse set of instances whose explanations jointly cover the model’s behaviour — LIME’s answer to “show me a handful of representative examples”. Two API quirks to know: sp_explanations is a plain list of Explanation objects, and because SubmodularPick generates explanations with top_labels, each explanation only carries the predicted label — calling exp.as_list() with its default label=1 raises a KeyError whenever the predicted class is 0, so ask each explanation which label it has:

sp_obj = submodular_pick.SubmodularPick(
    lime_explainer,
    X_test.values[:20],
    models['Random Forest'].predict_proba,
    sample_size=20,
    num_features=5,
    num_exps_desired=3
)

print("Submodular Pick Results:")
for i, exp in enumerate(sp_obj.sp_explanations, 1):
    label = exp.available_labels()[0]
    print(f"\nExplanation {i} (for class {label}):")
    for feature, weight in exp.as_list(label=label):
        print(f"  {feature}: {weight:.4f}")

Choosing the Distance Metric

The proximity weighting in step 2 of LIME’s recipe depends on a distance metric. This is configured per explanation via the distance_metric argument of explain_instance, which accepts any metric name understood by scikit-learn’s pairwise_distances ('euclidean', 'manhattan', 'cosine', …). A common misconception — found in several older tutorials — is that a custom distance callable can be passed to the LimeTabularExplainer constructor; it has no such parameter, and code doing so simply raises a TypeError.

instance = X_test.values[0]

exp_euclidean = lime_explainer.explain_instance(
    instance,
    models['Random Forest'].predict_proba,
    num_features=5,
    distance_metric='euclidean'   # the default
)

exp_manhattan = lime_explainer.explain_instance(
    instance,
    models['Random Forest'].predict_proba,
    num_features=5,
    distance_metric='manhattan'
)

print("Euclidean-distance LIME Explanation:")
for feature, weight in exp_euclidean.as_list():
    print(f"  {feature}: {weight:.4f}")

print("\nManhattan-distance LIME Explanation:")
for feature, weight in exp_manhattan.as_list():
    print(f"  {feature}: {weight:.4f}")

Manhattan distance is less sensitive to single-feature outliers when defining the neighbourhood; if the two metrics produce materially different explanations, that is a sign the local approximation is fragile for this instance.

Real-World Applications

Healthcare: Medical Diagnosis Model

Let’s create a realistic healthcare scenario. Because we skip scaling, every explanation below is expressed in clinical units a doctor would actually recognise:

import numpy as np
import pandas as pd
from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import train_test_split

# Create synthetic medical data
np.random.seed(42)
n_samples = 1000

medical_data = pd.DataFrame({
    'Age': np.random.normal(50, 15, n_samples),
    'Blood_Pressure': np.random.normal(120, 20, n_samples),
    'Cholesterol': np.random.normal(200, 40, n_samples),
    'Blood_Sugar': np.random.normal(100, 20, n_samples),
    'BMI': np.random.normal(25, 5, n_samples),
})

# Create target (heart disease risk) using clinical thresholds
heart_disease_risk = (
    (medical_data['Age'] > 60) * 0.3 +
    (medical_data['Blood_Pressure'] > 140) * 0.4 +
    (medical_data['Cholesterol'] > 240) * 0.3 +
    (medical_data['Blood_Sugar'] > 126) * 0.2 +
    (medical_data['BMI'] > 30) * 0.2 +
    np.random.random(n_samples) * 0.1
) > 0.5

# Split data — no scaling needed for a Random Forest
X_train_med, X_test_med, y_train_med, y_test_med = train_test_split(
    medical_data, heart_disease_risk, test_size=0.2, random_state=42
)

medical_model = RandomForestClassifier(n_estimators=100, random_state=42)
medical_model.fit(X_train_med, y_train_med)
print(f"Medical model accuracy: {medical_model.score(X_test_med, y_test_med):.4f}")

# SHAP analysis for the medical model
medical_explainer = shap.TreeExplainer(medical_model)
medical_explanation = medical_explainer(X_test_med)[..., 1]

# Global view: which clinical factors drive risk?
shap.plots.beeswarm(medical_explanation, show=False)
plt.title("SHAP Beeswarm — Heart Disease Risk Model")
plt.tight_layout()
plt.show()

# Explain a specific patient
patient_idx = 0
patient_data = X_test_med.iloc[patient_idx]
patient_prediction = medical_model.predict_proba(
    X_test_med.iloc[[patient_idx]]
)[0][1]

print("\nPatient Data:")
for feature, value in patient_data.items():
    print(f"  {feature}: {value:.2f}")
print(f"\nHeart Disease Risk Prediction: {patient_prediction:.4f}")

# Waterfall plot for this patient — readable clinical units throughout
shap.plots.waterfall(medical_explanation[patient_idx], show=False)
plt.title("SHAP Waterfall — Individual Patient")
plt.tight_layout()
plt.show()

The waterfall now reads like a clinical note: “blood pressure of 152 mmHg added 0.18 to this patient’s risk” is a sentence a cardiologist can argue with, which is precisely the point.

Finance: Credit Risk Assessment

# Create synthetic financial data
np.random.seed(42)
n_samples = 1000

financial_data = pd.DataFrame({
    'Income': np.random.lognormal(10, 0.5, n_samples),
    'Credit_Score': np.random.normal(700, 100, n_samples),
    'Debt_Ratio': np.random.beta(2, 5, n_samples) * 2,
    'Payment_History': np.random.normal(0, 1, n_samples),
    'Employment_Length': np.random.exponential(5, n_samples),
})

# Create target (credit default risk)
default_risk = (
    (financial_data['Income'] < 50000) * 0.4 +
    (financial_data['Credit_Score'] < 600) * 0.5 +
    (financial_data['Debt_Ratio'] > 1.5) * 0.3 +
    (financial_data['Payment_History'] < -1) * 0.4 +
    (financial_data['Employment_Length'] < 2) * 0.2 +
    np.random.random(n_samples) * 0.1
) > 0.5

X_train_fin, X_test_fin, y_train_fin, y_test_fin = train_test_split(
    financial_data, default_risk, test_size=0.2, random_state=42
)

financial_model = RandomForestClassifier(n_estimators=100, random_state=42)
financial_model.fit(X_train_fin, y_train_fin)
print(f"Financial model accuracy: {financial_model.score(X_test_fin, y_test_fin):.4f}")

# LIME analysis for the financial model
from lime.lime_tabular import LimeTabularExplainer

financial_explainer = LimeTabularExplainer(
    X_train_fin.values,
    feature_names=list(financial_data.columns),
    class_names=['Low Risk', 'High Risk'],
    mode='classification',
    random_state=42
)

# Explain a high-risk applicant
high_risk_idx = np.where(y_test_fin.values)[0][0]
high_risk_applicant = X_test_fin.iloc[high_risk_idx]

print("\nHigh-Risk Applicant Data:")
for feature, value in high_risk_applicant.items():
    print(f"  {feature}: {value:.2f}")

exp = financial_explainer.explain_instance(
    X_test_fin.values[high_risk_idx],
    financial_model.predict_proba,
    num_features=5
)

print("\nLIME Explanation for High-Risk Applicant:")
for feature, weight in exp.as_list():
    print(f"  {feature}: {weight:.4f}")

Because the features are unscaled, LIME produces rules such as Credit_Score <= 632.47 — directly actionable in an adverse-action letter, which regulations like the US Equal Credit Opportunity Act effectively require.

Best Practices for Explainable AI

1. Choose the Right Tool

def choose_explanation_tool(model_type, explanation_type):
    """Choose an appropriate explanation tool based on model and explanation type."""

    if explanation_type == "global":
        if model_type in ("tree", "ensemble"):
            return "SHAP TreeExplainer"
        elif model_type == "linear":
            return "SHAP LinearExplainer"
        else:
            return "SHAP KernelExplainer (with a sampled background set)"

    elif explanation_type == "local":
        return "LIME, or SHAP waterfall/force plots"

    elif explanation_type == "interaction":
        if model_type in ("tree", "ensemble"):
            return "SHAP Interaction Values"
        else:
            return "SHAP KernelExplainer"

    return "LIME"  # Default for unknown cases

print(choose_explanation_tool("tree", "global"))

2. Validate Explanation Stability

LIME’s sampling makes its explanations stochastic, so before trusting them, measure how stable they are. The function below aggregates absolute feature weights across many instances using exp.as_map() — which returns feature indices rather than rule strings, avoiding the trap of grouping by discretised rules that differ between instances — and reports the coefficient of variation per feature:

def lime_stability(explainer, clf, X_eval, num_features=5, n_instances=50):
    """Coefficient of variation of absolute LIME weights per feature."""
    n_cols = X_eval.shape[1]
    importances = {j: [] for j in range(n_cols)}

    for i in range(min(n_instances, len(X_eval))):
        exp = explainer.explain_instance(
            X_eval[i], clf.predict_proba, num_features=num_features,
            labels=(1,),  # explicit, though it matches LIME's own default —
                          # explain_instance() always targets class 1 unless
                          # you pass top_labels, regardless of which class
                          # was actually predicted, so this never KeyErrors
        )
        # as_map() keys by label; values are (feature_index, weight) pairs
        for feature_idx, weight in exp.as_map()[1]:
            importances[feature_idx].append(abs(weight))

    scores = {}
    for j, values in importances.items():
        if len(values) > 1 and np.mean(values) > 0:
            scores[feature_names[j]] = np.std(values) / np.mean(values)
    return scores

stability = lime_stability(
    lime_explainer, models['Random Forest'], X_test.values[:50]
)

print("Explanation stability (coefficient of variation, lower is better):")
for feature, cv in sorted(stability.items(), key=lambda kv: kv[1]):
    print(f"  {feature}: {cv:.4f}")

High variation on a supposedly important feature is a red flag: either the local approximation is poor, or the feature’s effect is highly context-dependent. SHAP values for the same model are deterministic (for TreeExplainer), which is one reason to cross-check the two tools against each other.

3. Communicate Results Effectively

def create_explanation_report(clf, explainer, instance, feature_names,
                              target_names=None):
    """Create a plain-text explanation report for one prediction."""

    if target_names is None:
        target_names = ['Class 0', 'Class 1']

    prediction = clf.predict_proba(instance.reshape(1, -1))[0]
    predicted_class = int(np.argmax(prediction))

    if hasattr(explainer, 'explain_instance'):
        # LIME
        exp = explainer.explain_instance(
            instance, clf.predict_proba, num_features=len(feature_names)
        )
        explanation = exp.as_list()
    else:
        # SHAP (modern Explanation API)
        shap_exp = explainer(instance.reshape(1, -1))
        values = shap_exp.values[0]
        if values.ndim == 2:          # (features, classes) -> positive class
            values = values[:, 1]
        explanation = sorted(
            zip(feature_names, values), key=lambda x: abs(x[1]), reverse=True
        )

    report = (
        "EXPLANATION REPORT\n"
        "==================\n\n"
        f"Prediction: {target_names[predicted_class]} "
        f"(Confidence: {prediction[predicted_class]:.4f})\n\n"
        "Top Contributing Features:\n"
    )

    for i, (feature, importance) in enumerate(explanation[:5], 1):
        direction = "increases" if importance > 0 else "decreases"
        report += f"{i}. {feature}: {importance:.4f} ({direction} prediction)\n"

    report += "\nModel Confidence by Class:\n"
    for class_name, conf in zip(target_names, prediction):
        report += f"  {class_name}: {conf:.4f}\n"

    return report

report = create_explanation_report(
    models['Random Forest'],
    shap.TreeExplainer(models['Random Forest']),
    X_test.values[0],
    feature_names,
    ['Low Risk', 'High Risk']
)
print(report)

4. Handle Edge Cases

Explanations occasionally fail — KernelExplainer with a degenerate background set, LIME on an instance far outside the training distribution. Wrap production explanation calls defensively and log failures rather than letting a reporting pipeline take the whole service down:

def robust_explanation(clf, explainer, instance, feature_names, max_attempts=3):
    """Explanation with basic retry and error handling."""

    for attempt in range(max_attempts):
        try:
            if hasattr(explainer, 'explain_instance'):
                # LIME
                exp = explainer.explain_instance(
                    instance, clf.predict_proba, num_features=5
                )
                return exp.as_list()
            else:
                # SHAP (modern Explanation API)
                shap_exp = explainer(instance.reshape(1, -1))
                values = shap_exp.values[0]
                if values.ndim == 2:
                    values = values[:, 1]
                return sorted(
                    zip(feature_names, values),
                    key=lambda x: abs(x[1]), reverse=True
                )
        except Exception as e:
            print(f"Explanation attempt {attempt + 1} failed: {e}")

    return [("Error", 0.0)]

robust_exp = robust_explanation(
    models['Random Forest'],
    lime_explainer,
    X_test.values[0],
    feature_names
)
print("Robust explanation:", robust_exp)

An Explainability Workflow

Pulling the pieces together, this is the order I follow on real projects:

Start with global feature importance (SHAP beeswarm or bar plot) as a model sanity check — a leaked identifier or proxy feature at the top of the list is the cheapest bug you will ever catch.
Use local explanations for disputed predictions — waterfall plots for internal review, LIME rules for customer-facing wording.
Compare explanations across data slices (by demographic group, region, or time period) for bias detection; a feature that matters only for one subgroup deserves scrutiny.
Pair explanations with confidence and error analysis — an explanation of a wrong, overconfident prediction is where the interesting debugging lives.
Store explanation artefacts (values, plots, library versions) alongside model versions for audits; regulators ask “why did the model say that, then?”, not “what would it say now?”.

Are These Methods Actually Used in Production?

Yes, and increasingly because regulators require it rather than because teams find it fun. AWS SageMaker Clarify runs a distributed KernelSHAP approximation, Google Vertex AI’s Explainable AI applies SHAP to tabular models, and Azure’s Responsible AI dashboard bundles comparable tooling — all three default to SHAP for structured data. As of writing, the SHAP library itself is at 0.51.0 (released March 2026); LIME remains actively maintained too. The pattern that has actually stuck in production is not explaining every prediction but sampling a slice of daily traffic, storing the SHAP values alongside the predictions, and watching for drift — if a feature’s SHAP distribution shifts week over week, the model’s relationship with that feature is changing, often before accuracy metrics notice.

The regulatory push behind this is worth tracking as it unfolds rather than citing as settled fact, because it keeps moving. My colleague Cláudia covered the original timeline in Regulation on artificial intelligence has already been published: the EU AI Act’s high-risk obligations — including the human-oversight and transparency requirements that make SHAP and LIME relevant to compliance teams in the first place — were due to apply from 2 August 2026. That date has since moved. At the end of June 2026, the Council of the EU and the European Parliament provisionally agreed to a “Digital Omnibus” simplification package deferring the high-risk compliance deadline to 2 December 2027, pending formal publication in the Official Journal. Whether that holds is genuinely open — EU AI legislation has slipped before, and the text still needs to clear formal adoption — so treat any specific date here, including this one, as provisional rather than settled. What is not moving is the underlying requirement: a high-risk system’s outputs need to be interpretable by a human reviewer, and SHAP or LIME remain the most common way vendors satisfy that. For the deeper question of whether explainability is achievable at all, see Explainable AI is possible.

Conclusion: SHAP and LIME for Trustworthy Machine Learning

SHAP and LIME are complementary rather than competing tools. SHAP offers theoretical guarantees — additivity, consistency, determinism for tree models — and covers both global and local explanations, at the cost of being tied to specific explainer implementations and, for KernelSHAP, considerable compute. LIME is quick, model-agnostic, and produces human-readable rules, at the cost of sampling instability that you should measure rather than assume away. In practice I reach for SHAP first and use LIME as a cross-check and as the source of plain-language rules for non-technical audiences.

Two engineering lessons from this post are worth restating. Keep features in their natural units whenever the model allows it — explanations exist to be read by humans, and nobody thinks in standard deviations of cholesterol. And pin your library versions: the shift to SHAP’s 3D output arrays quietly broke a generation of shap_values[1] tutorials, and shap 0.49 does not yet speak xgboost 3.x.

As AI systems are deployed in more sensitive domains, the ability to explain and interpret model decisions matters more. SHAP and LIME provide a solid foundation for building transparent AI systems that users can understand and trust. For the broader question of whether explainability is achievable at all — and its limits — see Explainable AI is possible.

SHAP and LIME Explainable AI FAQ

What is the difference between SHAP and LIME?

SHAP (SHapley Additive exPlanations) is based on game theory and gives consistent, additive feature-importance values for both global model behaviour and individual predictions. LIME (Local Interpretable Model-agnostic Explanations) approximates the model locally around one prediction with a simpler, interpretable model. SHAP is generally more theoretically grounded and better for global explanations; LIME is faster and works with any model type for local explanations.

How do I install SHAP and LIME in Python?

Run pip install shap lime, then install the supporting libraries with pip install numpy pandas scikit-learn matplotlib seaborn 'xgboost<3'. Verify the install by importing both packages and printing shap.__version__ and importlib.metadata.version('lime') (lime does not expose a top-level __version__). Note that shap 0.49 is not yet compatible with xgboost 3.x, hence the version pin.

Which SHAP explainer should I use for my model?

Use shap.TreeExplainer for tree-based models (Random Forest, XGBoost), shap.LinearExplainer for linear models such as Logistic Regression, and shap.KernelExplainer as a model-agnostic fallback for anything else (e.g. neural networks) — it is by far the slowest of the three. In modern SHAP (0.42+), calling the explainer directly, e.g. explainer(X), returns an Explanation object; for scikit-learn binary classifiers the values have shape (n_samples, n_features, n_classes), so select the positive class with explanation[..., 1].

Can I use LIME with any machine learning model?

Yes. LIME is model-agnostic — it only needs access to the model’s predict_proba (or predict) method, so it works the same way whether the underlying model is a Random Forest, an SVM, or a neural network.

Explainable AI with Python: SHAP and LIME for Model Transparency

📚 This post is part of the "Applied ML & Model Engineering" series

Introduction: Explainable AI with SHAP and LIME in Python

Understanding Explainable AI

Why Explainability Matters

Types of Model Explanations

Global Explanations

Local Explanations

Setting Up the Environment

Installing Required Libraries

Verifying Installation

Understanding SHAP (SHapley Additive exPlanations)

The Maths, Briefly

Training a Model to Explain

Basic SHAP Analysis with the Explanation API

SHAP Beeswarm (Summary) Plot

SHAP Force Plot for a Single Prediction

SHAP Dependence (Scatter) Plots

Understanding LIME (Local Interpretable Model-agnostic Explanations)

Basic LIME Analysis

LIME for Different Model Types

Advanced SHAP Analysis

SHAP for Different Model Types

SHAP Interaction Values

SHAP Waterfall Plots

Advanced LIME Analysis

Discretisation and Multiple Instances

Submodular Pick: a Global View from Local Explanations

Choosing the Distance Metric

Real-World Applications

Healthcare: Medical Diagnosis Model

Finance: Credit Risk Assessment

Best Practices for Explainable AI

1. Choose the Right Tool

2. Validate Explanation Stability

3. Communicate Results Effectively

4. Handle Edge Cases

An Explainability Workflow

Are These Methods Actually Used in Production?

Conclusion: SHAP and LIME for Trustworthy Machine Learning

SHAP and LIME Explainable AI FAQ

What is the difference between SHAP and LIME?

How do I install SHAP and LIME in Python?

Which SHAP explainer should I use for my model?

Can I use LIME with any machine learning model?

References

Enjoyed this? Get more like it.

References

Citation

Learn AI and Python without the hype