Explainable AI Part 2

Explainable AI Part 2#

Week 9 materials can be accessed here.

Today we will go through some more examples of Explainable AI (XAI) and our focus will be on the regression models we’ve talked about: Polynomial regression, Neural networks and Gaussian Processes.

Gaussian Processes#

Recall Gaussian Processes use several different types of kernels depending on the user’s choice, we’ve metioned RBF kernel, Matern kernel, nerual network kernel and periodic kernel and so on. The Automatic Relevance Determination (ARD) kernel is one specific type of kernel used within Gaussian Processes as well. It is particularly interesting because it can help provide feature importances in some way.

## ARD Kernel Explained The ARD kernel is a variation of the squared exponential (SE) kernel, which is also known as the radial basis function (RBF) kernel as we have introduced. The standard SE/RBF kernel uses a single length scale parameter to control how quickly the similarity between points decreases with distance. In contrast, the ARD kernel uses a separate length scale parameter for each input dimension (feature). This allows the GP to adjust the influence of each feature independently, leading to a more flexible model that can better capture the underlying structure of the data.

Feature Importance with ARD#

The concept of feature importances emerges naturally from the ARD kernel. Since each feature has its own length scale parameter, you can interpret the inverse of these length scale values as indicating the importance of the corresponding features: a shorter length scale means that the model is more sensitive to changes in that feature, implying that the feature is more important for predicting the output.

In practice, after fitting a Gaussian Process model with an ARD kernel to your data, you can examine the learned length scales to gain insights into which features are most important. This can be particularly useful for feature selection, understanding the data, and improving model interpretability.

User case#

We have tried to use GPs on predicting sea ice concentration (SIC) so now we test it based on that.

import numpy as np
from sklearn.model_selection import train_test_split
import matplotlib.pyplot as plt

features_path = '/content/drive/MyDrive/GEOL0069/Week_6/reshaped_array_condition_21.npy'
targets_path = '/content/drive/MyDrive/GEOL0069/Week_6/SICavg_condition.npy'

input_features = np.load(features_path)
target_variables = np.load(targets_path)

X_train, X_test, y_train, y_test = train_test_split(input_features, target_variables, test_size=0.2, random_state=42)

pip install Gpy

import GPy
import numpy as np
# Define a GP model with an ARD kernel
kernel = GPy.kern.RBF(input_dim=21, ARD=True)
num_inducing = 100
model = GPy.models.SparseGPRegression(X_train, y_train.reshape(-1, 1), kernel, num_inducing=num_inducing)

# Fit the model (optimize the hyperparameters)
model.optimize(messages=True)

# Retrieve the learned length scales
length_scales = model.kern.lengthscale

print(length_scales)

plt.figure(figsize=(12, 8))
# Plotting the inverse of the length scales to compare with feature importances directly
plt.bar(range(len(length_scales)), 1/np.abs(length_scales), align='center')
plt.xticks(range(len(length_scales)), ['Feature %d' % i for i in range(len(length_scales))], rotation=45)
plt.xlabel('Feature')
plt.ylabel('Inverse Length Scale')
plt.title('Inverse Feature Importance from Gaussian Process ARD Kernel')
plt.tight_layout()
plt.show()

Explainable AI in Polynomial Regression#

Polynomial regression extends linear regression by incorporating polynomial terms, making the model capable of capturing non-linear relationships between the independent variables and the dependent variable. While this increases the model’s flexibility and fit to complex datasets, it introduces challenges in interpreting the model, especially regarding the importance of each feature.

Feature Importance in Polynomial Regression#

In the context of polynomial regression, understanding feature importance becomes more complex than in linear regression. In a linear model, which can be considered a polynomial regression of degree 1, the coefficients associated with each feature directly reflect the feature’s importance. The magnitude and direction of each coefficient indicate how changes in a feature affect the target variable.

However, polynomial regression of a degree higher than 1 involves:

Linear terms: The original features.
Polynomial terms: Each feature raised to higher powers (e.g., squared, cubed).
Interaction terms: Combinations of different features.

This transformation means we cannot simply look at coefficients to gauge feature importance because each original feature contributes to the model in multiple, complex ways.

Assessing Feature Importance#

For polynomial models of degree higher than 1, permutation feature importance is a useful tool. It evaluates the model’s performance degradation when the values of each feature are shuffled. This degradation indicates the feature’s importance. However, it’s important to note that this method measures the importance of all terms derived from an original feature (polynomial, interaction terms) rather than the original feature itself.

Limitations and Acknowledgements#

It’s important to acknowledge a key limitation in extracting feature importance from polynomial regression models. Directly interpreting feature importance from the model’s coefficients is only straightforward for linear models (degree 1 polynomial regression). For models involving higher-degree polynomials, the interpretation of feature importance must consider the complexity introduced by polynomial and interaction terms.

While methods like permutation feature importance can provide insights, they do so for the transformed dataset rather than directly for the original features. This means that while we can understand the importance of the features in the context of the model, translating this back to the original features requires careful consideration and is not as direct as in linear regression.

Let’s now try to see if we can get feature importances in degree 1 polynomial regression (Linear regression).

from sklearn.preprocessing import PolynomialFeatures
from sklearn.linear_model import LinearRegression
import numpy as np
import matplotlib.pyplot as plt

# Assuming X_train and y_train are your training data and targets
polynomial_features = PolynomialFeatures(degree=1)
X_poly_train = polynomial_features.fit_transform(X_train)

# Fit a linear regression model
model_linear = LinearRegression()
model_linear.fit(X_poly_train, y_train)

# Get the coefficients as feature importances
feature_importances = model_linear.coef_

# Visualize the feature importances
plt.figure(figsize=(10, 6))

# Plotting the magnitude of the coefficients
plt.bar(range(len(feature_importances)), np.abs(feature_importances), align='center')
plt.xticks(range(len(feature_importances)), ['Feature %d' % i for i in range(len(feature_importances))], rotation=45)
plt.xlabel('Feature')
plt.ylabel('Coefficient Magnitude')
plt.title('Feature Importances from Linear Regression')
plt.tight_layout()
plt.show()

Neural Network#

For neural networks, what methods can you think of? Actually the sensitivity analysis we’ve adopted for CNNs can be used in the same way here. Please note that deep learning models like this are usually not as interepretable compared to probabilistic models like GPs and regression. However, we still can get some insights from it and it may infer the same thing as the other model suggests.

import numpy as np
import tensorflow as tf
from tensorflow.keras.layers import Dense
from tensorflow.keras.models import Sequential
import matplotlib.pyplot as plt


def create_model(input_shape):
    model = Sequential([
        Dense(256, activation='relu', input_shape=input_shape),
        Dense(256, activation='relu'),
        Dense(1)  
    ])
    return model


def sensitivity_analysis(model, input_data, class_idx):
    input_data_tensor = tf.convert_to_tensor(input_data, dtype=tf.float32)
    with tf.GradientTape() as tape:
        tape.watch(input_data_tensor)
        predictions = model(input_data_tensor, training=False)
        class_output = predictions[:, class_idx]

    gradients = tape.gradient(class_output, input_data_tensor)
    feature_sensitivity = tf.reduce_sum(tf.abs(gradients), axis=0)
    return feature_sensitivity.numpy()

num_models = 5  
ensemble_sensitivities = []

for i in range(num_models):
    model = create_model((X_train.shape[1],))  
    model.compile(optimizer='adam', loss='mean_squared_error')

    # Fit model
    model.fit(X_train, y_train, epochs=10, batch_size=32, verbose=0)  

    sample_data = X_test[:1]  
    class_idx = 0  

    feature_sensitivity = sensitivity_analysis(model, sample_data, class_idx)
    ensemble_sensitivities.append(feature_sensitivity)

average_sensitivity = np.mean(ensemble_sensitivities, axis=0)

plt.figure(figsize=(10, 6))
plt.bar(range(X_train.shape[1]), average_sensitivity)
plt.xlabel('Feature Number')
plt.ylabel('Average Sensitivity Score')
plt.title('Average Sensitivity of Prediction to Each Feature Across Ensemble')
plt.show()