/ 2020 KDD Tutorial

KDD2020 Tutorial on Interpreting and Explaining Deep Neural Networks: A Perspective on Time Series Data

1:00 - 4:00 PM, 23rd, August 2020

Abstract Program Tutors Prerequisites Reference

Introduction

Explainable and interpretable machine learning models and algorithms are important topics which have received growing attention from research, application and administration. Many advanced Deep Neural Networks (DNNs) are often perceived as black-boxes. Researchers would like to be able to interpret what the DNN has learned in order to identify biases and failure models and improve models. In this tutorial, we will provide a comprehensive overview on methods to analyze deep neural networks and an insight how those XAI methods help us understand time series data.

Program

Tutorial outline

Time Tutor Detail
50 min Jaesik Choi
(KAIST) 		Overview of Interpreting Deep Neural Networks
50 min Interpreting Inside of Deep Neural Networks
50 min Explaining TimeSeries Data

Overview of Interpreting Deep Neural Networks

At first, we will provide a comprehensive to explain fundamental principles in interpreting and explaining deep neural networks. The topic of the talk may include relevance score based methods [Layer Relevance Propagation (LRP), Deep Taylor Decomposition (DTD), PatternNet, and RAP] and gradient based methods [DeConvNet, DeepLIFT, Guided Backprop]. We will also present a new perspective by presenting methods on Meta-Explanations [CleverHans] and Neuralization methods [ClusterExplanations].

Interpreting Inside of Deep Neural Networks

In this section, we will present methods to interpreting internal nodes of deep neural networks. We will start the session by introducing method to visualize channels of Convolutional Neural Networks [Network Dissection] and Generative Adversarial Networks [GAN Dissection]. Then, We will present methods [Convex Hull, Cluster Explanations, E-GBAS]to analyze internal nodes of DNNs by analyzing a set of decision boundaries. When allowed, We will briefly overview the methods to explain DNNs by using attention methods.

Explaining Time Series Data

In the last section, we will introduce recent explainable methods on Time Series Domain. We will introduce [N-BEATS], a framework performing regression tasks and providing outputs that are interpretable without considerable loss in accuracy. [CPHAP] interprets the decision of temporal neural networks by extracting highly activated periods. Furthermore, this clustered results of this method provide an intuition of understanding data mining. Finally, we will present a method [Automatic Statistician] that predicts time series with a human-readable report, including the reason for prediction.

Tutors

Jaesik Choi is a director of the Explainable Artificial Intelligence Center of Korea since 2017. He is an associate professor of Graduate School of Artificial Intelligence at Korea Advanced Institute of Science and Technology (KAIST). He received his BS degree in computer science and engineering at Seoul National University in 2004. He received his PhD degree in computer science at the University of Illinois at Urbana-Champaign in 2012.

Email: jaesik.choi@kaist.ac.kr

Homepage: http://sailab.kaist.ac.kr/jaesik

Affiliation: Korea Advanced Institute of Science and Technology (KAIST)

Address: 8, Seongnam-daero 331,18F KINS Tower,Bundang, Seongnam, Gyeonggi, Republic of Korea

Prerequisites

Target Audience: Target audience of this tutorial is a general audience who are familiarized in basics of machine learning and statistics. Thus, a person with knowledge of master-level graduate student will not have difficulty to follow the tutorial.

Equipment attendees should bring : Nothing

Reference

1. Gunning, D. (2017). Explainable artificial intelligence (xai). Defense Advanced Research Projects Agency (DARPA).

2. Gunning, D., Stefik, D., Choi, J., Miller, T., Stumpf, S. and Yang G.-Z.(2019), XAI—Explainable artificial intelligence, Science Robotics, 4(37).

3. [LRP] Bach, S., Binder, A., Montavon, G., Klauschen, F., Müller, K. R., & Samek, W. (2015). On pixel-wise explanationsfor non-linear classifier decisions by layer-wise relevance propagation. PloS one, 10(7), e0130140.

4. [DTD] Montavon, G., Lapuschkin, S., Binder, A., Samek, W., & Müller, K. R. (2017). Explaining nonlinear classification decisions with deep taylor decomposition. Pattern Recognition, 65, 211-222.

5. [PatternNet] Li, H., Ellis, J. G., Zhang, L., & Chang, S. F. (2018). Patternnet: Visual pattern mining with deep neural network. In Proceedings of the ACM on International Conference on Multimedia Retrieval (pp. 291-299).

6. [Clever Hans] Lapuschkin, S., Wäldchen, S., Binder, A., Montavon, G., Samek, W., Muller, K.-R. (2019). Unmasking Clever Hans predictors and assessing what machines really learn. Nature Communication 10, 1096.

7. [RAP] Nam, W. J., Choi, J., & Lee, S. W. (2020). Relative Attributing Propagation: Interpreting the Comparative Contributions of Individual Units in Deep Neural Networks. AAAI Conference on Artificial Intelligence. Gradient Based

8. [DeConvNet] Zeiler, Matthew D., and Rob Fergus. (2014). Visualizing and understanding convolutional networks. European conference on computer vision. Springer.

9. [DeepLIFT] Shrikumar, A., Greenside, P., & Kundaje, A. (2017). Learning important features through propagating activation differences. In Proceedings of the 34th International Conference on Machine Learning-Volume 70 (pp. 3145-3153). JMLR. org.

10. [Guided Backprop] Springenberg, J. T., Dosovitskiy, A., Brox, T., & Riedmiller, M. (2014). Striving for simplicity: The all convolutional net. arXiv preprint arXiv:1412.6806.

11. [GradCAM] Selvaraju, R. R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., & Batra, D. (2017). Grad-cam: Visual explanations from deep networks via gradient-based localization. In Proceedings of the IEEE International Conference on Computer Vision(pp. 618-626). Explaining Internal Nodes

12. [Network Dissection] Bau, D., Zhou, B., Khosla, A., Oliva, A., & Torralba, A. (2017). Network dissection: Quantifying interpretability of deep visual representations. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (pp. 6541-6549).

13. [GAN Dissection] Bau, D., Zhu, J. Y., Strobelt, H., Zhou, B., Tenenbaum, J. B., Freeman, W. T., & Torralba, A. (2018). Gan dissection: Visualizing and understanding generative adversarial networks. arXiv preprint arXiv:1811.10597.

14. [E-GBAS] Jeon, G., Jeong, H., Choi, J. (2020). An Efficient Explorative Sampling Considering the Generative Boundaries of Deep Generative Neural Networks. In Thirty-Third AAAI Conference on Artificial Intelligence.

15. [Cluster Explanations] Kauffmann, J., Esders, M., Montavon, G., Samek, W., Muller, K.-R. (2019). From Clustering to Cluster Explanations via Neural Networks, Arxiv 1906.07633. Explaining though attention

16. [RETAIN] Choi, E., Bahadori, M. T., Sun, J., Kulas, J., Schuetz, A., & Stewart, W. (2016). Retain: An interpretable predictive model for healthcare using reverse time attention mechanism. In Advances in Neural Information Processing Systems (pp. 3504-3512).

17. [Saliency Maps] Zhao, R., Ouyang, W., Li, H., & Wang, X. (2015). Saliency detection by multicontext deep learning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (pp. 1265-1274).

18. [AMTL] Lee, G., Yang, E., & Hwang, S. (2016, June). Asymmetric multi-task learning based on task relatedness and loss. In International Conference on Machine Learning (pp. 230-238).

19. [UA] Heo, J., Lee, H. B., Kim, S., Lee, J., Kim, K. J., Yang, E., & Hwang, S. J. (2018). Uncertainty-aware attention for reliable interpretation and prediction. In Advances in Neural Information Processing Systems (pp. 909-918). Generating Explanations

20. [Neural Module Network] Andreas, J., Rohrbach, M., Darrell, T., & Klein, D. (2016). Neural module networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (pp. 39-48).

21. [GVE] Hendricks, L. A., Akata, Z., Rohrbach, M., Donahue, J., Schiele, B., & Darrell, T. (2016, October). Generating visual explanations. In European Conference on Computer Vision (pp. 3-19). Springer, Cham.

22. [10-K Reports] , Y. Chun at el, “Predicting and Explaining Cause of Changes in Stock Prices By Reading Annual Reports”, NeurIPS 2019 Workshop on Robust AI in Financial Services

23. [Regional Anomaly] Lee E., Choi J., Kim M., Suk H. (2019). Toward an interpretable Alzheimer's disease diagnostic model with regional abnormality representation via deep learning. In NeuroImage (vol. 202) Local Linear Explanation

24. [LIME] Ribeiro, M. T., Singh, S., & Guestrin, C. (2016, August). Why should i trust you?: Explaining the predictions of any classifier. In Proceedings of the 22nd ACM SIGKDD international conference on knowledge discovery and data mining (pp. 1135-1144). ACM.

25. [SHAP] Lundberg, S. M., & Lee, S. I. (2017). A unified approach to interpreting model predictions.In Advances in Neural Information Processing Systems (pp. 4765-4774). Decision Tree Explanator

26. [N-BEATS] Oreshkin, Boris N., et al. N-BEATS: Neural basis expansion analysis for interpretable time series forecasting. arXiv preprint arXiv:1905.10437 (2019).

27. [Automatic Statistician] Steinruecken, C., Smith, E., Janz, D., Lloyd, J., & Ghahramani, Z. (2019). The Automatic Statistician. In Automated Machine Learning (pp. 161-173). Springer, Cham.

28. [ABCD] Lloyd, J. R., Duvenaud, D., Grosse, R., Tenenbaum, J., & Ghahramani, Z. (2014, June). Automatic construction and natural-language description of nonparametric regression models. In Twenty-eighth AAAI conference on artificial intelligence.

29. [R-ABCD] Hwang, Y., Tong, A., & Choi, J. (2016, June). Automatic construction of nonparametric relational regression models for multiple time series. In International Conference on Machine Learning(pp. 3030-3039)

30. Trevor Darrell, “Recent progress towards XAI at UC Berkeley”, 2019 ICCV VXAI workshop http://xai.kaist.ac.kr/static/img/event/ICCV_2019_VXAI_Trevo_Talk.pdf

31. Wojciech Samek,“Meta-Explanations, Interpretable Clustering & Other Recent Development”, http://xai.kaist.ac.kr/static/img/event/ICCV_2019_VXAI_Samek_Talk.pdf

Additional References of Figure 1

Relevance Score based

1. [LRP] Bach, S., Binder, A., Montavon, G., Klauschen, F., Müller, K. R., & Samek, W. (2015). On pixel-wise explanations for non-linear classifier decisions by layer-wise relevance propagation. PloS one, 10(7), e0130140.

2. [DTD] Montavon, G., Lapuschkin, S., Binder, A., Samek, W., & Müller, K. R. (2017). Explaining nonlinear classification decisions with deep taylor decomposition. Pattern Recognition, 65, 211-222.

3. [PatternNet] Li, H., Ellis, J. G., Zhang, L., & Chang, S. F. (2018). Patternnet: Visual pattern mining with deep neural network. In Proceedings of the ACM on International Conference on Multimedia Retrieval (pp. 291-299).

4. [Clever Hans] Lapuschkin, S., Wäldchen, S., Binder, A., Montavon, G., Samek, W., Muller, K.-R. (2019). Unmasking Clever Hans predictors and assessing what machines really learn. Nature Communication 10, 1096.

5. [RAP] Nam, W. J., Choi, J., & Lee, S. W. (2020). Relative Attributing Propagation: Interpreting the Comparative Contributions of Individual Units in Deep Neural Networks. AAAI Conference on Artificial Intelligence.

Gradient Based

1. [DeConvNet] Zeiler, Matthew D., and Rob Fergus. (2014). Visualizing and understanding convolutional networks. European conference on computer vision. Springer.

2. [DeepLIFT] Shrikumar, A., Greenside, P., & Kundaje, A. (2017). Learning important features through propagating activation differences. In Proceedings of the 34th International Conference on Machine Learning-Volume 70 (pp. 3145-3153). JMLR. org.

3. [Guided Backprop] Springenberg, J. T., Dosovitskiy, A., Brox, T., & Riedmiller, M. (2014). Striving for simplicity: The all convolutional net. arXiv preprint arXiv:1412.6806.

4. [GradCAM] Selvaraju, R. R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., & Batra, D. (2017). Grad-cam: Visual explanations from deep networks via gradient-based localization. In Proceedings of the IEEE International Conference on Computer Vision (pp. 618-626).

Explaining Internal Nodes

1. [Network Dissection] Bau, D., Zhou, B., Khosla, A., Oliva, A., & Torralba, A. (2017). Network dissection: Quantifying interpretability of deep visual representations. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (pp. 6541-6549).

2. [GAN Dissection] Bau, D., Zhu, J. Y., Strobelt, H., Zhou, B., Tenenbaum, J. B., Freeman, W. T., & Torralba, A. (2018). Gan dissection: Visualizing and understanding generative adversarial networks. arXiv preprint arXiv:1811.10597.

3. [E-GBAS] Jeon, G., Jeong, H., Choi, J. (2020). An Efficient Explorative Sampling Considering the Generative Boundaries of Deep Generative Neural Networks. In Thirty-Third AAAI Conference on Artificial Intelligence.

4. [Cluster Explanations] Kauffmann, J., Esders, M., Montavon, G., Samek, W., Muller, K.-R. (2019). From Clustering to Cluster Explanations via Neural Networks, Arxiv 1906.07633.

Explaining though attention

1. [RETAIN] Choi, E., Bahadori, M. T., Sun, J., Kulas, J., Schuetz, A., & Stewart, W. (2016). Retain: An interpretable predictive model for healthcare using reverse time attention mechanism. In Advances in Neural Information Processing Systems (pp. 3504-3512).

2. [Saliency Maps] Zhao, R., Ouyang, W., Li, H., & Wang, X. (2015). Saliency detection by multicontext deep learning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (pp. 1265-1274).

3. [AMTL] Lee, G., Yang, E., & Hwang, S. (2016, June). Asymmetric multi-task learning based on task relatedness and loss. In International Conference on Machine Learning (pp. 230-238).

4. [UA] Heo, J., Lee, H. B., Kim, S., Lee, J., Kim, K. J., Yang, E., & Hwang, S. J. (2018). Uncertainty-aware attention for reliable interpretation and prediction. In Advances in Neural Information Processing Systems (pp. 909-918).

Generating Explanations

1. [Neural Module Network] Andreas, J., Rohrbach, M., Darrell, T., & Klein, D. (2016). Neural module networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (pp. 39-48).

2. [GVE] Hendricks, L. A., Akata, Z., Rohrbach, M., Donahue, J., Schiele, B., & Darrell, T. (2016, October). Generating visual explanations. In European Conference on Computer Vision (pp. 319). Springer, Cham.

3. [10-K Reports] , Y. Chun at el, “Predicting and Explaining Cause of Changes in Stock Prices By Reading Annual Reports”, NeurIPS 2019 Workshop on Robust AI in Financial Services

4. [Regional Anomaly] Lee E., Choi J., Kim M., Suk H. (2019). Toward an interpretable Alzheimer's disease diagnostic model with regional abnormality representation via deep learning. In NeuroImage (vol. 202)

Local Linear Explanation

1. [LIME] Ribeiro, M. T., Singh, S., & Guestrin, C. (2016, August). Why should i trust you?: Explaining the predictions of any classifier. In Proceedings of the 22nd ACM SIGKDD international conference on knowledge discovery and data mining (pp. 1135-1144). ACM.

2. [SHAP] Lundberg, S. M., & Lee, S. I. (2017). A unified approach to interpreting model predictions. In Advances in Neural Information Processing Systems (pp. 4765-4774).

Decision Tree Explanator

1. [RxREN] Augasta, M. G., & Kathirvalavakumar, T. (2012). Reverse engineering the neural networks for rule extraction in classification problems. Neural processing letters, 35(2), 131150.

2. [CRED] Sato, M., & Tsukimoto, H. (2001, July). Rule extraction from neural networks via decision tree induction. In IJCNN'01. International Joint Conference on Neural Networks. Proceedings (Cat. No. 01CH37222) (Vol. 3, pp. 1870-1875). IEEE.

3. [DeepRED] Zilke, J. R., Mencía, E. L., & Janssen, F. (2016, October). DeepRED–Rule extraction from deep neural networks. In International Conference on Discovery Science (pp. 457-473). Springer, Cham.

Rule Based Explanator

1. [KBANN] Towell, G. G., & Shavlik, J. W. (1994). Knowledge-based artificial neural networks. Artificial intelligence, 70(1-2), 119-165.

2. [MofN] Towell, G. G., & Shavlik, J. W. (1993). Extracting refined rules from knowledge-based neural networks. Machine learning, 13(1), 71-101.

3. [BRAINNE] Dillon, T. S., Hossain, T., Bloomer, W., & Witten, M. (1998). Improvements in supervised BRAINNE: A method for symbolic datamining using neural networks. In Data Mining and Reverse Engineering (pp. 67-88). Springer, Boston, MA.

4. [RULEX] Navarro, D. J. (2005). Analyzing the RULEX model of category learning. Journal of Mathematical Psychology, 49(4), 259-275.

Explaining with Examples

1. [MMD-critic] Kim, B., Khanna, R., & Koyejo, O. O. (2016). Examples are not enough, learn to criticize! criticism for interpretability. In Advances in Neural Information Processing Systems (pp. 2280-2288).

Explainable Model Composition

1. [Automatic Statistician] Steinruecken, C., Smith, E., Janz, D., Lloyd, J., & Ghahramani, Z. (2019). The Automatic Statistician. In Automated Machine Learning (pp. 161-173). Springer, Cham.

2. [ABCD] Lloyd, J. R., Duvenaud, D., Grosse, R., Tenenbaum, J., & Ghahramani, Z. (2014, June). Automatic construction and natural-language description of nonparametric regression models. In Twenty-eighth AAAI conference on artificial intelligence.

3. [R-ABCD] Hwang, Y., Tong, A., & Choi, J. (2016, June). Automatic construction of nonparametric relational regression models for multiple time series. In International Conference on Machine Learning (pp. 3030-3039).

4. [LKM] Saradhi, V. V., & Karnick, H. (2007, May). On the Stability and Bias-Variance Analysis of Kernel Matrix Learning. In Conference of the Canadian Society for Computational Studies of Intelligence (pp. 441-451). Springer, Berlin, Heidelberg.

Learning Extracting Causalit

1. [Causal reasoning] Dasgupta, I., Wang, J., Chiappa, S., Mitrovic, J., Ortega, P., Raposo, D., ... & Kurth-Nelson, Z. (2019). Causal reasoning from meta-reinforcement learning. arXiv preprint arXiv:1901.08162.

2. [Contrastive Explanation] LIPTON, Peter. Contrastive explanation. Royal Institute of Philosophy Supplements, 1990, 27: 247-266.

3. [Counter-factual Gen] CHEN, Sheng; ERWIG, Martin. Counter-factual typing for debugging type errors. In: ACM SIGPLAN Notices. ACM, 2014.

4. [Sequential FAC] SHIM, Hajin; HWANG, Sung Ju; YANG, Eunho. Joint active feature acquisition and classification with variable-size set encoding. In: Advances in Neural Information Processing Systems. 2018. p. 1368-1378.

5. [F-TC] S.-H. Kang, H.-G. Jung, and S.-W. Lee. Interpreting Undesirable Pixels for Image Classification on Black-Box Models. arXiv preprint arXiv:1909.12446.

Extracting Models

1. [Extract-FSM] Koul, Anurag, Sam Greydanus, and Alan Fern. (2019). Learning finite state representations of recurrent policy networks. In International Conference on Learning Representations (ICLR).

2. [PIPL] Verma, A., Murali, V., Singh, R., Kohli, P., & Chaudhuri, S. (2018, July). Programmatically Interpretable Reinforcement Learning. In International Conference on Machine Learning (pp. 5052-5061).

Explainable Actions

1. [Reward-Decomp] Juozapaitis, Z., Koul, A., Fern, A., Erwig, M., & Doshi-Velez, F. (2019). Explainable reinforcement learning via reward decomposition. In Proceedings of the IJCAI 2019 Workshop on Explainable Artificial Intelligence (pp. 47-53)

2. [Actor-Critic Saliency] Greydanus, S., Koul, A., Dodge, J., & Fern, A. (2018, July). Visualizing and Understanding Atari Agents. In International Conference on Machine Learning (pp. 1787-1796).

3. [Min-Suff-Exp] Khan, O. Z., Poupart, P., & Black, J. P. (2009, September). Minimal sufficient explanations for factored Markov Decision Processes. In Proceedings of the Nineteenth International Conference on International Conference on Automated Planning and Scheduling (pp. 194-200). AAAI Press.

4. [VG-UCT] LEE, Jongmin, et al. Monte-Carlo Tree Search in Continuous Action Spaces with Value Gradients. 2020, AAAI.