Homepage of Armando Vieira

Data Scientist, Entrepreneur and Speaker

Deep Learning Neural Networks: Overview and Business Applications

Introduction

Deep Neural Networks (DNN), or simply Deep Learning (DL), took Artificial Intelligence (AI) by storm and have infiltrated into business at an unprecedented rate. Access to vast amounts of data, recently made available by the Big Data revolution, extensive computational power and a new wave of efficient learning algorithms, helped Artificial Neural Networks to achieve state-of-the-art results in almost all AI challenges.

DL is the cornerstone technology behind products for image recognition and video annotation, voice recognition, personal assistants, automated translation and autonomous vehicles. DNN works similarly to the brain by extracting high-level, complex abstractions from data in a hierarchical and generative way.

A key feature of DL algorithms is their capability to learn from large amounts of data with minimal supervision – contrary to shallow models that normally require less data, but with labels, and easily reach an accuracy plateau. DL however, comes with a cost: there is no theory to guide the learning algorithms and architecture optimization and hyper-parameters selection rely on complex and time consuming heuristics – training a single model can take weeks on well equipped PCs.

In this review we present the most significant algorithms and applications, including Natural Language Processing (NLP), image and video processing, risk assessment, forecasting, robotics, gaming and finance. The implications of DL supported AI in business is tremendous, shaking to the foundations many industries – peraphs the biggest transformative force after the Internet.

In this review we refer some key findings and implications in business of DL, key companies and startups adopting this technology and some investments. We also refer to some frameworks for training the DL models, key methods and tricks to fine tune the models and finally some products and companies where the this technology is being applied.

Since this is a book for wide audience, I’ve tried to keep the mathematics and statistics to the bare minimum. However,  some knowledge of statistics, optimization and machine learning techniques is required – but can be founded elsewhere.

 

Table of Contents

Abstract……….. 1
Introduction……….. 5
1.1 AI: From a long winter to a blossom spring…….. 6
1.2 Why DL is making a difference?…….. 8
1.3 The age of the machines…….. 9
1.4 Resources…….. 10
1.5 DL as a service…. 13

2. Deep Neural Network models……….. 15
2.1 A Brief history of Neural Networks…….. 15
2.1.1 The MLP…. 15
2.2 Deep Neural Networks…….. 18
2.2.1 Convolutional Neural Networks (CNN)…. 20
2.2.2 Deep autoencoders…. 21
2.2.3 Deep Belief Networks…. 21
2.2.4 Recurrent Neural Networks…. 22
2.2.5 Recurrent networks for reinforcement learning…. 23
2.3 Training DNNs…….. 24
2.3.1 Initialization, overfitting and regularization…. 24
2.3.2 Hyperparameter optimization…. 25
2.3.3 Network compression…. 25

3. Natural Language Processing (NLP)……….. 27
3.1 Parsing…. 27
3.2 Distributed Representations…….. 28
3.3 Combining structure and unstructured data through Knowledge Graphs…….29
3.4 Natural Language Translation…….. 32
3.5 Other applications…….. 33
3.6 Multimodal learning …….. 34
3.7 Automatic Speech Recognition…….. 34
3.8 News and Resources…….. 35
3.9 Summary and a speculative outlook…. 36

4.Image processing……….. 37
4.1 Convolutional Neural Networks (CNNs)…….. 37
4.2 Imagenet and beyond…….. 38
4.3 Image captioning…….. 43
4.3.1 Image annotation…. 43
4.3.2 Image Q&A…. 44
4.4 Video analysis…….. 45
4.5 Other applications…….. 47
4.5.1 Satellite images…. 47
4.5.2 News and Companies…. 48
4.5.3 APIs…. 49

5. Reinforcement Learning, Robotics and Control……….. 50
5.1 Object manipulation…….. 50
5.2 Self-driving cars…….. 51
5.3 Conversational Bots (Chatbots)…….. 52
5.4 Reinforcement Learning and robotic motion…….. 56
5.5 Applications…….. 60

6 Recommendations Algorithms, Marketing and Advertising……….. 62
6.1 Ad Click prediction…….. 62
6.2 Online used behaviour…….. 63
6.3 Recommendation Algorithms…….. 63
6.3.1 Collaborative Filters…. 64
6.3.2 Deep Learning approaches to RS…. 64
6.3.3 Applications of Recommendation Algorithms…. 66

7 Games and Art……….. 67
7.1 The early steps in Chess…….. 67
7.2 From chess to GO…….. 67
7.5 Artificial Characters…….. 69
7.6 Applications in Art…….. 69

8.Other Business Applications……….. 74
8.1 Anomaly Detection and Fraud…….. 74
8.2 Security and prevention…….. 76
8.3 Forecasting…….. 77
8.3.1 Trading…. 78
8.4 Credit Risk…….. 79
8.5 Medicine and Biomedical…….. 79
8.6 User experience…….. 81
8.7 OTHER…….. 82

9 The Impact of DL in Society, Opportunities and Risks……….. 84
9.1 Business impact…….. 85
9.2 Risks…….. 87
9.3 Opportunities…… 89
9.4 Challenges …. 92

10 Appendix I: New Research and Future Directions……….. 97
10.1 One shot learning…….. 97
10.2 Control…….. 98
10.3 Generative Neural Networks (GNN)…….. 100
10.3.1 Generative Adversial Neural Networks (GANN)…. 102
10.4 Multimodal Learning…….. 105
10.5 Knowledge transfer…….. 107
10.6 What AI can’t solve yet…….. 109
10.7 News…….. 113

Bibliography……….. 120