{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"**How AI sees: What that means for You and Me**","metadata":{}},{"cell_type":"markdown","source":"Introduction:\n\nObject recognition is one of the most astounding yet incredibly complex processes of human vision. From distinguishing faces to identifying our environments, it is something we rely on in our everyday lives. This ability is known to be structured through a network of hierarchically interconnected brain areas. After our eyes form a retinal image, it gets sent and processed from one visual area to the next in hierarchical fashion, with a set of specific visual characteristics being discerned at each layer that then allows us to identify any objects within the image. With inspiration taken from this biological model, there have been many object recognizing models developed with applications including self-driving cars recognizing their surrounding environment, face detection, and so on. Much has been discovered about humans that have led to the optimization of these artificial object recognition models. As the application for object recognition keeps expanding, we expect to see this reflection from the human brain in artificial models develop much further. This relationship leads to the question: can we find a reflection also in the converse direction, and going further, what can we learn about our visual system from artificial object recognition models?\n\nTo approach this question, I will be going over literature that explores the relationship between the brain and artificial models in three parts. The first will be reviewing the perception and response for both models. The second part will then be discussing how information is being relayed from one level to the next. Finally, in the last section, I will be comparing how both biological and artificial models perform in the complex task of face identification.\n\nThe set of artificial models created to replicate the human system have been aptly named neural networks in the way they take inspiration from the brain in purpose and structure. The general organization of them follows a primary input layer that first takes in raw information, followed by one or more intermediate layers, and finally an output layer that is responsible for categorizing the data. In a similar fashion to how each visual area of the brain is composed of neurons, each layer of neural networks consists of nodes that, like neurons, act as singular units for taking in and sending information. In between these layers are weights that get applied to the information that is being sent from nodes in one layer to the next as means of signaling how each part of the data should be considered and how important that information is. The greater weight assigned, the more impact the information has in the recognition process. The final output layer utilizes the collection of all the information sent through the layers to ultimately come to a verdict on the raw data.\n\nOver the years, neural networks have evolved to provide increasingly biological plausible models of our own visual system. These neural networks have grown to resemble the human brain not just in performance but also in the way they behave and activate to stimuli. The increasing evidence indicating a significantly positive correlation between performance and response for these two models strongly implies that these models are in convergent evolution. A kind of neural network that has been especially prevalent within the field of objection recognition is the convolutional neural network (CNN) that is unique in the way it convolves information, or rather, how information is being broken down into smaller inputs such that these inputs share some overlap. If we were to take a 6x6 pixel image, for example, its convoluted mapping may be a 3x3 matrix. Each 4x4 pixel square within the 6x6 image can be summarized to a singular data entry that then becomes stored in the 3x3 convoluted mapping. While a complicated process, CNNs have been found to be extremely optimal when it comes to object recognition and may have implications for the way we also process visual information. Asides from explaining the fundamental processes behind the neural networks to be discussed, this paper will not be going in-depth behind the computation of these models but rather how the performances of these models correlate to human performance and what their processes may imply about ours.","metadata":{"_uuid":"8d7bc3e6-a379-4848-ab30-b550e7900141","_cell_guid":"5b8e7eb9-b4d8-42b2-8d7a-e405aa5ed08b","jupyter":{"outputs_hidden":false}}},{"cell_type":"markdown","source":"Part 1: Perception & Response\n\nThe first paper that I will be going over is “Performance-optimized hierarchical models predict neural responses in higher visual cortex,” written by Yamins et al. in 2014. High performance from both artificial and biological systems does not necessarily mean there exists a relationship between the two. However, by recording and comparing neural activations from macaque monkeys, Yamins et al. were able to find that invariant object recognition performance strongly correlates to IT neural predictivity. Predicting response at the IT, the inferior temporal cortex, is critical as it is known to be one of the higher and last layers responsible for object recognition. \n        \nTo collect neural data, multiple electrode arrays were used to observe the activation of 168 IT neurons from two rhesus macaques that were presented with a set of about 6000 images of photorealistic 3D objects, consisting of eight natural categories (animals, boats, cars, chairs, faces, fruits, planes, and tables). These images were constructed such that they had some complexity in orientation and scale that is robustly tolerant for human recognition but difficult for artificial visual systems. To ensure the background content of these images are also not in correlation with identifying objects, these objects were placed on cluttered natural scenes. After several iterations of generating neural networks for recognizing these objects, the highest performing networks were then used to create a model for predicting the activation of each IT neuron observed. The model was fed training data that matched images to the neural responses they stimulated from the macaques, after which the model was then tested to see how well it would predict IT activation based on images the macaques were presented with. The output from the neural networks that classified images would be fed into this predictive model and it was observed that the better these neural networks performed in object recognition, the better IT neural predictivity it also yielded.\n\nA model with strong neural predictivity in IT should also be able to perform well in object recognition. The converse being shown true in this study, that strong objection recognition also yields significant IT predictivity, indicates there is a relationship between the artificial and biological visual systems. For neural networks to be predictive of IT activation, it is likely they are mirroring some mechanisms of the human brain, meaning some of the processes at the intermediate levels may be the same. Two such processes that were found to be very sensitive across all the networks that performed high for object recognition, regardless of their architecture, was filtering and average pooling. Average pooling refers to one of the methods of mapping large information to smaller matrices. When breaking down images to smaller pieces, average pooling would map the average of the smaller pieces as opposed to some other statistic such as the maximum. The prevalence of average pooling in these high-performing models gives evidence of the significance of convolving in object recognition, meaning that our recognition system is most likely similar architecturally to that of convoluted neural networks.\n\nAnother paper relating the similarity of perception between artificial and human visual systems is “Illusory Motion Reproduced by Deep Neural Networks Trained for Prediction,” by Watanabe et al. (2018). Visual illusions is one such phenomenon considered to have been unique to animals but it has been observed that neural networks are also able to pick up on the aspects of images that lead us to perceive visual illusions, including spatial and temporal characteristics. In approaching this topic, the researchers in the study utilized a neural network known as PredNet to replicate the brain observations of visual illusions. To have PredNet first understand motion, it was first fed sets of propeller images of twenty frames, after which it had to predict the twenty-secondth frame. In seeing if the network would also be capable of perceiving motion, the network was also trained to put optical flow vectors on static images it was presented with to show where it predicted parts of the image to go in the subsequent frame. The tip of the propellers in an image, for example, may have vectors pointing clockwise. When presented with the well-known snake-ring illusion in various forms, however, the network was still able to identify motion for the rings, indicating that they were flowing in circular directions, which is also how we perceive the snake-ring illusion. To come to its conclusion of motion observed in static images, the network relied on previous data of predicting propellers.\n","metadata":{}},{"cell_type":"markdown","source":"Part 2: Relaying\n\nPart of what causes us to label the rings in the illusions as snakes is not just because of the scaled-like pattern but also because of the temporal aspects we observe. The study goes to show that while motion is not a trait often considered in object recognition, it may in some situations play a role. The experiments of Gunnar Johansson in 1973 supports this idea. In the trials he conducted of participants observing only lights in points, they were able to recognize just from motion that they were attached to people as they were moving in common human behavior. The further observations made from PredNet implies there may be connections between the two streams we know of the visual system, the dorsal and ventral stream, which are separated for motion and object recognition respectively. In recognizing one of these attributes, there is likely to be information from the other stream feeding in. The interconnections of these layers give evidence that the structure of human object recognition is more than just a simple bottom-up pathway.\n\nThe next paper that I will go over, which continues to build off illusory images, supports the idea of our human brain being a recurrent system in which information continuously gets sent from one layer to another then back. One such part of object recognition that utilizes this system is identifying images that are degraded, whether it is blurred, occluded, clustered, or distorted. Illusory contours are an example. They are images that elicit perceptions of edges and shapes, despite the lack of enclosing lines. For instance, for an arrangement of four identical squares that have been spaced out from each other, we see a plus sign despite there being no outline for it. “A neural network trained for prediction mimics diverse features of biological neurons and perception” by Lotter et al. (2020) goes over how artificial models also perceive illusory contours and compares their response to that of monkeys. Macaques were presented with a set of variations of squares that also included illusory contours that were degraded to different degrees. The firing rate at different brain areas was recorded at each presentation of a square and then compared to the response rate of PredNet. It was found for macaques that while illusory contours, especially those of higher degradation, elicited a response that was weaker and more latent that images with true contours, the recordings of the firing rates of each square variation was of similar curvatures. The performance of PredNet provided similar results that suggests the similarity in processing of both artificial and biological models. Responses were triggered later for more degraded illusory contours than for true contours meaning the model must have taken some time for the illusory contours to fill in for the gap of what it didn’t perceive immediately. The delay we see for both models heavily indicates the backpropagation happening for artificial models to also apply to the biological ones, in which there is information being passed top-down for higher levels to lower to correct for errors resulting from degradation. Further studies continue to show the correlation of improved performance with the capacity to backpropagate, especially with heavily degraded images. A possible biological explanation for this observed correlation is that as images get sent from one brain layer to another, there may be some filtering or a different process of pooling being done to clear some parts of the image better, before the image continues to get sent to higher levels.\n\nThe architecture of our brain is most likely to also be unsupervised, namely self-supervised. The difference, first, between supervised and unsupervised learning is that the former relies on labeled training data for a model to be able to classify correctly. Unsupervised learning, however, depends on training data that has not been labeled. Self-supervision is a subcategory of unsupervised learning in which a model is able to classify from unsupervised data. A study done in “Unsupervised neural network models of the ventral visual stream” by Zhuang (2020) compares the response of unsupervised artificial models to that of macaque monkeys to present a biologically plausible theory of human object recognition. The neural data of the macaques were collected from previous studies and includes the activations of V1/V4/IT when these monkeys were presented images from ImageNet. The artificial models developed in this study were also trained off the same images used to collect neural data. Results showed that some self-supervised models, namely contrastive embedding ones, were able to give comparable performances to that of supervised models, even outperforming in some recognition categories such as object position and size. These contrastive embedding models are better optimized in the way they also learn from making the comparisons of images and noticing similarities and dissimilarities with each other, not just learning what features best fit which classification. Contrastive embedding provides not only the best performance for object recognition but also the most biologically plausible explanation for the human visual system. Our classification process does not just consist of a single pipeline where an image gets surveyed for characteristics but also gets compared to other images within its classification. Similarly to the first paper discussed, this study also developed models that then predicted V1/V4/IT activity to which it found contrastive embedding networks to be efficient at. This correlation provides further evidence indicating the shared mechanisms between these specific artificial models and our brain.","metadata":{}},{"cell_type":"markdown","source":"Part 3: Facial Recognition\n\n\nThe most complex task we accomplish within object recognition may possibly be the identification of faces. Under the scope of disguising faces, “Seeing through disguise: Getting to know you with a deep convolutional neural network” by Noyes et al. (2021) gives even more results showing the significance of contrast learning in both artificial and biological models. The paper explores the capacity to identify faces in two categories: evasion and impersonation. The first concerns people using disguise to look unlike themselves, and the second, to look like someone else. Identifying faces under evasion challenges the ability to seek the same identity across a set of images trying to represent various identities whereas impersonation challenges the ability to differentiate identity across a set of images trying to represent a singular identity. A model that has been tuned in just classifying from similar features may make the mistake of identifying two similar faces as the same. To train and test artificial models, the research used the FACADE image dataset of faces from a study by Noyes & Jenkins (2019) as well as the response accuracy of human participants from the same study to compare performances. Models involved in the dataset had pictures taken of them with no disguise, an evasion disguise, and an impersonation disguise. Images were paired into the same identity, similar but different identity, and completely different identity. Of these pairs, they were also split into a no-disguise and disguise category. Under the latter, the same identity pair had evasion disguise and the two different identity pairs had impersonation disguise. To the human participants tasked with identifying a disguise within the image pair presented, some of the models were strangers but some were also familiar faces, being friends or family. The study done in 2021 employed contrast embedding networks to do the same with their accuracy also being recorded. The results showed similar performance for both models with each being able to identify faces in image pairs that had no disguises. Humans performed worse than the CNN with models that were strangers and better, unsurprisingly, with models that they were familiar with. This improvement with familiar models makes sense as participants were able to see these faces and make comparisons in a wide variety of differing orientations, stretches, and so on. This result shows the strong impact that having a large set of training data has on facial recognition and overall the consistency of the human brain as a contrastive learning model is plausible. Some of the characteristics that we can identify from faces are social and personality traits. \n\nA study, “Social Trait Information in Deep Convolutional Neural Networks Trained for Face Identification” was done by Parde et al. (2018) exploring how artificial networks could also perceive the same abstract categories that we humans do. For the methods of the study, participants were tasked with associating to 280 images of faces, 11 social traits that were taken from the Big Five Factors of Personality listed by Gosling et al. (2003), each that were classified into one of five personality domains: openness, conscientious, extraversion, agreeableness, or neuroticism. With training data of about 500,000 images generated from computers based on varying social traits, results have shown that CNNs performed comparably to the student responses. The same models were also able to identify faces, regardless of social traits, with high accuracy. The finding that these networks can accommodate both identity and social trait information is consistent with the distributed model of face processing. This distributed model claims there to be a functional division of identity and social information from faces. The results from the study supports the idea that the ventral stream responsible for object and face recognition is sensitive to changeable information from faces. What this supported model implies about the architecture of our recognition system is the existence of multiple concurrent output layers that even at that level still communicate with each other.","metadata":{}},{"cell_type":"markdown","source":"Conclusion\n\nThe capacity for neural networks to detect even social traits implies that there may be so much more than physical traits like age, sex, and ethnicity that we may recognize from people. From just first impressions, the discussed artificial models provide plausible explanations as to why with even missing information, we are able to imagine the backgrounds and come to conclusions of people who we haven’t even met. The same processes that our visual system and these advanced neural networks employ could also be applied to other parts of the brain such as the auditory and olfactory system. It is not hard to imagine that even these systems may intersect with the visual system as they all provide information about a subject to some capacity. As our retinotopic image or other information we pick from our senses may change, the recurrent and also contrast embedding system of our visual system is highly plausible. The structure that we also believe our brain to be is self-supervised and recurrent in the way it sends information from one layer to the next. From average pooling to filtering and filling in missing information, there has been much learned from artificial networks that have also been used to understand how our own system operates, to which there is still much left unknown. Biological methods to understand the mechanisms of our brains will undoubtedly improve but I also imagine that advancements will be made in artificial visual methods that will continue to provide us with theories and answers. As we also use these understandings to improve neural networks, I expect to see growth in the convergent evolution of both biological and artificial models.","metadata":{}},{"cell_type":"markdown","source":"Citations\n\nLotter, W., Kreiman, G. & Cox, D. A neural network trained for prediction mimics diverse features of biological neurons and perception. Nat Mach Intell 2, 210–219 (2020). https://doi.org/10.1038/s42256-020-0170-9 Noyes, E., Parde, C. J., Colón, Y. I., Hill, M. Q., Castillo, C. D., Jenkins, R., & O'Toole, A. J. (2021). Seeing through disguise: Getting to know you with a deep convolutional neural network. Cognition, 211, 104611. https://doi.org/10.1016/j.cognition.2021.104611 Parde, C. J., Hu, Y., Castillo, C., Sankaranarayanan, S., & O'Toole, A. J. (2019). Social trait information in deep convolutional neural networks trained for face identification. Cognitive Science, 43(6). https://doi.org/10.1111/cogs.12729 Watanabe, E., Kitaoka, A., Sakamoto, K., Yasugi, M., & Tanaka, K. (2018). Illusory Motion Reproduced by Deep Neural Networks Trained for Prediction. Frontiers in psychology, 9, 345. https://doi.org/10.3389/fpsyg.2018.00345 Yamins, D. L., Hong, H., Cadieu, C. F., Solomon, E. A., Seibert, D., & DiCarlo, J. J. (2014). Performance-optimized hierarchical models predict neural responses in higher visual cortex. Proceedings of the National Academy of Sciences of the United States of America, 111(23), 8619–8624. https://doi.org/10.1073/pnas.1403112111 Zhuang, C., Yan, S., Nayebi, A., Schrimpf, M., Frank, M. C., DiCarlo, J. J., & Yamins, D. L. K. (2021). Unsupervised neural network models of the ventral visual stream. PNAS Proceedings of the National Academy of Sciences of the United States of America, 118(3), 1–11. https://doi.org/10.1073/pnas.2014196118 11","metadata":{}}]}