There are aspects of the brain which can be viewed as purely mathematical in their nature. As in many other cases in science, we can ignore some incidental properties in order to discover the deep laws of a system. In Newton’s law of universal gravitation, we ignore a multitude of physical or chemical aspects of bodies and express the attraction between them only as a function of mass and distance. In the same way, we can look at the brain’s neural networks as simple directed graphs, in which signals are processed by “mathematical” neurons, which receive signals from their inputs (synapses), combine them according to certain functions and pass signals further to other neurons.
Artificial neural networks (ANNs) were so far developed for the purpose of providing intelligent and functional solutions to a series of problems, such as image recognition or language translation. We can conceive that as neural science continues to discover more facts, ANNs may be developed to help discover certain aspects of the brain. (See the Conclusion section of this article for more discussion on this particular subject.)
There are multiple problems and difficulties in this approach. One of them is the magnitude of the brain networks, which dwarf the size and complexity of the ANNs. If we discover a certain property of the ANNs, we may attempt to extrapolate them to very large neural networks, but this extrapolation is not necessarily linear. A simple computation shows that the complexity of a neural network with N neurons, each neuron having an average of S synapses, is of the order of
N * log(N) * S * log(S)
which is not a linear function.
I consider here a simple definition of complexity: The length of the most compressed description of a neural network. The complexity only takes in account the neuron connections, the synapse strengths and the bias of each neuron, ignoring all other aspects. My computations showed some actual numbers:
| Organ | Binary Complexity | Characters | Pages | Books |
| Brain | 9.30E+24 | 3.89E+15 | 7.79E+11 | 2.60E+09 |
| Cortex | 3.72E+23 | 7.40E+14 | 1.48E+11 | 4.93E+08 |
| Language | 9.30E+20 | 3.33E+13 | 6.67E+09 | 2.22E+07 |
The following assumptions were used:
Brain: 80 billion neurons
Cortex: 16 billion neurons
Language: 800 million neurons (5% of Cortex)
Page: 5000 characters
Book: 300 pages
The binary complexity shows the minimum length of description in bits, while the characters complexity shows the minimum length of the description in human readable characters. If the number of characters is known, one can estimate the number of pages or books which would contain that description. In particular, the description of the whole neural network of the brain would need a minimum of 2.6 billion books.
ANN Software
I have created a program called CORTEST in order to perform experiments on a simple ANN. The software can generate an ANN according to the parameters specified by the user. These parameters include: number of layers, number of neurons in each layer, a function to generate the initial synapse strengths and neuron biases and the learning rate. A low learning rate results in more precision but takes longer time to learn, while a high learning rate results in less precision but shorter time to learn. The CORTEST generated ANNs have as a main task the learning of a small English/Latin dictionary, which results in the capability to translate words. While at the time it is first generated, it cannot perform a translation, after learning it can do it with a high (but not perfect) precision.
CORTEST can also run multiple experiments and collect the results in special reports.
Experiments
I have conducted a series of experiments using the CORTEST program, in order to study some of the ANNs properties. I was interested in such properties as the speed of learning, information holding capacity and resilience to accidents which disable a significant number of neurons.
Some of the results of these experiments are presented here, but with the strong caution that there is still a large gap between the artificial and the biological neural networks. A mathematical or software model of a biological network is only as good as it can accurately capture some characteristics such as neuron biases (tendency to fire after some input is received from synapses), activation functions, or average number of synapses.
Learning curve
This is an expected result: the finetuning of the synapses and neuron biases through learning follows an asymptotic curve, with diminishing returns. Learning goes very fast in the beginning, but the resulting network configuration is still prone to errors. With more learning, the number of errors is reduced, but it takes more and more epochs (trial and correction cycles) to make minute improvements.
Using Artificial Neural Networks for Simulation of Biological Neural Networks
There are aspects of the brain which can be viewed as purely mathematical in their nature. As in many other cases in science, we can ignore some incidental properties in order to discover the deep laws of a system. In Newton’s law of universal gravitation, we ignore a multitude of physical or chemical aspects of bodies and express the attraction between them only as a function of mass and distance. In the same way, we can look at the brain’s neural networks as simple directed graphs, in which signals are processed by “mathematical” neurons, which receive signals from their inputs (synapses), combine them according to certain functions and pass signals further to other neurons.
Artificial neural networks (ANNs) were so far developed for the purpose of providing intelligent and functional solutions to a series of problems, such as image recognition or language translation. We can conceive that as neural science continues to discover more facts, ANNs may be developed to help discover certain aspects of the brain. (See the Conclusion section of this article for more discussion on this particular subject.)
There are multiple problems and difficulties in this approach. One of them is the magnitude of the brain networks, which dwarf the size and complexity of the ANNs. If we discover a certain property of the ANNs, we may attempt to extrapolate them to very large neural networks, but this extrapolation is not necessarily linear. A simple computation shows that the complexity of a neural network with N neurons, each neuron having an average of S synapses, is of the order of
N * log(N) * S * log(S)
which is not a linear function.
I consider here a simple definition of complexity: The length of the most compressed description of a neural network. The complexity only takes in account the neuron connections, the synapse strengths and the bias of each neuron, ignoring all other aspects. My computations showed some actual numbers:
| Organ | Binary Complexity | Characters | Pages | Books |
| Brain | 9.30E+24 | 3.89E+15 | 7.79E+11 | 2.60E+09 |
| Cortex | 3.72E+23 | 7.40E+14 | 1.48E+11 | 4.93E+08 |
| Language | 9.30E+20 | 3.33E+13 | 6.67E+09 | 2.22E+07 |
The following assumptions were used:
Brain: 80 billion neurons
Cortex: 16 billion neurons
Language: 800 million neurons (5% of Cortex)
Page: 5000 characters
Book: 300 pages
The binary complexity shows the minimum length of description in bits, while the characters complexity shows the minimum length of the description in human readable characters. If the number of characters is known, one can estimate the number of pages or books which would contain that description. In particular, the description of the whole neural network of the brain would need a minimum of 2.6 billion books.
ANN Software
I have created a program called CORTEST in order to perform experiments on a simple ANN. The software can generate an ANN according to the parameters specified by the user. These parameters include: number of layers, number of neurons in each layer, a function to generate the initial synapse strengths and neuron biases and the learning rate. A low learning rate results in more precision but takes longer time to learn, while a high learning rate results in less precision but shorter time to learn. The CORTEST generated ANNs have as a main task the learning of a small English/Latin dictionary, which results in the capability to translate words. While at the time it is first generated, it cannot perform a translation, after learning it can do it with a high (but not perfect) precision.
CORTEST can also run multiple experiments and collect the results in special reports.
Experiments
I have conducted a series of experiments using the CORTEST program, in order to study some of the ANNs properties. I was interested in such properties as the speed of learning, information holding capacity and resilience to accidents which disable a significant number of neurons.
Some of the results of these experiments are presented here, but with the strong caution that there is still a large gap between the artificial and the biological neural networks. A mathematical or software model of a biological network is only as good as it can accurately capture some characteristics such as neuron biases (tendency to fire after some input is received from synapses), activation functions, or average number of synapses.
Learning curve
This is an expected result: the finetuning of the synapses and neuron biases through learning follows an asymptotic curve, with diminishing returns. Learning goes very fast in the beginning, but the resulting network configuration is still prone to errors. With more learning, the number of errors is reduced, but it takes more and more epochs (trial and correction cycles) to make minute improvements.
Resilience
How quickly can an already trained (finetuned) neural network recover after an accident wiped out the functionality of some of its neurons? After the accident, the ANN can re-learn, but this takes an effort, expressed in the number of learning epochs. In this experiment, I started with a trained network and then performed multiple tests which showed the number of epochs needed to regain functionality after a number of neurons were knocked out. The table below shows the number of epochs needed to recover after neuron functionality was lost. (Lost neurons generate no output signals.) The ANN used in this experiment consisted of 5 layers of 50 neurons each. The maxim number of epochs allowed was 10,000.
| Neurons Lost in Layer | In Layer 1 | In Layers 1-2 | In Layers 1-3 | In Layers 1-4 | In Layers 1-5 |
| 10% | 162 | 279 | 345 | 438 | 681 |
| 20% | 243 | 341 | 480 | 565 | 921 |
| 30% | 310 | 450 | 559 | 719 | 1221 |
| 40% | 323 | 461 | 605 | 807 | 1375 |
| 50% | 389 | 579 | 789 | 1062 | 1650 |
| 60% | 460 | 620 | 864 | 1259 | 2520 |
| 70% | 507 | 725 | 1056 | 1885 | 5409 |
| 80% | 535 | 891 | 1261 | 3137 | 8142 |
| 90% | 959 | 1369 | 3637 | 10001 | 10001 |
As expected, the larger the number of neurons lost, the larger the effort (number of learning epochs) to recover.
The last row shows that for 45 neurons lost (90%) in each of the 5 layers, the ANN could not recover at all.
Learning effort
Using CORTEST, I generated ANNs of various sizes, and tested their learning speed. The results are displayed in the table below. I fixed the maximum number of learning epochs allowed at 10,000. An effort larger that 10,000 shows a failure to learn. The task was to learn a list of 10 English/Latin words.
| Neurons | 1 Layer | 2 Layers | 3 Layers | 4 Layers | 5 Layers |
| 5 | 10001 | 10001 | 10001 | 10001 | 10001 |
| 10 | 4503 | 4000 | 10001 | 10001 | 10001 |
| 15 | 3338 | 2411 | 10001 | 10001 | 10001 |
| 20 | 2855 | 3153 | 4440 | 5894 | 10001 |
| 25 | 3505 | 2646 | 2695 | 5287 | 10001 |
| 30 | 2357 | 2201 | 3372 | 4207 | 9551 |
| 35 | 2159 | 1806 | 4855 | 3070 | 8522 |
| 40 | 3066 | 1574 | 1753 | 1581 | 7504 |
| 45 | 2052 | 1809 | 1773 | 1489 | 2768 |
| 50 | 2815 | 1391 | 1319 | 1136 | 1545 |
One interesting conclusion is that ANNs with more layers have difficulties if the number of neurons per layer is small, but once they have a sufficiently large number of neurons, they perform better that ANNs with less layers.
One has to notice that “layers” in this case do not have the same meaning as, for instance, anatomical layers in the cortex. The first layer, in this case, is the one which receives the signal from the input layer (perceptron), the second receives signals from the first layer, etc.
Conclusions
Artificial neural networks are inspired by the biological networks, but in general their purpose is to perform, not to simulate. While the same principles appear in both biological and artificial NNs, some differences remain, and even become larger, as the ANNs move on their own development path. It is, however, entirely possible that new ANNs could be developed for the exact purpose to simulate biological neural networks. To facilitate such a development, a number of questions need to be answered:
- What is the activation function of a biological neuron? While we know that the signals from the synapses are combined to make the neuron fire, what is the exact, if possible mathematical function which determines the firing event?
- If an activation function can be determined in some particular cases, is it possible that different types of biological neurons have different types of activation functions? What is the activation function for each type?
- What is the approximate number and structure of neurons for each functional area, such as language or symbolic thinking?
- What is the number of “logical” (not biological) layers in various areas of the cortex? Logical layers are defined by the network topology (first layer receives the signal, second layer receives from the first layer, etc.)
- While in ANNs the neurons of each layer receive signals only from the previous layer, it is possible that in biological NNs the topology of the network is much more complicated. Some signals may travel back and forth through the layers. Can this topology be mathematically characterized?
- ANNs learn using a technique called backpropagation, in which a neuron’s synapses and bias are adjusted based on complex mathematical formulas. The whole backpropagation algorithm is run outside of the network itself and its results are simply retrofitted into the network configuration. It must be different in a biological NN, which must handle the finetuning itself, without an externally run algorithm. What is the biological mechanism for learning?
- Is there some built-in redundancy? While the general principles of brain plasticity are known, is there some redundancy, such that if some signal paths are blocked, some other paths are already available to perform the same tasks?
- Does the neuron perceive any continuous value of a synapse signal, or just discrete ranges? It is conceivable that a very small change in the signal from a synapse does not affect the function of a receiving neuron, but what is the threshold at which the neuron perceives that the signal is different? Can we establish a limited number of ranges of signal strengths which are relevant?
These questions may be viewed as a research program. Perhaps more such questions would emerge.
Brain neural networks simulation by ANNs could open a whole new range of scientific discoveries. While some would be mostly theoretical, helping to understand the brain, some of them may have immediate practical applications. I have shown above results from the ANN simulation of a stroke, with computations of recovery effort. Supposing that we can better simulate biological neural networks, such a study could make predictions on the recovery possibilities after a real stroke.
Resilience
How quickly can an already trained (finetuned) neural network recover after an accident wiped out the functionality of some of its neurons? After the accident, the ANN can re-learn, but this takes an effort, expressed in the number of learning epochs. In this experiment, I started with a trained network and then performed multiple tests which showed the number of epochs needed to regain functionality after a number of neurons were knocked out. The table below shows the number of epochs needed to recover after neuron functionality was lost. (Lost neurons generate no output signals.) The ANN used in this experiment consisted of 5 layers of 50 neurons each. The maxim number of epochs allowed was 10,000.
| Neurons Lost in Layer | In Layer 1 | In Layers 1-2 | In Layers 1-3 | In Layers 1-4 | In Layers 1-5 |
| 10% | 162 | 279 | 345 | 438 | 681 |
| 20% | 243 | 341 | 480 | 565 | 921 |
| 30% | 310 | 450 | 559 | 719 | 1221 |
| 40% | 323 | 461 | 605 | 807 | 1375 |
| 50% | 389 | 579 | 789 | 1062 | 1650 |
| 60% | 460 | 620 | 864 | 1259 | 2520 |
| 70% | 507 | 725 | 1056 | 1885 | 5409 |
| 80% | 535 | 891 | 1261 | 3137 | 8142 |
| 90% | 959 | 1369 | 3637 | 10001 | 10001 |
As expected, the larger the number of neurons lost, the larger the effort (number of learning epochs) to recover.
The last row shows that for 45 neurons lost (90%) in each of the 5 layers, the ANN could not recover at all.
Learning effort
Using CORTEST, I generated ANNs of various sizes, and tested their learning speed. The results are displayed in the table below. I fixed the maximum number of learning epochs allowed at 10,000. An effort larger that 10,000 shows a failure to learn. The task was to learn a list of 10 English/Latin words.
| Neurons | 1 Layer | 2 Layers | 3 Layers | 4 Layers | 5 Layers |
| 5 | 10001 | 10001 | 10001 | 10001 | 10001 |
| 10 | 4503 | 4000 | 10001 | 10001 | 10001 |
| 15 | 3338 | 2411 | 10001 | 10001 | 10001 |
| 20 | 2855 | 3153 | 4440 | 5894 | 10001 |
| 25 | 3505 | 2646 | 2695 | 5287 | 10001 |
| 30 | 2357 | 2201 | 3372 | 4207 | 9551 |
| 35 | 2159 | 1806 | 4855 | 3070 | 8522 |
| 40 | 3066 | 1574 | 1753 | 1581 | 7504 |
| 45 | 2052 | 1809 | 1773 | 1489 | 2768 |
| 50 | 2815 | 1391 | 1319 | 1136 | 1545 |
One interesting conclusion is that ANNs with more layers have difficulties if the number of neurons per layer is small, but once they have a sufficiently large number of neurons, they perform better that ANNs with less layers.
One has to notice that “layers” in this case do not have the same meaning as, for instance, anatomical layers in the cortex. The first layer, in this case, is the one which receives the signal from the input layer (perceptron), the second receives signals from the first layer, etc.
Conclusions
Artificial neural networks are inspired by the biological networks, but in general their purpose is to perform, not to simulate. While the same principles appear in both biological and artificial NNs, some differences remain, and even become larger, as the ANNs move on their own development path. It is, however, entirely possible that new ANNs could be developed for the exact purpose to simulate biological neural networks. To facilitate such a development, a number of questions need to be answered:
- What is the activation function of a biological neuron? While we know that the signals from the synapses are combined to make the neuron fire, what is the exact, if possible mathematical function which determines the firing event?
- If an activation function can be determined in some particular cases, is it possible that different types of biological neurons have different types of activation functions? What is the activation function for each type?
- What is the approximate number and structure of neurons for each functional area, such as language or symbolic thinking?
- What is the number of “logical” (not biological) layers in various areas of the cortex? Logical layers are defined by the network topology (first layer receives the signal, second layer receives from the first layer, etc.)
- While in ANNs the neurons of each layer receive signals only from the previous layer, it is possible that in biological NNs the topology of the network is much more complicated. Some signals may travel back and forth through the layers. Can this topology be mathematically characterized?
- ANNs learn using a technique called backpropagation, in which a neuron’s synapses and bias are adjusted based on complex mathematical formulas. The whole backpropagation algorithm is run outside of the network itself and its results are simply retrofitted into the network configuration. It must be different in a biological NN, which must handle the finetuning itself, without an externally run algorithm. What is the biological mechanism for learning?
- Is there some built-in redundancy? While the general principles of brain plasticity are known, is there some redundancy, such that if some signal paths are blocked, some other paths are already available to perform the same tasks?
- Does the neuron perceive any continuous value of a synapse signal, or just discrete ranges? It is conceivable that a very small change in the signal from a synapse does not affect the function of a receiving neuron, but what is the threshold at which the neuron perceives that the signal is different? Can we establish a limited number of ranges of signal strengths which are relevant?
These questions may be viewed as a research program. Perhaps more such questions would emerge.
Brain neural networks simulation by ANNs could open a whole new range of scientific discoveries. While some would be mostly theoretical, helping to understand the brain, some of them may have immediate practical applications. I have shown above results from the ANN simulation of a stroke, with computations of recovery effort. Supposing that we can better simulate biological neural networks, such a study could make predictions on the recovery possibilities after a real stroke.