The Project
Neural Networks are kind of these scary mighty blackboxes. That was the reason, why I took a deeper look into that topic, researching all kinds of neural networks. After weeks of reading papers and documetations, I tested the first models using PyTorch. It's pretty easy to use this framework to create even complicated deep networks, without knowing how it's working under the hood.
It was time for nodevember, so I challenged myself and wanted to find out, if it's possible to implement a fully functional convolutional neural network in Designer?
The answer is YES! I managed to implement all the layers of the famous VGG16 Model from scratch using the pretrained weights from the original model trained on the ImageNet dataset predicting 1000 different classes.
The result is probably the largest graph I've ever done. The size of the graph is 86851px x 2148px (Substance 3D Designer didn't render the whole graph so I had to find a workaround for showing the graph as well). I also created a set of nodes, which I will publish soon, to create your own neural networks in Designer.
Breakdown
Before we go through a detailed breakdown, some impressive statistics about the graph:
- 8,488 Pixelprocessors
- 4,225 Valueprocessors
- 138,357,544 Trainable Parameters using 527.79MB of memory
- the total size of the .sbsar is 594.65MB
- time to compute the node around 3.5 seconds
A short introduction into the VGG16 Network
VGG16 Model is a deep convolutional neural network architecture known for its simplicity and effectiveness in image classification tasks. Developed by the Visual Geometry Group (VGG) at Oxford, it consists of 16 weight layers, including 13 convolutional layers and 3 fully connected layers. The network's uniform structure with small 3x3 convolutional filters helps in learning hierarchical features. Despite its depth, VGG16's straightforward design makes it a widely used and referenced model in computer vision.
The total learnable parameters in VGG16 is 138,357,544:
- Convolution Block 1: 2 • (3 • 3 • 3 • 64 + 64) = 3,840
- Convolution Block 2: 2 • (3 • 3 • 64 • 128 + 128) = 221,440
- Convolution Block 3: 3 • (3 • 3 • 128 • 256 + 256) = 1,475,840
- Convolution Block 4: 3 • (3 • 3 • 256 • 512 + 512) = 5,899,776
- Convolution Block 5: 3 • (3 • 3 • 512 • 512 + 512) = 11,783,424
- Fully Connected Block: 102,764,544 + 16,781,312 + 4,097,000
This makes VGG16 a relatively extensive network with a large number of parameters, even by today’s standards. However, the simplicity of the VGGNet16 architecture is its main attraction.
The VGG16 Model Architecture
The architecture of the Model consists of 5 Convolution Blocks, after each convolution block, a pooling layer reduces the size of the input. While the size is reduced to a quarter of the input of the layer, the depth gets doubled. This technique reduces the spatial dimension of the feature maps until almost every spatial information is gone and the result is a 25088 element long feature vector. The fully connected layers reduce this giant vector to a size of 1000 output classes.
to better understand, the reduction of the spatial dimension, I made a 3D visualisation of the models architecture:
What is convolution
Convolution is a mathematical operation that combines two functions to produce a third function. In the context of image processing and deep learning, convolution is often used to process and analyze data. In the case of discrete convolution, which is commonly employed in digital signal processing and deep learning, it involves an element-wise multiplication of two arrays followed by a summation. Convolution in image processing is often used to filter an image, for extracting features like edges, corners, blurring or sharpening.
Substance 3D Designer has a node which can handle these operations very well, by using the GPU - the Pixelprocessor, that's why I started to implement the network with only Pixelprocessors.
The discrete convolution between the Input Image and the Kernel is calculated as follows:
Substance 3D Designer implementation
The convolution operation in a neural network works very similar, except the input image and the kernels are 3-dimensional, so basically the input image has a depth-component as well as the kernel. These 3-dimensional Arrays are called tensors.
In Substance 3D Designer this can easily get performend in a pixel processor, the problem is, that image textures in Designer can have a maximum of 4 channels (R, G, B, A). For a network like this I needed a possibility to store more than 4 channels of an image, since the output layers have 64, 128, 256 and 512 channels.
The idea was to flatten the depth channels and reducing the third dimension to a 2-dimensional image. This is achieved by creating a texture, where each channel of the feature maps is flattened out and stored in the same image texture next to each other, this is the red, green and blue channel. of the input image, side by side in one texture.
The coefficients (weights) of the convolution is also stored in a layout like this, so it is actually quite simple to convolve the input with the kernels in one Pixelprocessor. it convolves a 3-dimensional input with a 3-dimensional kernel, outputting a single 2-dimensional channel of the feature map. In order to produce another 3-dimensional "tensor" this step is repeated for every output-channel.
After packing all the feature channels the same way as shown before, the result of the first layer is this:
The second convolution layer, performs another pass over all the feature maps, every convolution filter learns to extract other features, like edges, corners, contours and so on. 
Following the models architecture, this process is repeated until the end of the block, the pooling layer is then used to reduce the spatial size of the feature-maps. This can also be done in a Pixelprocessor.
This gets repeated until the last Pooling Layer, here are the resulting feature maps, for each layer of the network
After the last Convolution Block, all features get flattened out in a 1x1x25088 element big vector. For this I also used a Pixelprocessor 
After the flatten layer, three fully connected layer reduce this feature vector until there is only one feature highlighted. This Operation is basically a Vector - Matrix - Multiplication performed by two Pixelprocessors:
Conclusion
Since this was a personal challenge the graph needs a lot of cleanup before I can share it publicly, so please have some patience but I will upload the complete framework of nodes created for CNNs soon. Until then I uploaded the .sbsar, the high resolution image of the graph and a list of available classes to GoogleDrive feel free to download it and play with it as much as you want.
Thank you for reading, if you have questions or feedback, or is something is missing in this post, just leave a comment below.
Stay healthy and creative Marco