A fully connected layer, also known as a dense layer, is a fundamental component of neural networks, especially in feedforward neural networks and the later stages of Convolutional Neural Networks (CNNs). In a fully connected layer, each neuron is connected to every neuron in the previous layer. This layer performs a linear transformation followed by an activation function, enabling the model to learn complex representations.
Key Concepts
- Neurons:
- Each neuron in a fully connected layer takes input from all neurons in the previous layer.
- The connections between neurons are represented by weights, which are learned during training.
- Weights and Biases:
- Weights: Each connection between neurons has an associated weight, which is adjusted during training to minimize the loss function.
- Bias: Each neuron has an additional parameter called bias, which is added to the weighted sum of inputs.
- Activation Function:
- After the linear transformation (weighted sum plus bias), an activation function is applied to introduce non-linearity.
- Common activation functions include ReLU (Rectified Linear Unit), Sigmoid, and Tanh.
How It Works
- Input: A vector of activations from the previous layer.
- Linear Transformation: Each neuron computes a weighted sum of its inputs plus a bias. z=∑i=1n(wi⋅xi)+bz = \sum_{i=1}^{n} (w_i \cdot x_i) + bz=i=1∑n(wi⋅xi)+b where wiw_iwi are the weights, xix_ixi are the input activations, and bbb is the bias.
- Activation Function: An activation function is applied to the linear transformation to produce the output of the neuron.a=activation(z)a = \text{activation}(z)a=activation(z)
- Output: The outputs of the activation functions from all neurons in the layer are passed to the next layer.
Example in Keras
Here’s an example of how to create a simple neural network with a fully connected layer using Keras:
pythonCopy codefrom tensorflow.keras.models import Sequential
from tensorflow.keras.layers import Dense
# Create a simple model with one hidden dense layer
model = Sequential()
model.add(Dense(units=64, activation='relu', input_shape=(784,))) # Input layer with 784 neurons (e.g., flattened 28x28 image)
model.add(Dense(units=10, activation='softmax')) # Output layer with 10 neurons (e.g., for 10 classes)
# Print the model summary
model.summary()
Explanation of the Example Code
- Dense: This function creates a fully connected (dense) layer.
units=64: The number of neurons in the layer.activation='relu': The activation function applied to the layer’s output.input_shape=(784,): The shape of the input data (e.g., a flattened 28×28 image).
Common Activation Functions
- ReLU (Rectified Linear Unit):ReLU(x)=max(0,x)\text{ReLU}(x) = \max(0, x)ReLU(x)=max(0,x)
- Most commonly used activation function in hidden layers.
- Efficient and helps mitigate the vanishing gradient problem.
- Sigmoid:σ(x)=11+e−x\sigma(x) = \frac{1}{1 + e^{-x}}σ(x)=1+e−x1
- Maps the input to a range between 0 and 1.
- Used in the output layer for binary classification.
- Tanh (Hyperbolic Tangent):tanh(x)=ex−e−xex+e−x\tanh(x) = \frac{e^x – e^{-x}}{e^x + e^{-x}}tanh(x)=ex+e−xex−e−x
- Maps the input to a range between -1 and 1.
- Can be used in hidden layers, especially when dealing with normalized input data.
- Softmax:softmax(xi)=exi∑jexj\text{softmax}(x_i) = \frac{e^{x_i}}{\sum_{j} e^{x_j}}softmax(xi)=∑jexjexi
- Used in the output layer for multi-class classification.
- Produces a probability distribution over multiple classes.
Importance of Fully Connected Layers
- Feature Combination: Fully connected layers combine features learned by convolutional and pooling layers, helping to make final decisions based on the extracted features.
- Flexibility: They can model complex relationships by learning the appropriate weights and biases.
- Adaptability: Can be used in various types of neural networks and architectures, including CNNs, RNNs, and more.
Applications
- Classification: Commonly used in the output layer of classification networks.
- Regression: Can be used for regression tasks by having a single neuron with a linear activation function in the output layer.
- Feature Extraction: In some networks, fully connected layers are used to extract high-level features before passing them to the final output layer.
Conclusion
Fully connected layers are crucial components in deep learning models, enabling the network to learn and make predictions based on the combined features from previous layers. They are versatile and can be used in various neural network architectures to solve a wide range of tasks.