Overview

you will learn all these Topics and lot more 1. Core Concepts1. Perceptron2. MLP and its Notation3. Forward Propagation4. Backpropagation5. Chain Rule of Derivative in Backpropagation6. Vanishing Gradient Problem7. Exploding GradientActivation FunctionsList of Activation Functions1. Linear Function2. Binary Step Function3. Sigmoid Function (Logistic Function)4. Tanh (Hyperbolic Tangent Function)5. ReLU (Rectified Linear Unit)6. Leaky ReLU7. Parametric ReLU (PReLU)8. Exponential Linear Unit (ELU)9. Scaled Exponential Linear Unit (SELU)10. Softmax11. Swish.12. SoftPlus13. Mish14. Maxout15. GELU (Gaussian Error Linear Unit)16. SiLU (Sigmoid Linear Unit)17. Gated Linear Unit (GLU)18. SwiGLU19. Mish Activation FunctionDerivative of Activation FunctionsProperties of Activation Functions1. Saturating vs Non-Saturating2. Smooth vs Non-Smooth3. Generalized vs Specialized4. Underflow and Overflow5. Undefined and Defined6. Computationally Expensive vs Inexpensive.7. 0-Centered and Non-0-Centered8. Differentiable vs Non-Differentiable9. Bounded and Unbounded10. Monotonicity11. Linear Vs Non LinearIdeal Activation Function Characteristics1. Non-Linearity2. Differentiability3. Computational Efficiency4. Avoids Saturation5. Non-Sparse (Dense) Gradients6. Centered Output (0-Centered)7. Prevents Exploding Gradients8. Monotonicity (Optional)9. Sparse Activations (Optional)10. Resilience to Outliers11. Noise Robustness12. Stable Training Dynamics13. Minimal Parameter Dependency14. Compatibility with Modern Techniques15. Efficient in Hardware16. The Function Must Be Continuous and Infinite in Domain17. Vanishing Gradient Problem18. Dynamic Range Adaptation19. Scalability to Deeper Networks20. Biological Plausibility (Optional)21. Simplicity in Implementation22. Gradient Smoothness23. Compatibility with Unsupervised ObjectivesLoss Functions1. Mean Squared Error (MSE)2. Mean Absolute Error (MAE)3. Root Mean Squared Error (RMSE)4. Root Mean Squared Log Error (RMSLE)5. Huber Loss6. Hinge Loss7. Binary Cross-Entropy (BCE)8. Categorical Cross-Entropy9. Focal Loss10. Contrastive Loss11. KL Divergence (Kullback-Leibler Divergence)12. Triplet Loss13. Smooth L1 Loss:14. Dice Loss:Optimizers1. Gradient Descent2. Stochastic Gradient Descent (SGD)3. Mini-Batch Gradient Descent4. Exponentially Weighted Moving Average (EWMA)5. Gradient Descent with Momentum6. Nesterov Accelerated Gradient7. AdaGrad (Adaptive Gradient)8. RMSProp (Root Mean Squared Propagation)9. AdaDelta10. Adam (Adaptive Moment Estimation)11. Nadam (Nesterov-accelerated Adaptive Moment Estimation)12. LAMB (Layer-wise Adaptive Moments):13. SGDW/AdamWImproving Performance of Neural Networks· Effect of Batch Size on Training· MemoizationWeight Initialization1. Zero Initialization2. Non-Zero Constant Value Initialization3. Random Initialization (with small values, large values)4. Xavier (Glorot) Initialization5. He Initialization6. LeCun Initialization7. Uniform Initialization8. Normal (Gaussian) Initialization9. Bilinear Initialization10. Orthogonal InitializationRegularization1. L1 Regularization (Lasso)2. L2 Regularization (Ridge) (weight decay)3. Elastic Net Regularization4. Dropout5. Early Stopping6. Data Augmentation7. Batch Normalization8. Residual Connections9. Label Smoothing10. Parameter Sharing11. Weight Constraint12. Adversarial TrainingNormalization1. Normalizing Inputs2. Batch Normalization (BatchNorm)3. Layer Normalization (LayerNorm)4. Instance Normalization (InstanceNorm)5. Group Normalization (GroupNorm)6. RMSNorm7. Filter Response Normalization8. Weight NormalizationOther TechniquesGradient Clipping and Gradient Checking Hyperparameter TuningLearning Rate Scheduling1. Step Decay2. Exponential Decay3. Cosine Annealing4. Cyclical learning rate5. OneCycleLR6. WarmupRecurrent Neural Networks (RNNs)1. RNN2. LSTM3. GRU4. Deep Stacked RNN, BidirectionalSequence-to-Sequence Models1. Encoder-Decoder Architecture2. Attention MechanismNatural Language Processing (NLP)· Tokenization: Sentence tokenization, word tokenization, and subword tokenization (BPE, WordPiece).Text Preprocessing: Lowercasing, stemming, lemmatization, stopword removal etc...Text Vectorization:1. One-Hot Encoding2. Bag of Words (BoW)3. TF-IDF4. Word Embeddings (Word2Vec, GloVe, FastText)5. Contextual Embeddings (ELMo, BERT, GPT, etc.) Complete NLP BasicsTransformers1. Vanilla Transformer2. Vision Transformer3. Swin Transformerand lot more

Skills

Reviews