◇LLM Notes12 notes

Editor · Graph · ◐

10/28/2026

Waning Gibbous

Deep Learning Fundamentals

Reasoning and Long Context

Large Language Model Overview

Transformer Architecture

Attention Mechanism

Pre-training and Fine-tuning

Prompt Engineering

RAG

Model Evaluation

AI Safety and Alignment

Generative AI Applications

Multimodal Large Models

+ New Note · Manage

Edit · Preview

Delete

Deep Learning Fundamentals

Deep Learning Fundamentals

Deep Learning is a subfield of machine learning that uses multi-layer neural networks to learn hierarchical representations of data. It is the technical foundation of large language models.

Neural Network Basics

Perceptron

The simplest neural network unit:

y = σ(w·x + b)

where σ is the activation function, w the weights, and b the bias.

Multi-Layer Perceptron (MLP)

Stack perceptrons into layers:

Input layer → Hidden layer → Hidden layer → Output layer

• Each hidden layer learns a different level of abstraction
• Deeper networks are more expressive (but harder to train)

Core Components

Activation Functions

Function	Formula	Notes
ReLU	max(0, x)	Cheap to compute, mitigates vanishing gradients
Sigmoid	1/(1+e^(-x))	Outputs 0-1, suited to binary classification
Tanh	(e^x - e^(-x))/(e^x + e^(-x))	Outputs -1~1, zero-centered
GELU	x·Φ(x)	The standard activation in Transformers
Softmax	e^(x_i)/Σe^(x_j)	Multi-class output layer

Loss Functions

• Cross-entropy loss: classification tasks
• Mean squared error (MSE): regression tasks
• Contrastive loss: contrastive learning

Optimizers

• SGD: stochastic gradient descent, the most basic
• Adam: adaptive learning rates, the most common
• AdamW: Adam with weight decay, the standard for LLM training

Key Training Techniques

Backpropagation

Compute gradients via the chain rule, updating parameters layer by layer from output back to input.

Regularization

Technique	Purpose
Dropout	Randomly drop neurons to prevent overfitting
Weight decay	Constrain parameter magnitudes
Layer normalization	Stabilize per-layer distributions
Residual connections	Ease vanishing gradients, enable deep networks

Learning Rate Schedules

• Warmup: gradually increase the learning rate early in training
• Cosine decay: anneal the learning rate along a cosine curve
• Scheduling: adjust dynamically based on validation performance

From Deep Learning to LLMs

Deep learning → Sequence models (RNN/LSTM) → Transformer → Pre-training + fine-tuning → Large language models

Scaling Laws

Model performance improves predictably as you scale:

• Number of parameters
• Amount of training data
• Compute (FLOPs)

"The Bitter Lesson": in the long run, compute and scale beat hand-crafted cleverness.

See also Transformer Architecture, Attention Mechanism, Pre-training and Fine-tuning

Links:Transformer ArchitectureAttention MechanismPre-training and Fine-tuning