Skip to content

Batching and Fusing: Concepts, Math, and Practical Impact

Batching and fusing are key techniques for optimizing neural network computations, particularly matrix multiplications.

Batching

  • Concept: Instead of processing one input at a time, batching processes multiple inputs simultaneously. Each input in the batch is represented as a row in a matrix, allowing a single matrix multiplication to compute the outputs for all batch items in parallel[2][5][7].

  • Math:
    If $$ X $$ is a $$ b \times m $$ input matrix (batch size $$ b $$, input features $$ m $$), and $$ W $$ is an $$ m \times n $$ weight matrix (output features $$ n $$), then the output is: $$ Z = XW $$ $$ Z $$ is a $$ b \times n $$ matrix, where each row is the output for one input in the batch[5][7].

  • Impact:

  • Maximizes hardware utilization (especially on GPUs/TPUs).
  • Reduces overhead from repeated kernel launches or function calls.
  • Enables efficient memory access patterns due to contiguous storage[5].
  • Essential for high-throughput training and inference.

Fusing

  • Concept: Fusing combines multiple sequential operations (like matrix multiplication, bias addition, and activation) into a single computational kernel.

  • Math:
    Instead of computing: $$ Z = XW \ Z' = Z + b \ A = \text{activation}(Z') $$ Fusing performs all steps in a single pass, reducing the need to write/read intermediate results to memory.

  • Impact:

  • Reduces memory bandwidth usage by minimizing intermediate reads/writes.
  • Lowers kernel launch overhead.
  • Improves cache locality and overall throughput, especially for deep or wide networks.

Practical Implementation

  • Modern frameworks (PyTorch, TensorFlow) automatically handle batching and often fuse common patterns like linear + bias + activation.
  • When implementing custom kernels, ensure input matrices are stored in row-major order for efficient batching[5].
  • Always verify input/output shapes to avoid common dimension mismatches[6].

Summary Table

Technique Main Benefit Typical Use Case
Batching Maximizes parallelism, speed Training/inference with batches
Fusing Reduces memory ops, overhead Linear + bias + activation

Batching and fusing are foundational for efficient neural network computation, enabling high throughput and effective hardware utilization[4][5].

Citations 1:

  • [1] https://christopher5106.github.io/deep/learning/2018/10/28/understand-batch-matrix-multiplication.html
  • [2] https://www.gilesthomas.com/2025/02/basic-neural-network-matrix-maths-part-1
  • [3] https://massedcompute.com/faq-answers/?question=Can+you+explain+the+difference+between+matrix+multiplication+and+batch+matrix+multiplication%3F
  • [4] https://docs.nvidia.com/deeplearning/performance/dl-performance-matrix-multiplication/index.html
  • [5] https://www.gilesthomas.com/2025/02/basic-neural-network-matrix-maths-part-2
  • [6] https://www.youtube.com/watch?v=weDmbIFDyJE
  • [7] https://modelpredict.com/batched-backpropagation-connecting-math-and-code/

Practical Use Case: Batched & Fused Linear Layer in Python (NumPy)

Here's a minimal example of a batched matrix multiplication with bias addition and activation (fusing) using NumPy.

import numpy as np

def relu(x):
    return np.maximum(0, x)

def linear_fused(X, W, b):
    # X: (batch_size, input_dim)
    # W: (input_dim, output_dim)
    # b: (output_dim,)
    # Fused: matmul + bias + activation
    return relu(np.dot(X, W) + b)

# Example data
batch_size, input_dim, output_dim = 4, 3, 2
X = np.random.randn(batch_size, input_dim)
W = np.random.randn(input_dim, output_dim)
b = np.random.randn(output_dim)

# Batched and fused forward pass
output = linear_fused(X, W, b)
print("Output:\n", output)

Explanation: - X is a batch of 4 samples, each with 3 features. - W is the weight matrix, and b is the bias. - linear_fused computes all outputs in one step: matrix multiply, add bias, and apply ReLU activation—demonstrating both batching and fusing.

Impact: - Processes all samples in the batch simultaneously (batching). - Reduces memory operations and improves cache locality by fusing steps. - This approach is the basis for efficient implementations in deep learning libraries[1][3][4].

Citations 2:

  • [1] https://realpython.com/python-ai-neural-network/
  • [2] https://www.machinelearningmastery.com/tutorial-first-neural-network-python-keras/
  • [3] https://www.kaggle.com/code/shivamb/a-very-comprehensive-tutorial-nn-cnn
  • [4] https://www.quarkml.com/2023/07/build-a-cnn-from-scratch-using-python.html
  • [5] https://machinelearningmastery.com/kernel-methods-machine-learning-python/
  • [6] https://www.youtube.com/watch?v=Lakz2MoHy6o
  • [7] https://scikit-learn.org/stable/modules/neural_networks_supervised.html

Source (Perplexity)