Vision Models Format

The expected input formats for vision models vary across deep learning frameworks. Below is a comparison of common frameworks and their typical requirements:

Framework	Expected Shape	Channel Order	Input Type	Key Notes
PyTorch	(N, C, H, W)	NCHW	Float tensor	Default for torchvision.models [User Context]
TensorFlow	(N, H, W, C)	NHWC	Float array	Some models (e.g., Hugging Face ViT) may use NCHW via transposition 1
MXNet	(N, C, H, W)	NCHW	Binary JPEG/NDArray	Pre-trained models like SqueezeNet expect (1, 3, 224, 224) 3
Caffe	(N, C, H, W)	NCHW	Blob (float32)	Uses caffe.io.load_image with CHW transformation 4
ONNX	(N, C, H, W)	NCHW	Preprocessed float32	Requires normalization + resizing (e.g., (1, 3, 224, 224)) 5

Where: - N: Batch size — the number of images processed at once (here, a single image).

• C: Number of channels — typically 3 for RGB images.
• W: Image height (in pixels).
• H: Image width (in pixels).

Key Observations:

1. Channel Order Differences:
   • PyTorch/MXNet/Caffe/ONNX use channel-first (NCHW)
   • TensorFlow/Keras defaults to channel-last (NHWC) unless modified 2
2. Preprocessing Requirements:
   • MXNet and ONNX often require explicit resizing/normalization
   • TensorFlow models may need axis permutation for NCHW compatibility 1
   • Caffe uses dedicated blob storage with CHW ordering 4

3. Batch Support:

   All frameworks support batch processing via the leading dimension N, though some models require fixed batch sizes during deployment.

For your original example (1, 3, 1024, 2048), this matches PyTorch's NCHW format directly. To use this input with TensorFlow, you would need to permute the axes to (1, 1024, 2048, 3) or modify the model's input signature 12.

References

1. https://huggingface.co/blog/tf-serving-vision
2. https://pyimagesearch.com/2019/06/24/change-input-shape-dimensions-for-fine-tuning-with-keras/
3. https://awslabs.github.io/multi-model-server/examples/mxnet_vision/
4. https://github.com/arundasan91/Deep-Learning-with-Caffe/blob/master/Deep-Neural-Network-with-Caffe/Deep%20Neural%20Network%20with%20Caffe.md
5. https://docs.azure.cn/en-us/machine-learning/how-to-inference-onnx-automl-image-models?view=azureml-api-2
6. https://discuss.pytorch.org/t/input-format-for-pretrained-torchvision-models/48759
7. https://docs.pytorch.org/vision/main/models.html
8. https://pytorch.org/vision/0.9/models
9. https://www.learnpytorch.io/03_pytorch_computer_vision/
10. https://www.youtube.com/watch?v=vAmKB7iPkWw