Vision Transformer (ViT) Benchmark for Image Recognition Tasks¶

In [2]:

# Install the required libraries
!pip install transformers
!pip install transformers datasets
!pip install torch
!pip install torchvision
!pip install pillow

# Install the required libraries !pip install transformers !pip install transformers datasets !pip install torch !pip install torchvision !pip install pillow

Requirement already satisfied: transformers in /usr/local/lib/python3.10/dist-packages (4.41.2)
Requirement already satisfied: filelock in /usr/local/lib/python3.10/dist-packages (from transformers) (3.14.0)
Requirement already satisfied: huggingface-hub<1.0,>=0.23.0 in /usr/local/lib/python3.10/dist-packages (from transformers) (0.23.2)
Requirement already satisfied: numpy>=1.17 in /usr/local/lib/python3.10/dist-packages (from transformers) (1.25.2)
Requirement already satisfied: packaging>=20.0 in /usr/local/lib/python3.10/dist-packages (from transformers) (24.0)
Requirement already satisfied: pyyaml>=5.1 in /usr/local/lib/python3.10/dist-packages (from transformers) (6.0.1)
Requirement already satisfied: regex!=2019.12.17 in /usr/local/lib/python3.10/dist-packages (from transformers) (2024.5.15)
Requirement already satisfied: requests in /usr/local/lib/python3.10/dist-packages (from transformers) (2.31.0)
Requirement already satisfied: tokenizers<0.20,>=0.19 in /usr/local/lib/python3.10/dist-packages (from transformers) (0.19.1)
Requirement already satisfied: safetensors>=0.4.1 in /usr/local/lib/python3.10/dist-packages (from transformers) (0.4.3)
Requirement already satisfied: tqdm>=4.27 in /usr/local/lib/python3.10/dist-packages (from transformers) (4.66.4)
Requirement already satisfied: fsspec>=2023.5.0 in /usr/local/lib/python3.10/dist-packages (from huggingface-hub<1.0,>=0.23.0->transformers) (2023.6.0)
Requirement already satisfied: typing-extensions>=3.7.4.3 in /usr/local/lib/python3.10/dist-packages (from huggingface-hub<1.0,>=0.23.0->transformers) (4.12.1)
Requirement already satisfied: charset-normalizer<4,>=2 in /usr/local/lib/python3.10/dist-packages (from requests->transformers) (3.3.2)
Requirement already satisfied: idna<4,>=2.5 in /usr/local/lib/python3.10/dist-packages (from requests->transformers) (3.7)
Requirement already satisfied: urllib3<3,>=1.21.1 in /usr/local/lib/python3.10/dist-packages (from requests->transformers) (2.0.7)
Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.10/dist-packages (from requests->transformers) (2024.6.2)
Requirement already satisfied: transformers in /usr/local/lib/python3.10/dist-packages (4.41.2)
Collecting datasets
  Downloading datasets-2.19.2-py3-none-any.whl (542 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 542.1/542.1 kB 9.0 MB/s eta 0:00:00
Requirement already satisfied: filelock in /usr/local/lib/python3.10/dist-packages (from transformers) (3.14.0)
Requirement already satisfied: huggingface-hub<1.0,>=0.23.0 in /usr/local/lib/python3.10/dist-packages (from transformers) (0.23.2)
Requirement already satisfied: numpy>=1.17 in /usr/local/lib/python3.10/dist-packages (from transformers) (1.25.2)
Requirement already satisfied: packaging>=20.0 in /usr/local/lib/python3.10/dist-packages (from transformers) (24.0)
Requirement already satisfied: pyyaml>=5.1 in /usr/local/lib/python3.10/dist-packages (from transformers) (6.0.1)
Requirement already satisfied: regex!=2019.12.17 in /usr/local/lib/python3.10/dist-packages (from transformers) (2024.5.15)
Requirement already satisfied: requests in /usr/local/lib/python3.10/dist-packages (from transformers) (2.31.0)
Requirement already satisfied: tokenizers<0.20,>=0.19 in /usr/local/lib/python3.10/dist-packages (from transformers) (0.19.1)
Requirement already satisfied: safetensors>=0.4.1 in /usr/local/lib/python3.10/dist-packages (from transformers) (0.4.3)
Requirement already satisfied: tqdm>=4.27 in /usr/local/lib/python3.10/dist-packages (from transformers) (4.66.4)
Requirement already satisfied: pyarrow>=12.0.0 in /usr/local/lib/python3.10/dist-packages (from datasets) (14.0.2)
Requirement already satisfied: pyarrow-hotfix in /usr/local/lib/python3.10/dist-packages (from datasets) (0.6)
Collecting dill<0.3.9,>=0.3.0 (from datasets)
  Downloading dill-0.3.8-py3-none-any.whl (116 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 116.3/116.3 kB 7.7 MB/s eta 0:00:00
Requirement already satisfied: pandas in /usr/local/lib/python3.10/dist-packages (from datasets) (2.0.3)
Collecting requests (from transformers)
  Downloading requests-2.32.3-py3-none-any.whl (64 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 64.9/64.9 kB 7.0 MB/s eta 0:00:00
Collecting xxhash (from datasets)
  Downloading xxhash-3.4.1-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (194 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 194.1/194.1 kB 11.2 MB/s eta 0:00:00
Collecting multiprocess (from datasets)
  Downloading multiprocess-0.70.16-py310-none-any.whl (134 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 134.8/134.8 kB 11.5 MB/s eta 0:00:00
Requirement already satisfied: fsspec[http]<=2024.3.1,>=2023.1.0 in /usr/local/lib/python3.10/dist-packages (from datasets) (2023.6.0)
Requirement already satisfied: aiohttp in /usr/local/lib/python3.10/dist-packages (from datasets) (3.9.5)
Requirement already satisfied: aiosignal>=1.1.2 in /usr/local/lib/python3.10/dist-packages (from aiohttp->datasets) (1.3.1)
Requirement already satisfied: attrs>=17.3.0 in /usr/local/lib/python3.10/dist-packages (from aiohttp->datasets) (23.2.0)
Requirement already satisfied: frozenlist>=1.1.1 in /usr/local/lib/python3.10/dist-packages (from aiohttp->datasets) (1.4.1)
Requirement already satisfied: multidict<7.0,>=4.5 in /usr/local/lib/python3.10/dist-packages (from aiohttp->datasets) (6.0.5)
Requirement already satisfied: yarl<2.0,>=1.0 in /usr/local/lib/python3.10/dist-packages (from aiohttp->datasets) (1.9.4)
Requirement already satisfied: async-timeout<5.0,>=4.0 in /usr/local/lib/python3.10/dist-packages (from aiohttp->datasets) (4.0.3)
Requirement already satisfied: typing-extensions>=3.7.4.3 in /usr/local/lib/python3.10/dist-packages (from huggingface-hub<1.0,>=0.23.0->transformers) (4.12.1)
Requirement already satisfied: charset-normalizer<4,>=2 in /usr/local/lib/python3.10/dist-packages (from requests->transformers) (3.3.2)
Requirement already satisfied: idna<4,>=2.5 in /usr/local/lib/python3.10/dist-packages (from requests->transformers) (3.7)
Requirement already satisfied: urllib3<3,>=1.21.1 in /usr/local/lib/python3.10/dist-packages (from requests->transformers) (2.0.7)
Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.10/dist-packages (from requests->transformers) (2024.6.2)
Requirement already satisfied: python-dateutil>=2.8.2 in /usr/local/lib/python3.10/dist-packages (from pandas->datasets) (2.8.2)
Requirement already satisfied: pytz>=2020.1 in /usr/local/lib/python3.10/dist-packages (from pandas->datasets) (2023.4)
Requirement already satisfied: tzdata>=2022.1 in /usr/local/lib/python3.10/dist-packages (from pandas->datasets) (2024.1)
Requirement already satisfied: six>=1.5 in /usr/local/lib/python3.10/dist-packages (from python-dateutil>=2.8.2->pandas->datasets) (1.16.0)
Installing collected packages: xxhash, requests, dill, multiprocess, datasets
  Attempting uninstall: requests
    Found existing installation: requests 2.31.0
    Uninstalling requests-2.31.0:
      Successfully uninstalled requests-2.31.0
ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
google-colab 1.0.0 requires requests==2.31.0, but you have requests 2.32.3 which is incompatible.
Successfully installed datasets-2.19.2 dill-0.3.8 multiprocess-0.70.16 requests-2.32.3 xxhash-3.4.1
Requirement already satisfied: torch in /usr/local/lib/python3.10/dist-packages (2.3.0+cu121)
Requirement already satisfied: filelock in /usr/local/lib/python3.10/dist-packages (from torch) (3.14.0)
Requirement already satisfied: typing-extensions>=4.8.0 in /usr/local/lib/python3.10/dist-packages (from torch) (4.12.1)
Requirement already satisfied: sympy in /usr/local/lib/python3.10/dist-packages (from torch) (1.12.1)
Requirement already satisfied: networkx in /usr/local/lib/python3.10/dist-packages (from torch) (3.3)
Requirement already satisfied: jinja2 in /usr/local/lib/python3.10/dist-packages (from torch) (3.1.4)
Requirement already satisfied: fsspec in /usr/local/lib/python3.10/dist-packages (from torch) (2023.6.0)
Requirement already satisfied: nvidia-cuda-nvrtc-cu12==12.1.105 in /usr/local/lib/python3.10/dist-packages (from torch) (12.1.105)
Requirement already satisfied: nvidia-cuda-runtime-cu12==12.1.105 in /usr/local/lib/python3.10/dist-packages (from torch) (12.1.105)
Requirement already satisfied: nvidia-cuda-cupti-cu12==12.1.105 in /usr/local/lib/python3.10/dist-packages (from torch) (12.1.105)
Requirement already satisfied: nvidia-cudnn-cu12==8.9.2.26 in /usr/local/lib/python3.10/dist-packages (from torch) (8.9.2.26)
Requirement already satisfied: nvidia-cublas-cu12==12.1.3.1 in /usr/local/lib/python3.10/dist-packages (from torch) (12.1.3.1)
Requirement already satisfied: nvidia-cufft-cu12==11.0.2.54 in /usr/local/lib/python3.10/dist-packages (from torch) (11.0.2.54)
Requirement already satisfied: nvidia-curand-cu12==10.3.2.106 in /usr/local/lib/python3.10/dist-packages (from torch) (10.3.2.106)
Requirement already satisfied: nvidia-cusolver-cu12==11.4.5.107 in /usr/local/lib/python3.10/dist-packages (from torch) (11.4.5.107)
Requirement already satisfied: nvidia-cusparse-cu12==12.1.0.106 in /usr/local/lib/python3.10/dist-packages (from torch) (12.1.0.106)
Requirement already satisfied: nvidia-nccl-cu12==2.20.5 in /usr/local/lib/python3.10/dist-packages (from torch) (2.20.5)
Requirement already satisfied: nvidia-nvtx-cu12==12.1.105 in /usr/local/lib/python3.10/dist-packages (from torch) (12.1.105)
Requirement already satisfied: triton==2.3.0 in /usr/local/lib/python3.10/dist-packages (from torch) (2.3.0)
Requirement already satisfied: nvidia-nvjitlink-cu12 in /usr/local/lib/python3.10/dist-packages (from nvidia-cusolver-cu12==11.4.5.107->torch) (12.5.40)
Requirement already satisfied: MarkupSafe>=2.0 in /usr/local/lib/python3.10/dist-packages (from jinja2->torch) (2.1.5)
Requirement already satisfied: mpmath<1.4.0,>=1.1.0 in /usr/local/lib/python3.10/dist-packages (from sympy->torch) (1.3.0)
Requirement already satisfied: torchvision in /usr/local/lib/python3.10/dist-packages (0.18.0+cu121)
Requirement already satisfied: numpy in /usr/local/lib/python3.10/dist-packages (from torchvision) (1.25.2)
Requirement already satisfied: torch==2.3.0 in /usr/local/lib/python3.10/dist-packages (from torchvision) (2.3.0+cu121)
Requirement already satisfied: pillow!=8.3.*,>=5.3.0 in /usr/local/lib/python3.10/dist-packages (from torchvision) (9.4.0)
Requirement already satisfied: filelock in /usr/local/lib/python3.10/dist-packages (from torch==2.3.0->torchvision) (3.14.0)
Requirement already satisfied: typing-extensions>=4.8.0 in /usr/local/lib/python3.10/dist-packages (from torch==2.3.0->torchvision) (4.12.1)
Requirement already satisfied: sympy in /usr/local/lib/python3.10/dist-packages (from torch==2.3.0->torchvision) (1.12.1)
Requirement already satisfied: networkx in /usr/local/lib/python3.10/dist-packages (from torch==2.3.0->torchvision) (3.3)
Requirement already satisfied: jinja2 in /usr/local/lib/python3.10/dist-packages (from torch==2.3.0->torchvision) (3.1.4)
Requirement already satisfied: fsspec in /usr/local/lib/python3.10/dist-packages (from torch==2.3.0->torchvision) (2023.6.0)
Requirement already satisfied: nvidia-cuda-nvrtc-cu12==12.1.105 in /usr/local/lib/python3.10/dist-packages (from torch==2.3.0->torchvision) (12.1.105)
Requirement already satisfied: nvidia-cuda-runtime-cu12==12.1.105 in /usr/local/lib/python3.10/dist-packages (from torch==2.3.0->torchvision) (12.1.105)
Requirement already satisfied: nvidia-cuda-cupti-cu12==12.1.105 in /usr/local/lib/python3.10/dist-packages (from torch==2.3.0->torchvision) (12.1.105)
Requirement already satisfied: nvidia-cudnn-cu12==8.9.2.26 in /usr/local/lib/python3.10/dist-packages (from torch==2.3.0->torchvision) (8.9.2.26)
Requirement already satisfied: nvidia-cublas-cu12==12.1.3.1 in /usr/local/lib/python3.10/dist-packages (from torch==2.3.0->torchvision) (12.1.3.1)
Requirement already satisfied: nvidia-cufft-cu12==11.0.2.54 in /usr/local/lib/python3.10/dist-packages (from torch==2.3.0->torchvision) (11.0.2.54)
Requirement already satisfied: nvidia-curand-cu12==10.3.2.106 in /usr/local/lib/python3.10/dist-packages (from torch==2.3.0->torchvision) (10.3.2.106)
Requirement already satisfied: nvidia-cusolver-cu12==11.4.5.107 in /usr/local/lib/python3.10/dist-packages (from torch==2.3.0->torchvision) (11.4.5.107)
Requirement already satisfied: nvidia-cusparse-cu12==12.1.0.106 in /usr/local/lib/python3.10/dist-packages (from torch==2.3.0->torchvision) (12.1.0.106)
Requirement already satisfied: nvidia-nccl-cu12==2.20.5 in /usr/local/lib/python3.10/dist-packages (from torch==2.3.0->torchvision) (2.20.5)
Requirement already satisfied: nvidia-nvtx-cu12==12.1.105 in /usr/local/lib/python3.10/dist-packages (from torch==2.3.0->torchvision) (12.1.105)
Requirement already satisfied: triton==2.3.0 in /usr/local/lib/python3.10/dist-packages (from torch==2.3.0->torchvision) (2.3.0)
Requirement already satisfied: nvidia-nvjitlink-cu12 in /usr/local/lib/python3.10/dist-packages (from nvidia-cusolver-cu12==11.4.5.107->torch==2.3.0->torchvision) (12.5.40)
Requirement already satisfied: MarkupSafe>=2.0 in /usr/local/lib/python3.10/dist-packages (from jinja2->torch==2.3.0->torchvision) (2.1.5)
Requirement already satisfied: mpmath<1.4.0,>=1.1.0 in /usr/local/lib/python3.10/dist-packages (from sympy->torch==2.3.0->torchvision) (1.3.0)
Requirement already satisfied: pillow in /usr/local/lib/python3.10/dist-packages (9.4.0)

In [3]:

# Import necessary libraries
import torch
from transformers import ViTFeatureExtractor, ViTForImageClassification
from PIL import Image
import requests

# Import necessary libraries import torch from transformers import ViTFeatureExtractor, ViTForImageClassification from PIL import Image import requests

Image Classification¶

In [4]:

# Load the pre-trained ViT model and feature extractor
model = ViTForImageClassification.from_pretrained('google/vit-base-patch16-224')
feature_extractor = ViTFeatureExtractor.from_pretrained('google/vit-base-patch16-224')

# Load an example image
url = 'http://images.cocodataset.org/val2017/000000039769.jpg'
image = Image.open(requests.get(url, stream=True).raw)

# Prepare the image
inputs = feature_extractor(images=image, return_tensors="pt")

# Perform inference
outputs = model(**inputs)
logits = outputs.logits
predicted_class_idx = logits.argmax(-1).item()
print(f'Predicted class: {model.config.id2label[predicted_class_idx]}')

# Load the pre-trained ViT model and feature extractor model = ViTForImageClassification.from_pretrained('google/vit-base-patch16-224') feature_extractor = ViTFeatureExtractor.from_pretrained('google/vit-base-patch16-224') # Load an example image url = 'http://images.cocodataset.org/val2017/000000039769.jpg' image = Image.open(requests.get(url, stream=True).raw) # Prepare the image inputs = feature_extractor(images=image, return_tensors="pt") # Perform inference outputs = model(**inputs) logits = outputs.logits predicted_class_idx = logits.argmax(-1).item() print(f'Predicted class: {model.config.id2label[predicted_class_idx]}')

/usr/local/lib/python3.10/dist-packages/huggingface_hub/utils/_token.py:89: UserWarning: 
The secret `HF_TOKEN` does not exist in your Colab secrets.
To authenticate with the Hugging Face Hub, create a token in your settings tab (https://huggingface.co/settings/tokens), set it as secret in your Google Colab and restart your session.
You will be able to reuse this secret in all of your notebooks.
Please note that authentication is recommended but still optional to access public models or datasets.
  warnings.warn(

config.json:   0%|          | 0.00/69.7k [00:00<?, ?B/s]

model.safetensors:   0%|          | 0.00/346M [00:00<?, ?B/s]

preprocessor_config.json:   0%|          | 0.00/160 [00:00<?, ?B/s]

/usr/local/lib/python3.10/dist-packages/transformers/models/vit/feature_extraction_vit.py:28: FutureWarning: The class ViTFeatureExtractor is deprecated and will be removed in version 5 of Transformers. Please use ViTImageProcessor instead.
  warnings.warn(

Predicted class: Egyptian cat

Object detection¶

In [ ]:

!pip install transformers
!pip install timm

!pip install transformers !pip install timm

In [ ]:

import torch
import requests
from PIL import Image, ImageDraw
from transformers import DetrImageProcessor, DetrForObjectDetection

# Load pre-trained model and processor
model = DetrForObjectDetection.from_pretrained('facebook/detr-resnet-50')
processor = DetrImageProcessor.from_pretrained('facebook/detr-resnet-50')

# Load an example image
url = 'http://images.cocodataset.org/val2017/000000039769.jpg'
image = Image.open(requests.get(url, stream=True).raw)

# Prepare the image
inputs = processor(images=image, return_tensors="pt")

# Perform inference
outputs = model(**inputs)

# Process the outputs
target_sizes = torch.tensor([image.size[::-1]])
results = processor.post_process_object_detection(outputs, target_sizes=target_sizes)[0]

# Draw boxes on the image
draw = ImageDraw.Draw(image)
for score, label, box in zip(results["scores"], results["labels"], results["boxes"]):
    if score > 0.9:
        box = [round(i, 2) for i in box.tolist()]
        draw.rectangle(box, outline="red", width=3)
        draw.text((box[0], box[1]), f'{model.config.id2label[label.item()]}: {round(score.item(), 2)}', fill="red")

image.show()

import torch import requests from PIL import Image, ImageDraw from transformers import DetrImageProcessor, DetrForObjectDetection # Load pre-trained model and processor model = DetrForObjectDetection.from_pretrained('facebook/detr-resnet-50') processor = DetrImageProcessor.from_pretrained('facebook/detr-resnet-50') # Load an example image url = 'http://images.cocodataset.org/val2017/000000039769.jpg' image = Image.open(requests.get(url, stream=True).raw) # Prepare the image inputs = processor(images=image, return_tensors="pt") # Perform inference outputs = model(**inputs) # Process the outputs target_sizes = torch.tensor([image.size[::-1]]) results = processor.post_process_object_detection(outputs, target_sizes=target_sizes)[0] # Draw boxes on the image draw = ImageDraw.Draw(image) for score, label, box in zip(results["scores"], results["labels"], results["boxes"]): if score > 0.9: box = [round(i, 2) for i in box.tolist()] draw.rectangle(box, outline="red", width=3) draw.text((box[0], box[1]), f'{model.config.id2label[label.item()]}: {round(score.item(), 2)}', fill="red") image.show()

Semantic segmentation¶

In [ ]:

!pip install transformers

!pip install transformers

In [ ]:

import torch
from transformers import SegformerForSemanticSegmentation, SegformerFeatureExtractor
from PIL import Image
import requests
import numpy as np

# Load pre-trained model and feature extractor
model = SegformerForSemanticSegmentation.from_pretrained('nvidia/segformer-b0-finetuned-ade-512-512')
feature_extractor = SegformerFeatureExtractor.from_pretrained('nvidia/segformer-b0-finetuned-ade-512-512')

# Load an example image
url = 'http://images.cocodataset.org/val2017/000000039769.jpg'
image = Image.open(requests.get(url, stream=True).raw)

# Prepare the image
inputs = feature_extractor(images=image, return_tensors="pt")

# Perform inference
outputs = model(**inputs)
logits = outputs.logits  # shape (batch_size, num_classes, height, width)

# Get the predicted class for each pixel
seg = torch.argmax(logits, dim=1)[0]  # shape (height, width)

# Convert to a PIL image
segmentation = Image.fromarray(seg.byte().cpu().numpy())

# Display the segmentation map
segmentation.show()

import torch from transformers import SegformerForSemanticSegmentation, SegformerFeatureExtractor from PIL import Image import requests import numpy as np # Load pre-trained model and feature extractor model = SegformerForSemanticSegmentation.from_pretrained('nvidia/segformer-b0-finetuned-ade-512-512') feature_extractor = SegformerFeatureExtractor.from_pretrained('nvidia/segformer-b0-finetuned-ade-512-512') # Load an example image url = 'http://images.cocodataset.org/val2017/000000039769.jpg' image = Image.open(requests.get(url, stream=True).raw) # Prepare the image inputs = feature_extractor(images=image, return_tensors="pt") # Perform inference outputs = model(**inputs) logits = outputs.logits # shape (batch_size, num_classes, height, width) # Get the predicted class for each pixel seg = torch.argmax(logits, dim=1)[0] # shape (height, width) # Convert to a PIL image segmentation = Image.fromarray(seg.byte().cpu().numpy()) # Display the segmentation map segmentation.show()

Instance Segmentation¶

In [ ]:

!pip install pyyaml==5.1
!pip install detectron2 -f https://dl.fbaipublicfiles.com/detectron2/wheels/cu113/torch1.10/index.html

!pip install pyyaml==5.1 !pip install detectron2 -f https://dl.fbaipublicfiles.com/detectron2/wheels/cu113/torch1.10/index.html

In [ ]:

import torch
import detectron2
from detectron2.engine import DefaultPredictor
from detectron2.config import get_cfg
from detectron2.utils.visualizer import Visualizer
from detectron2.data import MetadataCatalog
import cv2
import matplotlib.pyplot as plt

# Load the config and weights
cfg = get_cfg()
cfg.merge_from_file(detectron2.model_zoo.get_config_file("COCO-InstanceSegmentation/mask_rcnn_R_50_FPN_3x.yaml"))
cfg.MODEL.ROI_HEADS.SCORE_THRESH_TEST = 0.5  # set the threshold for this model
cfg.MODEL.WEIGHTS = detectron2.model_zoo.get_checkpoint_url("COCO-InstanceSegmentation/mask_rcnn_R_50_FPN_3x.yaml")
predictor = DefaultPredictor(cfg)

# Load an image
img_path = 'http://images.cocodataset.org/val2017/000000039769.jpg'
im = cv2.imread(img_path)

# Perform inference
outputs = predictor(im)

# Visualize the results
v = Visualizer(im[:, :, ::-1], MetadataCatalog.get(cfg.DATASETS.TRAIN[0]), scale=1.2)
out = v.draw_instance_predictions(outputs["instances"].to("cpu"))

plt.figure(figsize=(14, 10))
plt.imshow(out.get_image()[:, :, ::-1])
plt.show()

import torch import detectron2 from detectron2.engine import DefaultPredictor from detectron2.config import get_cfg from detectron2.utils.visualizer import Visualizer from detectron2.data import MetadataCatalog import cv2 import matplotlib.pyplot as plt # Load the config and weights cfg = get_cfg() cfg.merge_from_file(detectron2.model_zoo.get_config_file("COCO-InstanceSegmentation/mask_rcnn_R_50_FPN_3x.yaml")) cfg.MODEL.ROI_HEADS.SCORE_THRESH_TEST = 0.5 # set the threshold for this model cfg.MODEL.WEIGHTS = detectron2.model_zoo.get_checkpoint_url("COCO-InstanceSegmentation/mask_rcnn_R_50_FPN_3x.yaml") predictor = DefaultPredictor(cfg) # Load an image img_path = 'http://images.cocodataset.org/val2017/000000039769.jpg' im = cv2.imread(img_path) # Perform inference outputs = predictor(im) # Visualize the results v = Visualizer(im[:, :, ::-1], MetadataCatalog.get(cfg.DATASETS.TRAIN[0]), scale=1.2) out = v.draw_instance_predictions(outputs["instances"].to("cpu")) plt.figure(figsize=(14, 10)) plt.imshow(out.get_image()[:, :, ::-1]) plt.show()

Action Recognition¶

In [ ]:

!pip install pytorchvideo

!pip install pytorchvideo

In [ ]:

import torch
from pytorchvideo.models.hub import slowfast_r50

# Load pre-trained SlowFast model
model = slowfast_r50(pretrained=True)

# Assuming we have a video tensor of shape (B, C, T, H, W)
# Here B is batch size, C is number of channels, T is time frames, H and W are height and width
# Example: Dummy tensor for demonstration
video = torch.randn(1, 3, 32, 224, 224)  # Adjust dimensions according to your video

# Perform inference
model = model.eval()
with torch.no_grad():
    outputs = model(video)
    predicted_class_idx = torch.argmax(outputs, dim=1)
    print(f'Predicted action class: {predicted_class_idx.item()}')

import torch from pytorchvideo.models.hub import slowfast_r50 # Load pre-trained SlowFast model model = slowfast_r50(pretrained=True) # Assuming we have a video tensor of shape (B, C, T, H, W) # Here B is batch size, C is number of channels, T is time frames, H and W are height and width # Example: Dummy tensor for demonstration video = torch.randn(1, 3, 32, 224, 224) # Adjust dimensions according to your video # Perform inference model = model.eval() with torch.no_grad(): outputs = model(video) predicted_class_idx = torch.argmax(outputs, dim=1) print(f'Predicted action class: {predicted_class_idx.item()}')

Video classification¶

In [ ]:

!pip install transformers

!pip install transformers

In [ ]:

from transformers import VideoMAEFeatureExtractor, VideoMAEForVideoClassification
import torch
import requests
from PIL import Image
from io import BytesIO

# Load pre-trained model and feature extractor
model = VideoMAEForVideoClassification.from_pretrained('MCG-NJU/videomae-base')
feature_extractor = VideoMAEFeatureExtractor.from_pretrained('MCG-NJU/videomae-base')

# Load a video
url = 'https://path_to_your_video.mp4'  # replace with your video URL
video = ...  # Load your video here

# Prepare the video
inputs = feature_extractor(video, return_tensors="pt")

# Perform inference
outputs = model(**inputs)
logits = outputs.logits
predicted_class_idx = logits.argmax(-1).item()
print(f'Predicted class: {model.config.id2label[predicted_class_idx]}')

from transformers import VideoMAEFeatureExtractor, VideoMAEForVideoClassification import torch import requests from PIL import Image from io import BytesIO # Load pre-trained model and feature extractor model = VideoMAEForVideoClassification.from_pretrained('MCG-NJU/videomae-base') feature_extractor = VideoMAEFeatureExtractor.from_pretrained('MCG-NJU/videomae-base') # Load a video url = 'https://path_to_your_video.mp4' # replace with your video URL video = ... # Load your video here # Prepare the video inputs = feature_extractor(video, return_tensors="pt") # Perform inference outputs = model(**inputs) logits = outputs.logits predicted_class_idx = logits.argmax(-1).item() print(f'Predicted class: {model.config.id2label[predicted_class_idx]}')

Video object detection¶

In [ ]:

# This code is highly dependent on the dataset and specific pre-trained models.
# For a demonstration, we'll use the SlowFast model again as an example.

!pip install pytorchvideo

# This code is highly dependent on the dataset and specific pre-trained models. # For a demonstration, we'll use the SlowFast model again as an example. !pip install pytorchvideo

In [ ]:

import torch
from pytorchvideo.models.hub import slowfast_r50

# Load pre-trained SlowFast model
model = slowfast_r50(pretrained=True)

# Assuming we have a video tensor of shape (B, C, T, H, W)
# Here B is batch size, C is number of channels, T is time frames, H and W are height and width
# Example: Dummy tensor for demonstration
video = torch.randn(1, 3, 32, 224, 224)  # Adjust dimensions according to your video

# Perform inference
model = model.eval()
with torch.no_grad():
    outputs = model(video)
    predicted_class_idx = torch.argmax(outputs, dim=1)
    print(f'Predicted action class: {predicted_class_idx.item()}')

import torch from pytorchvideo.models.hub import slowfast_r50 # Load pre-trained SlowFast model model = slowfast_r50(pretrained=True) # Assuming we have a video tensor of shape (B, C, T, H, W) # Here B is batch size, C is number of channels, T is time frames, H and W are height and width # Example: Dummy tensor for demonstration video = torch.randn(1, 3, 32, 224, 224) # Adjust dimensions according to your video # Perform inference model = model.eval() with torch.no_grad(): outputs = model(video) predicted_class_idx = torch.argmax(outputs, dim=1) print(f'Predicted action class: {predicted_class_idx.item()}')

References¶

Docs

• Vision transformer - Wikipedia
• ViT Docs - huggingface.co

Paper:

• An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
• Transformers in Vision: A Survey - 2022

Blog:

• Vision Transformers (ViT) in Image Recognition – 2024 Guide: viso.ai

Image Classification:

• Image classification with Vision Transformer

Object Detection (2D/3D?):

• Can We Solve 3D Vision Tasks Starting from A 2D Vision Transformer?

Semantic Segmentation:

• Semantic Segmentation using Vision Transformers: A survey

Instance Segmentation:

• A Simple Single-Scale Vision Transformer for Object Localization and Instance Segmentation

Action Recognition:

• Vision Transformers for Action Recognition: A Survey

Code:

• Google Research - Github Repo
• Googl - Hugging Face

In [ ]:

In [ ]:

Datasets¶

List of datasets for Vision transformers:

Image classification

• ImageNet
• CIFAR-10
• CIFAR-100
• SVHN
• Tiny ImageNet

Object detection

• COCO
• Pascal VOC
• ImageNet Detection
• Open Images Dataset

Semantic segmentation

• ADE20K
• Cityscapes
• Pascal Context
• Mapillary Vistas

Video classification

• Kinetics
• UCF101
• HMDB51
• Something-Something V2

Video object detection

• YouTube-VOS
• DAVIS
• SegTrack V2

ViT Models¶

The current state-of-the-art Vision Transformer (ViT) model for image recognition tasks is ViT-Huge:

• 18 billion parameters.
• 89.5% top-1 accuracy on the ImageNet-1k validation set.

Other notable ViT models for image recognition include:

• ViT-B/16: 300 million parameters, 84.6% top-1 accuracy on ImageNet-1k.
• ViT-L/16: 700 million parameters, 86.3% top-1 accuracy on ImageNet-1k.
• ViT-H/14: 1.2 billion parameters, 87.8% top-1 accuracy on ImageNet-1k.

ref: https://github.com/google-research/vision_transformer?tab=readme-ov-file#available-vit-models

Google cloud storage: https://console.cloud.google.com/storage/browser/vit_models/imagenet21k?pageState=(%22StorageObjectListTable%22:(%22f%22:%22%255B%255D%22))

In [ ]:

In [ ]: