Skip to content

Visual Prompting - Notes

Table of Contents

Introduction

Visual Prompting is a novel method for adapting large-scale models in computer vision with minimal human supervision.

Fundamentals

  • Visual Prompting is inspired by the success of text prompting in natural language processing (NLP), where a single API call can prompt a pre-trained model to perform a new task.
  • Visual Prompting leverages the power of pre-trained vision transformers, which can learn from a few visual prompts (such as regions of interest or labels) provided by the user .
  • Visual Prompting is faster and easier than conventional labeling, which requires annotating every image in the training set.
  • Visual Prompting can be applied to various computer vision tasks, such as object detection, segmentation, classification, and captioning.

Tools & Frameworks

  • Landing AI is a platform that offers Visual Prompting as a capability for building computer vision applications.
  • Landing AI's Visual Prompting allows users to specify a visual prompt by painting over object classes they wish the system to detect, using just one or a few images.
  • Landing AI's Visual Prompting can transform an unlabeled dataset into a deployed model in minutes, resulting in a simplified, faster, and more user-friendly workflow.

Hello World!

# Import Landing AI's Visual Prompting module
from landingai import visual_prompting

# Load a pre-trained vision transformer model
model = visual_prompting.load_model("clip")

# Create a visual prompt for detecting cats and dogs
prompt = visual_prompting.create_prompt(["cat", "dog"])

# Apply the prompt to an image
image = visual_prompting.load_image("cat_dog.jpg")
result = model(image, prompt)

# Display the result
visual_prompting.show_result(result)

Lab: Zero to Hero Projects

  • Try Visual Prompting on different computer vision tasks and datasets, such as face recognition, scene understanding, or medical imaging.
  • Compare the performance of Visual Prompting with conventional labeling and fine-tuning methods.
  • Explore the properties of the downstream dataset, prompt design, and output transformation in regard to adaptation performance .
  • Experiment with different pre-trained vision transformer models and visual prompt types.

References

Lectures & Online Courses: - Prompt Engineering for Vision Models Crash Course - Notes (DeepLearningAI)