MobileClip-S2-FP16
Image Captioning
FP16
post
MobileClip-S2: Image Captioning

MobileClip-S2 is a lightweight image-text matching model designed specifically for mobile and resource-constrained devices. It is an optimized version of the CLIP (Contrastive Language-Image Pre-training) model, capable of mapping image and text semantics into a shared feature space using contrastive learning. This allows for efficient cross-modal retrieval and understanding. Compared to the standard CLIP model, MobileClip-S2 is smaller in size and requires less computational power, making it ideal for fast inference on mobile devices. The model is widely used in image search, image-text matching, and multimodal AI applications, supporting joint processing of images and text to perform tasks such as image classification and image caption generation.

Source model

  • Input shape: 1x3x256x256, 1x77
  • Number of parameters: 35.7M, 63.4M
  • Model size: 141M, 243.77M
  • Output shape: 1x512, 1x512

Source model repository: MobileClip-S2

Performance Reference
Device
Backend
Precision
Inference Time
Accuracy Loss
File Size
Model Optimization

When the user has fine-tuned the source model, the model conversion process must be performed again.

Users can refer to either of the following two methods to complete the model conversion:

  • Using AIMO for model conversion: Click Model Conversion Reference in the Performance Reference section on the right to view the conversion steps.

  • Using Qualcomm QNN for model conversion: Please refer to the Qualcomm QNN Documentation.

Model Inference

The model performance benchmarks and example code provided by Model Farm are all implemented based on the APLUX AidLite SDK.

For models in .bin format, you can use either of the following two inference engines to run inference on Qualcomm chips:

Inference using APLUX AidLite: please refer to the APLUX AidLite Developer Documentation

Inference using Qualcomm QNN: Please refer to the Qualcomm QNN Documentation

Inference Example Code

The inference example code is implemented using the AidLite SDK.

Click Model & Code to download the model files and the inference code package. The file structure is as follows:

/model_farm_{model_name}_aidlite
    
    |__ models # folder where model files are stored

    |__ python # aidlite python model inference example

    |__ cpp # aidlite cpp model inference example

    |__ README.md
License
Source Model:MIT
Deployable Model:APLUX-MODEL-FARM-LICENSE
Performance Reference
Device
Backend
Precision
Inference Time
Accuracy Loss
File Size