Vie-BLIP

Vie-BLIP is a Vietnamese image captioning model fine-tuned from Salesforce/blip-image-captioning-base. The model takes an image as input and generates a Vietnamese natural-language caption describing the image content.

This repository contains the model weights, processor configuration, tokenizer files, and generation configuration required to load the model with 🤗 Transformers.

Model Details

• Model name: Vie-BLIP
• Repository: nviquang/Vie-BLIP
• Task: Image captioning / Image-to-text
• Language: Vietnamese (vi)
• Base model: Salesforce/blip-image-captioning-base
• Architecture: BlipForConditionalGeneration
• Library: 🤗 Transformers
• Image processor: BlipImageProcessorFast
• Tokenizer: PhobertTokenizer
• Image size: 384 x 384
• Model weights format: safetensors

Intended Use

Vie-BLIP is intended for generating Vietnamese captions for images. Example use cases include:

• Vietnamese image captioning research
• Vision-language model experimentation
• Educational projects
• Assistive captioning prototypes
• Webcam or local image captioning demos

Repository Files

The repository includes:

.
├── added_tokens.json
├── bpe.codes
├── config.json
├── generation_config.json
├── model.safetensors
├── processor_config.json
├── tokenizer_config.json
├── training_curves.png
└── vocab.txt

Installation

Install the required dependencies:

pip install -U transformers torch safetensors pillow

If tokenizer loading requires additional dependencies in your environment, install:

pip install sentencepiece

Usage

from PIL import Image
import torch
from transformers import BlipProcessor, BlipForConditionalGeneration

repo_id = "nviquang/Vie-BLIP"

device = "cuda" if torch.cuda.is_available() else "cpu"

processor = BlipProcessor.from_pretrained(repo_id)
model = BlipForConditionalGeneration.from_pretrained(repo_id).to(device)
model.eval()

image = Image.open("example.jpg").convert("RGB")

inputs = processor(images=image, return_tensors="pt").to(device)

with torch.no_grad():
    generated_ids = model.generate(
        pixel_values=inputs["pixel_values"],
        max_length=64,
        num_beams=5,
        early_stopping=True,
        no_repeat_ngram_size=3,
        repetition_penalty=1.5,
    )

caption = processor.batch_decode(generated_ids, skip_special_tokens=True)[0]
print(caption.strip())

Example Output

một người đang đi xe máy trên đường phố

Actual outputs may vary depending on the image content and generation parameters.

Dataset

The model was fine-tuned for Vietnamese image captioning using the UIT-OpenViIC dataset.

Preprocessing and dataset analysis used in the project reported the following statistics:

Split	Images	Captions
Train	9,088	41,238
Dev	2,011	10,002
Test	2,001	10,001
Total	13,100	61,241

After basic cleaning, 61,239 valid captions were retained.

Training Summary

The model was fine-tuned from Salesforce/blip-image-captioning-base for Vietnamese caption generation. The training pipeline used image-caption pairs from UIT-OpenViIC and saved the final checkpoint in the standard 🤗 Transformers format.

The included training_curves.png file shows the training curves from the fine-tuning process.

Evaluation

The project evaluation pipeline used common image captioning and text generation metrics:

• BLEU
• ROUGE-L
• METEOR

Exact metric values are not included in this model card yet. If available, they should be added here for reproducibility.

Limitations

• The model is optimized for Vietnamese image captioning and may not perform well in other languages.
• The model may produce generic, incomplete, or incorrect captions.
• The model may struggle with images that are very different from the UIT-OpenViIC training distribution.
• Generated captions may miss fine-grained details, counts, text in images, or rare objects.
• The model may inherit biases or annotation patterns from the training dataset.

Ethical Considerations

This model should be used as an assistive caption generation system, not as a source of factual truth. Outputs should be reviewed before use in high-stakes settings such as accessibility, education, medical, legal, or safety-critical applications.

License

The license is currently marked as other. Please verify the license terms of the base model and the UIT-OpenViIC dataset before using this model in commercial or public production settings.

Citation

If you use this model, please cite the original BLIP work and the UIT-OpenViIC dataset where appropriate.

@inproceedings{li2022blip,
  title={BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation},
  author={Li, Junnan and Li, Dongxu and Xiong, Caiming and Hoi, Steven},
  booktitle={International Conference on Machine Learning},
  year={2022}
}

Acknowledgements

This model is based on Salesforce/blip-image-captioning-base and was fine-tuned for Vietnamese image captioning.