Research on CLIP Cross-modal Feature Optimization Based on PCA Fusion Strategy

Abstract

The CLIP (Contrastive Language-Image Pretraining) model achieves cross-modal semantic alignment through a joint embedding space, excelling in zero-shot learning and open-domain retrieval. However, CLIP’s image-text matching matrix lacks sufficient main-diagonal dominance, causing semantic shifts in cross-modal retrieval. Empirical studies on the MSCOCO dataset reveal that CLIP’s mean similarity on the main diagonal is only $0.62 \pm 0.15$, while the maximum noise pair similarity is $0.71 \pm 0.13$, leading to a $19.7 \%$ loss in Top-1 retrieval accuracy. Existing optimization methods, such as full-parameter fine-tuning or complex projection networks, degrade zero-shot capability and increase computational complexity. This study proposes a dual-branch feature fusion optimization method based on PCA (Principal Component Analysis) dimension reduction. We independently construct image/text PCA subspaces (with a 93.75% dimension compression rate) to capture modality-specific semantics and realize cross-modal feature calibration through an $\alpha$-weighted fusion strategy ($\alpha=0.98$). Experiments show this method enhances semantic consistency without retraining, with less than $3 \%$ computational overhead. The method boosts the average similarity on the main diagonal by $18.7 \%$ without changing the row-wise maxima of the similarity matrix. On the MSCOCO and Flickr30K datasets, the Top-1 retrieval accuracy rates reach $80.2 \%$ and $67.3 \%$, respectively, representing a $12.3 \%$ and $12.6 \%$ increase from the baseline.