How to run PCA and KNN on big-data

Question

I work with python and images of watches (examples: watch_1, watch_2, watch_3). My aim is to take a photo of a random watch and then find the most similar watches to it in my database. Obviously, one main feature which distinguishes the watches are their shape (square, rectangular, round, oval) but there are also other ones.

For now, I am just running a PCA and a KNN on rgb images of watches to find the most similar ones among them. My source code is the following:

import cv2
import numpy as np
import os
from glob import glob
from sklearn.decomposition import PCA
from sklearn import neighbors
from sklearn import preprocessing


data = []

# Read images from file
for filename in glob('Watches/*.jpg'):

    img = cv2.imread(filename)
    height, width = img.shape[:2]
    img = np.array(img)

    # Check that all my images are of the same resolution
    if height == 529 and width == 940:

        # Reshape each image so that it is stored in one line
        img = np.concatenate(img, axis=0)
        img = np.concatenate(img, axis=0)
        data.append(img)

# Normalise data
data = np.array(data)
Norm = preprocessing.Normalizer()
Norm.fit(data)
data = Norm.transform(data)

# PCA model
pca = PCA(0.95)
pca.fit(data)
data = pca.transform(data)

# K-Nearest neighbours
knn = neighbors.NearestNeighbors(n_neighbors=4, algorithm='ball_tree', metric='minkowski').fit(data)
distances, indices = knn.kneighbors(data)

print(indices)

However when I try to run this script for more than 1500 rgb images then I get a MemoryError at the point where the data are processed by the PCA method.

Is this normal for a pc with 24GB RAM and 3.6GHz Intel Core CPU without any discrete GPU?

How can I overcome this?

Shall I use another method like Incremental PCA (or a deep learning algorithm) or simply shall I buy a discrete GPU?

Answer 1

KNN is instance based so it will store all training instances in memory. Since you are using images this will add up quickly. KNN on untransformed images might not perform that well anyway, you could look into filter banks to transform your images to a bag-of-word-representation (which is smaller and more invariant).

However if it is accuracy you are aiming for I would recommend skipping all that (it is very 2012 anyway) in favor of using deep learning, fi: construct an auto-encoder and determine similarity on the encoded representation of an image (which could in turn be done using knn btw).