U.S. flag

An official website of the United States government, Department of Justice.

Face Search at Scale

NCJ Number
Ieee Transactions on Pattern Analysis and Machine Intelligence 39(6): 1122-1136. Volume: 39 Issue: 6 Dated: 2017 Pages: 122-1136
Dayong Wang; Charles Otto; Anil K. Jain
Date Published
15 pages
Since despite significant progress in face recognition, searching a large collection of unconstrained face images remains a difficult problem, the current project addressed this challenge by proposing a face search system that combines a fast search procedure, coupled with a state-of-the-art commercial off the shelf (COTS) matcher, in a cascaded framework.
Given the prevalence of social media websites, one challenge facing computer vision researchers is to devise methods to search for persons of interest among the billions of shared photos on these websites. Given a probe face, the current study first filtered the large gallery of photos to find the top-k most similar faces using features learned by a convolutional neural network. The k retrieved candidates are re-ranked by combining similarities based on deep features and those output by the COTS matcher. The proposed face search system was evaluated on a gallery containing 80 million web-downloaded face images. Experimental results demonstrate that although the deep features perform worse than the COTS matcher on a mugshot dataset (93.7 percent versus 98.6 percent [email protected] of 0.01 percent), fusing the deep features with the COTS matcher improves the overall performance (99.5 percent [email protected] of 0.01 percent). These findings show that the learned deep features provide complementary information over representations used in state-of-the-art face matchers. On the unconstrained face image benchmarks, the performance of the learned deep features is competitive with reported accuracies. LFW database: 98.20 percent accuracy under the standard protocol and 88.03 percent [email protected] of 0.1 percent under the BLUFR protocol; IJB-A benchmark: 51.0 percent [email protected] of 0.1 percent (verification), rank 1 retrieval of 82.2 percent (closed-set search), 61.5 percent [email protected] of 1 percent (open-set search). The proposed face search system offers an excellent trade-off between accuracy and scalability on galleries with millions of images. Also, a face search experiment involving photos of the Tsarnaev brothers, who were convicted of the Boston Marathon bombing, the proposed cascade face search system could find the younger brother's (Dzhokhar Tsarnaev) photo at rank 1 in 1 second on a 5 M gallery and at rank 8 in 7 seconds on an 80 M gallery. (Publisher abstract modified)