arXiv.itsnick.co

an independent field guide to 23 million papers from arXiv & OpenAlex — not affiliated with arXiv.org

Author

Jack Sim

15 papers · 2011–2022

2011: 32012: 02013: 02014: 02015: 02016: 12017: 12018: 22019: 12020: 32021: 22022: 2
2011peak 3 in 20112022

Frequent co-authors

Papers

showing 1–15 of 15 RSS

Large-Scale Image Retrieval with Attentive Deep Local Features

Hyeonwoo Noh, Andre Araujo, Jack Sim +2 more

cs.CV · · cited 866× · arXiv ↗ · more like this ·

We propose an attentive local feature descriptor suitable for large-scale image retrieval, referred to as DELF (DEep Local Feature). The new feature is based on convolutional neural networks, which are trained only with image-level annotations on a landmark image dataset. To identify semantically useful local features for image retrieval, we also propose an attention mechanism for keypoint selection, which shares most network layers with the descriptor. This framework can be used for image retrieval as a drop-in replacement for other keypoint detectors and descriptors, enabling more accurate feature matching and geometric verification. Our system produces reliable confidence scores to reject false positives---in particular, it is robust against queries that have no correct match in the database. To evaluate the proposed descriptor, we introduce a new large-scale dataset, referred to as…

Google Landmarks Dataset v2 -- A Large-Scale Benchmark for Instance-Level Recognition and Retrieval

Tobias Weyand, Andre Araujo, Bingyi Cao +1 more

cs.CV · · cited 497× · arXiv ↗ · more like this ·

While image retrieval and instance recognition techniques are progressing rapidly, there is a need for challenging datasets to accurately measure their performance -- while posing novel challenges that are relevant for practical applications. We introduce the Google Landmarks Dataset v2 (GLDv2), a new benchmark for large-scale, fine-grained instance recognition and image retrieval in the domain of human-made and natural landmarks. GLDv2 is the largest such dataset to date by a large margin, including over 5M images and 200k distinct instance labels. Its test set consists of 118k images with ground truth annotations for both the retrieval and recognition tasks. The ground truth construction involved over 800 hours of human annotator work. Our new dataset has several challenging properties inspired by real world applications that previous datasets did not consider: An extremely…

BranchOut: Regularization for Online Ensemble Tracking with Convolutional Neural Networks

Bohyung Han, Jack Sim, Hartwig Adam

Computer Vision and Pattern Recognition · · cited 172× · DOI ↗ · more like this ·

We propose an extremely simple but effective regularization technique of convolutional neural networks (CNNs), referred to as BranchOut, for online ensemble tracking. Our algorithm employs a CNN for target representation, which has a common convolutional layers but has multiple branches of fully connected layers. For better regularization, a subset of branches in the CNN are selected randomly for online learning whenever target appearance models need to be updated. Each branch may have a different number of layers to maintain variable abstraction levels of target appearances. BranchOut with multi-level target representation allows us to learn robust target appearance models with diversity and handle various challenges in visual tracking problem effectively. The proposed algorithm is evaluated in standard tracking benchmarks and shows the state-of-the-art performance even without…

Detect-to-Retrieve: Efficient Regional Aggregation for Image Search

Marvin Teichmann, Andre Araujo, Menglong Zhu +1 more

cs.CV · · cited 128× · arXiv ↗ · more like this ·

Retrieving object instances among cluttered scenes efficiently requires compact yet comprehensive regional image representations. Intuitively, object semantics can help build the index that focuses on the most relevant regions. However, due to the lack of bounding-box datasets for objects of interest among retrieval benchmarks, most recent work on regional representations has focused on either uniform or class-agnostic region selection. In this paper, we first fill the void by providing a new dataset of landmark bounding boxes, based on the Google Landmarks dataset, that includes $86k$ images with manually curated boxes from $15k$ unique landmarks. Then, we demonstrate how a trained landmark detector, using our new dataset, can be leveraged to index image regions and improve retrieval accuracy while being much more efficient than existing regional methods. In addition, we introduce a…

Multi-hypothesis motion planning for visual object tracking

Haifeng Gong, Jack Sim, Maxim Likhachev +1 more

Computer Vision and Pattern Recognition · ScholarlyCommons (University of Pennsylvania) · · cited 105× · DOI ↗ · more like this ·

In this paper, we propose a long-term motion model for visual object tracking. In crowded street scenes, persistent occlusions are a frequent challenge for tracking algorithm and a robust, long-term motion model could help in these situations. Motivated by progresses in robot motion planning, we propose to construct a set of `plausible' plans for each person, which are composed of multiple long-term motion prediction hypotheses that do not include redundancies, unnecessary loops or collisions with other objects. Constructing plausible plan is the key step in utilizing motion planning in object tracking, which has not been fully investigate in robot motion planning. We propose a novel method of efficiently constructing disjoint plans in different homotopy classes, based on winding numbers and winding angles of planned paths around all obstacles. As the goals can be specified by winding…

Personalized video summarization with human in the loop

Bohyung Han, Jihun Hamm, Jack Sim

Computer Vision and Pattern Recognition · · cited 42× · DOI ↗ · more like this ·

In automatic video summarization, visual summary is constructed typically based on the analysis of low-level features with little consideration of video semantics. However, the contextual and semantic information of a video is marginally related to low-level features in practice although they are useful to compute visual similarity between frames. Therefore, we propose a novel video summarization technique, where the semantically important information is extracted from a set of keyframes given by human and the summary of a video is constructed based on the automatic temporal segmentation using the analysis of inter-frame similarity to the keyframes. Toward this goal, we model a video sequence with a dissimilarity matrix based on bidirectional similarity measure between every pair of frames, and subsequently characterize the structure of the video by a nonlinear manifold embedding. Then…

Unifying Deep Local and Global Features for Image Search

Bingyi Cao, Andre Araujo, Jack Sim

cs.CV · · cited 24× · arXiv ↗ · more like this ·

Image retrieval is the problem of searching an image database for items that are similar to a query image. To address this task, two main types of image representations have been studied: global and local image features. In this work, our key contribution is to unify global and local features into a single deep model, enabling accurate retrieval with efficient feature extraction. We refer to the new model as DELG, standing for DEep Local and Global features. We leverage lessons from recent feature learning work and propose a model that combines generalized mean pooling for global features and attentive selection for local features. The entire network can be learned end-to-end by carefully balancing the gradient flow between two heads -- requiring only image-level labels. We also introduce an autoencoder-based dimensionality reduction technique for local features, which is integrated…

Unifying Deep Local and Global Features for Efficient Image Search.

Bingyi Cao, André Araujo, Jack Sim

Computer Vision and Pattern Recognition · · cited 18× · OpenAlex ↗ · more like this ·

Image retrieval is the problem of searching an image database for items that are similar to a query image. To address this task, two main types of image representations have been studied: global and local image features. In this work, our key contribution is to unify global and local features into a single deep model, enabling accurate retrieval with efficient feature extraction. We refer to the new model as DELG, standing for DEep Local and Global features. We leverage lessons from recent feature learning work and propose a model that combines generalized mean pooling for global features and attentive selection for local features. The entire network can be learned end-to-end by carefully balancing the gradient flow between two heads -- requiring only image-level labels. We also introduce an autoencoder-based dimensionality reduction technique for local features, which is integrated…

Bringing Vision to the Blind: From Coarse to Fine, One Dollar at a Time

Tri Huynh, Jay J. Pillai, Eunyoung Kim +4 more

Computer Vision and Pattern Recognition · · cited 5× · DOI ↗ · more like this ·

While deep learning has achieved great success in building vision applications for mainstream users, there is relatively less work for the blind and visually impaired to have a personal, on-device visual assistant for their daily life. Unlike mainstream applications, vision system for the blind must be robust, reliable and safe-to-use. In this paper, we propose a fine-grained currency recognizer based on CONGAS, which significantly surpasses other popular local features by a large margin. In addition, we introduce an effective and light-weight coarse classifier that gates the fine-grained recognizer on resource-constrained mobile devices. The coarse-to-fine approach is orchestrated to provide an extensible mobile-vision architecture, that demonstrates how the benefits of coordinating deep learning and local feature based methods can help in resolving a challenging problem for the blind…

Towards A Fairer Landmark Recognition Dataset

Zu Kim, André Araujo, Bingyi Cao +5 more

cs.CV · · cited 4× · arXiv ↗ · more like this ·

We introduce a new landmark recognition dataset, which is created with a focus on fair worldwide representation. While previous work proposes to collect as many images as possible from web repositories, we instead argue that such approaches can lead to biased data. To create a more comprehensive and equitable dataset, we start by defining the fair relevance of a landmark to the world population. These relevances are estimated by combining anonymized Google Maps user contribution statistics with the contributors' demographic information. We present a stratification approach and analysis which leads to a much fairer coverage of the world, compared to existing datasets. The resulting datasets are used to evaluate computer vision models as part of the the Google Landmark Recognition and RetrievalChallenges 2021.

Improving Fairness in Large-Scale Object Recognition by CrowdSourced Demographic Information

Zu Kim, André Araujo, Bingyi Cao +5 more

cs.CV · · cited 2× · arXiv ↗ · more like this ·

There has been increasing awareness of ethical issues in machine learning, and fairness has become an important research topic. Most fairness efforts in computer vision have been focused on human sensing applications and preventing discrimination by people's physical attributes such as race, skin color or age by increasing visual representation for particular demographic groups. We argue that ML fairness efforts should extend to object recognition as well. Buildings, artwork, food and clothing are examples of the objects that define human culture. Representing these objects fairly in machine learning datasets will lead to models that are less biased towards a particular culture and more inclusive of different traditions and values. There exist many research datasets for object recognition, but they have not carefully considered which classes should be included, or how much training data…

CPlaNet: Enhancing Image Geolocalization by Combinatorial Partitioning of Maps

Paul Hongsuck Seo, Tobias Weyand, Jack Sim +1 more

cs.CV · · cited 2× · arXiv ↗ · more like this ·

Image geolocalization is the task of identifying the location depicted in a photo based only on its visual information. This task is inherently challenging since many photos have only few, possibly ambiguous cues to their geolocation. Recent work has cast this task as a classification problem by partitioning the earth into a set of discrete cells that correspond to geographic regions. The granularity of this partitioning presents a critical trade-off; using fewer but larger cells results in lower location accuracy while using more but smaller cells reduces the number of training examples per class and increases model size, making the model prone to overfitting. To tackle this issue, we propose a simple but effective algorithm, combinatorial partitioning, which generates a large number of fine-grained output classes by intersecting multiple coarse-grained partitionings of the earth. Each…

Sanitation advocacy services

Jack Sim, Peter Emmanuel Cookey

Political Science and International Relations · IWA Publishing eBooks · · cited 0× · DOI ↗ · more like this ·

The value chain (VC) system is a key way to address important sanitation technological and institutional gaps in production and service delivery and could constitute a natural platform for development actions and also serve as a market systems approach to improve access to safely-managed sanitation. It has been suggested that sanitation could boost local and national economies and global interconnections with a growing recognition that the private sector can play a bigger role in delivering the Sustainable Development Goal for sanitation, and help businesses understand value-added and product opportunities. This book proposes a pathway towards re-thinking the sanitation value chain (SVC) and suggests that it should cover all processes, activities and products of enterprises/actors in the sanitation supply chain that provide value-added services within each stage. Following the…

Portal to Portaloo

Jack Sim, Karl Dayson

Artificial Intelligence · IGI Global eBooks · · cited 0× · DOI ↗ · more like this ·

2.5 billion people do not have access to a toilet; instead they have no choice but to practice open defecation, having a potentially detrimental effect on their health. The chapter asks whether microfinance and IT can play a role in tackling the problem. Drawing on the experience of Grameen Telephone it is argued that this is analogous with attempts to promote the purchase of toilets, in particular the technological leap where expensive infrastructure is bypassed. Drawing on three case studies we show that such a process is underway and while there are a limited number of microfinance institutions (MFIs) engaged in this market it is insufficient to address the myriad of both organizational and cultural problems. However, the use of a prospective web portal may help create the environment for a viable market to emerge.

Nutrition5k: Towards Automatic Nutritional Understanding of Generic Food

Quin Thames, Arjun Karpur, Wade Norris +4 more

cs.CV · · arXiv ↗ · more like this ·

Understanding the nutritional content of food from visual data is a challenging computer vision problem, with the potential to have a positive and widespread impact on public health. Studies in this area are limited to existing datasets in the field that lack sufficient diversity or labels required for training models with nutritional understanding capability. We introduce Nutrition5k, a novel dataset of 5k diverse, real world food dishes with corresponding video streams, depth images, component weights, and high accuracy nutritional content annotation. We demonstrate the potential of this dataset by training a computer vision algorithm capable of predicting the caloric and macronutrient values of a complex, real world dish at an accuracy that outperforms professional nutritionists. Further we present a baseline for incorporating depth sensor data to improve nutrition predictions. We…