Computer Science · Artificial Intelligence
A Probabilistic Deduplication, Record Linkage and Geocoding System
See on the mapAbstract
In many data mining projects in the health sector information from multiple data sources needs to be cleaned, deduplicated and linked in order to allow more detailed analysis. The aim of such linkages is to merge all records relating to the same entity, such as a patient. Most of the time the linkage process is challenged by the lack of a common unique entity identifier. Additionally, personal information, like names and addresses, are frequently recorded with typographical errors, can be formatted differently, and parts can even be missing or swapped, making the duplication or linkage task non-trivial. A special case of linkage is geocoding, the process of matching user records with geocoded reference data, allowing spatial data analysis and mining, for example of disease outbreaks, or correlations with environmental factors. In this paper we present an overview of the Febrl (Freely extensible biomedical record linkage) project, which aims at developing improved algorithms and techniques for large scale data cleaning and standardisation, record linkage,