We found a match
Your institution may have access to this item. Find your institution then sign in to continue.
- Title
Estimating parameters for probabilistic linkage of privacy-preserved datasets.
- Authors
Brown, Adrian P.; Randall, Sean M.; Ferrante, Anna M.; Semmens, James B.; Boyd, James H.
- Abstract
<bold>Background: </bold>Probabilistic record linkage is a process used to bring together person-based records from within the same dataset (de-duplication) or from disparate datasets using pairwise comparisons and matching probabilities. The linkage strategy and associated match probabilities are often estimated through investigations into data quality and manual inspection. However, as privacy-preserved datasets comprise encrypted data, such methods are not possible. In this paper, we present a method for estimating the probabilities and threshold values for probabilistic privacy-preserved record linkage using Bloom filters.<bold>Methods: </bold>Our method was tested through a simulation study using synthetic data, followed by an application using real-world administrative data. Synthetic datasets were generated with error rates from zero to 20% error. Our method was used to estimate parameters (probabilities and thresholds) for de-duplication linkages. Linkage quality was determined by F-measure. Each dataset was privacy-preserved using separate Bloom filters for each field. Match probabilities were estimated using the expectation-maximisation (EM) algorithm on the privacy-preserved data. Threshold cut-off values were determined by an extension to the EM algorithm allowing linkage quality to be estimated for each possible threshold. De-duplication linkages of each privacy-preserved dataset were performed using both estimated and calculated probabilities. Linkage quality using the F-measure at the estimated threshold values was also compared to the highest F-measure. Three large administrative datasets were used to demonstrate the applicability of the probability and threshold estimation technique on real-world data.<bold>Results: </bold>Linkage of the synthetic datasets using the estimated probabilities produced an F-measure that was comparable to the F-measure using calculated probabilities, even with up to 20% error. Linkage of the administrative datasets using estimated probabilities produced an F-measure that was higher than the F-measure using calculated probabilities. Further, the threshold estimation yielded results for F-measure that were only slightly below the highest possible for those probabilities.<bold>Conclusions: </bold>The method appears highly accurate across a spectrum of datasets with varying degrees of error. As there are few alternatives for parameter estimation, the approach is a major step towards providing a complete operational approach for probabilistic linkage of privacy-preserved datasets.
- Subjects
MEDICAL records; PROBABILITY theory; DATA quality; ALGORITHMS; ESTIMATION theory; MEDICAL ethics; MEDICAL record linkage; PRIVACY; RESEARCH evaluation; DATA security; ACQUISITION of data
- Publication
BMC Medical Research Methodology, 2017, Vol 17, p1
- ISSN
1471-2288
- Publication type
journal article
- DOI
10.1186/s12874-017-0370-0