Deduplication:
two records, one person.

Find duplicates even when they are written differently. Example · Netherlands: two records, and why they are the same person.

Free every month: 50 address checks, 100 records to find duplicates and 500 email and phone checks. All we ask for is your name, email and a password, no credit card: create your free account or log in.

What deduplication does for you

A duplicate is one person's history split in two. Putting it back together has four concrete effects.

Reply once, and with full knowledge

Whoever replies looks at one record: if the order is registered on the other, that person receives nothing — and notices. With a single record the confirmation always goes out, and the communication arrives once, with the name spelled one way.

Every extra envelope is money thrown away

Every duplicate is a print, an envelope and postage paid for a person your mailing has already reached. In a large campaign it is the easiest cost to cut, without losing a single recipient.

Spot your most valuable contacts

Someone who bought three times under one record and twice under the other shows up in reports as two lukewarm contacts. Merging the records shows them for what they are, a loyal customer, and changes who is worth calling, who deserves different treatment, who to offer more to.

The numbers add up

How many customers do I really have? What is a contact worth on average? With a database full of duplicates every answer is inflated, and the decisions that follow are flawed from the start.

Where duplicates come from

They are not careless mistakes: duplicates come from three ordinary situations, repeated in every database that is meant to grow.

The order comes in and there is no time

Whoever enters the data has a pile of forms to get through (or of payment slips, in a non-profit) and doesn't stop to check whether that person is already on file. Opening a new record takes thirty seconds; really searching for it, among the variants of the name and the address, takes much longer.

The record is there, but the search doesn't find it

It is the most frequent case and the hardest to avoid: the record exists, but is written just slightly differently from how it is searched — “Kalverstraat 92” against “Kalverstr. 92”. The search returns nothing, and the record is created again in good faith.

A whole list is imported

Sign-ups from the website, a list collected at an event, a file from a supplier: they come in as a block without being compared with the existing database. It happens less often than the other two cases, but when it does, duplicates come in by the hundreds.

All three are solved the same way: a check of the database at regular intervals, and one before every important campaign.

First fix the addresses, then find the duplicates

A word-by-word comparison treats Hendrik and Henk as two people and Kalverstr. 92 and Kalverstraat 92 as two addresses, while they are the same person and the same address. That is why we work in two steps: first we fix the two addresses — street name in full, correct postcode, the right city — and only then do we compare them, together with the name, the email and the phone. Two spellings of the same street become the same street before the comparison begins.

The verdict has three grades — certain, probable or ambiguous — with the reason next to it. In the clean file, certain and probable groups come out already merged into a single row. Ambiguous ones stay whole and go into the rows to review: two different people can have the same name in the same town, and there you decide.

Normalisation, duplicate detection, deduplication, enrichment

Four different operations, usually all filed under the word “cleaning”. Telling them apart matters, because each one is the premise of the next.

Address normalisation

Reducing every address to a single, correct spelling, completing what is missing: the postcode, the province, the street name in full.

“Kalverstr. 92” and “Kalverstraat 92” become the same line.

Duplicate detection

Recognising that two records are the same subject even when written differently: “Kalverstr. 92” and “Kalverstraat 92”, surname and first name swapped.

It comes after normalisation, so the comparison works on addresses already written in the correct form.

Deduplication

Deciding what to do with the duplicates found: which record to keep, which fields to take from one and which from the other.

The final record takes the best field from each record and keeps track of where it came from.

Enrichment

Adding what is missing or can be deduced from what is there: the salutation from the name, the title, the house number suffix, the tax code.

Data your database already contains implicitly.

Why it matters, beyond cleaning: whoever receives the same letter twice notices, and whoever doesn't receive the thank-you because the donation was registered on one record while you looked at the other notices even more.

Duplicate matching: you choose how strict, we tell you how sure we are

The engine forms the groups of records, and for each one it states how certain it is. Two knobs. The minimum level, that is how different two records can be and still be the same subject: three values, the website and file jobs use the default, the API lets you choose with livello_minimo. And the precision, the same 1 to 5 scale as the address check, which here acts on the surname: up to 3 it is read letter by letter, at 4 records whose surname is written with or without a space (“La Pera”, “Lapera”) are paired, at 5 also those whose surname sounds alike (“Lepera”, “La Pera”); these groups come out as ambiguous at most.

1 · identical

Strict

Name and address coincide after normalisation. No group you wouldn't have made by hand.

2 · slightly different

The default

Variants of the same name (Henk and Hendrik), abbreviations and small typos come in. It is the setting most jobs run with.

3 · different

Loose

Weak links too: same address and similar names. It finds more, and you have to look at more.

The verdict returned on every group — certain, probable, ambiguous — is our classification: it says how solid the group is and decides what happens in the clean file, where certain and probable groups come out merged and ambiguous ones are left to review.

The best of each record in a single one

We take the best field from each record and build a new one: even the email that was on only one of the two ends up in the record that remains.

Example of merging two records: which field remains and where it comes from
Field	Record A	Record B	Record that remains
Name	Jan de Vries	J. De Vries	Jan de Vries the most complete spelling
Address	Kalverstr. 92	Kalverstraat 92	Kalverstraat 92 from the more complete record
Postcode and city	1012PH Amsterdam	1012 PH AMSTERDAM	1012 PH AMSTERDAM in the country's postal form
Email	—	j.devries@example.com	j.devries@example.com was only on B

Keeping A you would have lost the email, keeping B the full spelling of the name. The merged record has both, and for every field it stays on record which one it comes from: if one day the merge doesn't convince you, you know exactly what to touch.

Duplicates that an exact search misses

Henk de Vries=Hendrik de Vries equivalent name
de Vries Jan=Jan de Vries surname and first name swapped
kalverstr 92=Kalverstraat 92 abbreviation expanded
de Vries Jan, kalverstr. 92, amsterdam=Jan de Vries, Kalverstraat 92, 1012 PH AMSTERDAM address fixed before the comparison

We also recognise a typo in the first name or surname, and flag it as a possible typing error.

The examples use addresses of public buildings and city-centre streets; the names, where they appear, are invented for demonstration purposes: any reference to real persons is purely coincidental.

Deduplication:two records, one person.

Record A

Record B