Map Extract
Free tool

Remove duplicate lines

Paste a list and get it back with every repeat taken out. The first appearance of each line is kept where it was, and a note tells you how many duplicates went.

Runs in your browser. Nothing you enter is uploaded.

Example: a clean list of towns to search

A roofing supplier wants to find builders in every town along a stretch of the Yorkshire coast. Three colleagues each sent a list of towns, and once pasted together they overlap. One person typed whitby in lower case with a trailing space.

With the default settings, capitals and outer spaces are ignored, so both forms of Whitby count as one, and the empty line goes too. Two duplicates are removed and three towns remain, each ready to become a separate map search.

Input
Whitby
whitby 
Scarborough

Whitby
Pickering
Result
Whitby
Scarborough
Pickering

2 duplicates removed

Removing duplicates

  1. Paste the list

    One item per line. Example buttons load an email list, product names and keywords if you want to see it work first.

  2. Choose how strict to be

    Case sensitive (off by default) treats York and york as different lines. Ignore spaces at the start and end (on) treats York and York followed by a space as the same.

  3. Decide about blank lines

    Keep empty lines is off, so blanks are removed along with duplicates. Tick it to keep blank lines where they were, for example between groups.

  4. Copy or download

    Take the result with the copy button, or save it as unique.txt.

When exact line matching is not enough

This tool compares whole lines. It is ideal for a single column of towns, keywords, domains or email addresses, but it will not spot that Harbour Café Ltd and Harbour Cafe are the same business, or that two rows of a spreadsheet share a phone number while the names differ.

For rows with several columns, use the CSV deduplicator, which matches on the column you choose. For domains, strip https:// and www. first with find and replace, otherwise the same site written two ways survives as two lines. Running a list once with case sensitivity on and once with it off shows how many entries differ only in capitals.

Questions people ask

How do I remove duplicates from a list?

Paste the list here, one item per line. Every line that has already appeared is removed, the first copy stays in its original position, and the count of removed lines is shown. Copy the result back to wherever the list came from. Nothing is sorted, so your order is kept.

How do I remove duplicates in Excel?

Select the column, then on the Data tab choose Remove Duplicates. Excel compares cells without regard to capitals but does not ignore spaces at the ends, so York and York with a space survive as two. Pasting the column here first, with spaces ignored, catches those too.

Which copy of a duplicate is kept?

The first one, exactly as it was written, apart from outer spaces if that option is on. In the example, Whitby with a capital is kept because it came first. If you would rather keep a different spelling, move that line to the top before removing duplicates.

Where your data goes

Duplicate checking runs in the page, so email lists and contact names you paste are not uploaded to any server.

Checked against the tool on

Top