Remove Duplicate Lines
Paste a list with one item per line and get each item once. Decide whether case and stray spaces matter, which copy to keep, and how to sort what's left.
9 lines
How to compare lines
When a line repeats, keep
Your rules for a match
Ignore case and surrounding spaces, or compare lines exactly.
Sort the way people read
Natural order puts item2 before item10.
Lists stay on your device
Email lists and keywords are never sent to a server.
Clean up a list in a few seconds
Mailing lists exported from two systems, keyword lists merged from several tools, log lines and URL lists all collect repeats. Spotting them by eye stops working after a few dozen lines. This tool compares every line with every other and keeps one copy of each, in the order they first appeared unless you ask for sorting.
Most of the time duplicates aren't perfect copies. Anna.Berg@example.com and anna.berg@example.com are the same address, and a keyword with a trailing space is still the same keyword. That's why Ignore upper and lower case and Ignore spaces at the start and end are switched on to begin with. Turn them off when the difference matters, for example in case-sensitive file paths or passwords.
Nothing is rewritten behind your back. The tool only decides which lines stay, so every line in the result is one you pasted, letter for letter. If you also want to change spelling or case, run the list through the Case Converter or Find and Replace afterwards.
How to use it
- 1Paste your list or open a .txt or .csv file. Each line is one item.
- 2Choose how lines are compared: case, surrounding spaces and empty lines.
- 3Pick whether to keep the first or last copy and how to sort the result.
- 4Copy or download the clean list, or switch to Only duplicates to see what repeated.
How the comparison works
Each line is turned into a comparison key: trimmed if you ignore surrounding spaces, lower-cased if you ignore case. Lines with the same key count as duplicates. The key is only used for comparing. The line that ends up in your result is the original line, with its own capitals and spacing, so ignoring case never changes how a name is written.
A line break at the very end of the text doesn't count as an extra empty line. Lines with nothing but spaces in them count as empty when Remove empty lines is on.
Keeping the first or the last copy only matters when duplicates differ in some way you told the tool to ignore, or when you care about position. First copy holds on to the earliest version and its place in the list. Last copy does the opposite: the item moves to where it last appeared and keeps that version's spelling. If you've appended a corrected list to an older one, keeping the last copy means the corrections win.
Sorting options
- Keep original order: nothing moves apart from the removed lines.
- A to Z and Z to A: alphabetical, using your browser's language rules, so accented letters sort next to their plain versions instead of after Z.
- Natural: numbers inside lines are read as numbers, so item2 comes before item10 and chapter 9 before chapter 11. Plain A to Z would put item10 first because 1 comes before 2.
Where people use it
- Merging newsletter exports before an import, so nobody gets the same email twice.
- Cleaning keyword research from several tools into one list for a spreadsheet.
- Tidying a list of URLs before adding them to a sitemap or a redirect file.
- Reducing a long log file to the distinct error messages it contains.
- Checking a guest list or a class register for people who were entered twice, using Only duplicates.
Finding the duplicates instead
Switch the result to Only duplicates to list every line that appeared two or more times, each one once. It's a quick way to check which customers signed up twice or which URLs your sitemap lists more than once. The counters underneath show total lines, unique lines, how many were removed (empty lines included) and how many different lines were repeated.
Related tools
What Our Users Say
Real feedback from people using WeblixTools, checked by our team before it appears.
5.0 out of 5 · 1 review
very nice
Frequently Asked Questions
How do I remove duplicates but keep the original order?
Leave Sort the result on Keep original order. Repeats are removed and everything else stays where it was. With First copy selected, each item sits where it first appeared.
Are "Apple" and "apple" treated as duplicates?
Yes, while Ignore upper and lower case is on, which it is by default. The copy you keep is written as it was in your list. Turn the option off to treat them as two different lines.
Can I see which lines were duplicated?
Yes. Switch the result to Only duplicates. You'll get every line that appeared more than once, each listed a single time, and the Repeated counter tells you how many there are.
What does natural sort do?
It compares numbers inside text by their value. A list of file1, file10, file2 becomes file1, file2, file10. Plain alphabetical order compares character by character and would give file1, file10, file2.
Does it remove blank lines too?
Only when Remove empty lines is on (it is by default). With it off, empty lines are compared like any other line, so several blank lines shrink to one.
How big can the list be?
Lists with tens of thousands of lines work fine, since everything runs locally and the result is recalculated in the background while you type. Very large files depend on your device's memory more than on the tool.
Can I dedupe a CSV file with it?
It compares whole lines, so it removes rows that are repeated in full. It doesn't look at single columns. To dedupe on one column, copy that column into the tool on its own.