Duplicate Line Remover

Paste text to instantly remove duplicate lines. Control case, whitespace, empty lines, and order — see what was removed and download the cleaned list.

Duplicate line remover

Updates as you type

Mode

What do you want to do? ?

Options

Sort

Transform

Filter: only keep lines containing ?

Actions

⌘/Ctrl+K clear · ⌘/Ctrl+Enter copy output

Input 0 lines · 0 chars One entry per line · drop a .txt file

Output —

Examples

How It Works

Each line of your input is compared against every line above it. The tool keeps the first occurrence of a line and skips the rest. A line is considered a duplicate when its normalised form matches a line already seen — normalisation is controlled by the options you set.

"Ignore case" lowercases lines before comparing, so `Apple` and `apple` collapse into one. "Trim whitespace" strips leading and trailing spaces before comparing, so ` apple ` and `apple` collapse into one. "Remove empty lines" drops blank lines entirely instead of treating them as a deduplicatable value. A filter can further narrow the output to only lines containing a substring.

"Keep order" preserves the order lines first appeared in. Turn it off to sort the output alphabetically (A→Z) or reverse (Z→A). In "Keep only duplicates" mode, the tool inverts the output to show only lines that appeared more than once. In "Count occurrences" mode, each unique line is tagged with how many times it appeared, so you get a frequency list rather than a dedup list.

Tips & Best Practices

Cleaning email lists: turn on "Ignore case" and "Trim whitespace" — most duplicate emails differ only in casing or stray spaces from CSV exports.

Deduplicating code identifiers: keep "Ignore case" OFF. `MyVar` and `myvar` are different symbols and collapsing them can break refactors.

For very large lists (50k+ lines), paste in chunks. Browsers can stall on multi-megabyte textareas — split, dedupe, then concatenate.

Use "Keep only duplicates" mode to audit a CSV column for repeats before importing — much faster than scanning visually.

If the result looks wrong, check "Remove empty lines". Invisible empty lines often count as their own "unique" value and inflate the output.

Frequently Asked Questions

Does the tool change the order of lines?

By default, "Keep order" is on — output lines appear in the order they first appeared in your input. Turn it off to get A→Z or Z→A sorted output.

Is my text sent to a server?

No. Duplicate detection runs entirely in your browser. Nothing is uploaded, logged, or stored anywhere on our side.

What counts as a duplicate with "Ignore case" on?

Lines are compared after lowercasing, so `Apple`, `apple`, and `APPLE` are all treated as the same value. The output preserves whichever casing appeared first.

What does "Trim whitespace" do?

It strips leading and trailing spaces and tabs before comparing. ` hello ` and `hello` collapse into one. Internal whitespace inside a line is preserved.

What is "Keep only duplicates" mode?

It inverts the tool — instead of removing duplicates, it shows only the lines that appeared at least twice. Useful for auditing a list for repeats.

What is "Count occurrences" mode?

It outputs each unique line with a count of how many times it appeared in the input — like a frequency table. Handy for quickly summarising log files or survey responses.

Can I remove empty lines too?

Yes — turn on "Remove empty lines". Otherwise the first empty line is kept and subsequent empties are treated as duplicates of it.

Is there a size limit?

There is no hard limit, but browsers can stall on extremely large textareas. For lists over ~100k lines or several megabytes, consider a desktop tool or splitting the input.

Can I upload a file?

Yes — drop a `.txt`, `.csv`, or `.log` file onto the input pane and its contents will be loaded. The file is read locally; nothing is uploaded.

How do I deduplicate a CSV column?

Paste only the column you care about (one value per line). If values contain extra whitespace, turn on "Trim whitespace". For case-insensitive matching (emails, domains), turn on "Ignore case".