Duplicate Line Remover

Long lists copied from spreadsheets, logs, exports, or scraped data often contain the same entry many times — sometimes dozens of times. This tool removes exact duplicate lines while keeping your original ordering intact, and tells you exactly how many it stripped so you can verify the cleanup made sense.

How it works

The input is split on newlines. For keep-first, a set tracks which line keys have already been seen, and any line whose key is already in the set is skipped:

for each line:
  key = caseInsensitive ? line.toLowerCase() : line
  key = trim ? key.trim() : key
  if key not in seen: keep line, add key to seen

For keep-last, the tool makes a first pass to record the index of the final occurrence of each key, then a second pass emitting only the lines that sit at those final indices. Either way the surviving lines stay in their natural position — nothing is re-sorted. The removed-count at the end lets you confirm how many duplicates were present.

When to use each option

Keep first vs. keep last — for most lists such as email addresses or product SKUs, first-occurrence is correct: the original entry is the canonical one and later repeats are noise. Use keep-last for logs or config-file lines where a later entry overrides an earlier one, for example a file of key=value settings where the same key appears twice and the later value should win.

Case-insensitive matching — essential for email lists ([email protected] and [email protected] are the same address), domain lists, and any text where case is not semantically meaningful. Without it, two lines that are visually identical but differ in capitalisation survive as distinct entries.

Trim whitespace — important for data copied from terminals, PDFs, or cell-by-cell spreadsheet exports, where invisible trailing spaces or tabs are common. Without trimming, London (with a trailing space) and London are seen as different lines and both survive.

Practical use cases

• Email list hygiene: paste a subscriber export, enable case-insensitive and trim, keep first — removes re-subscriptions and duplicate sign-ups.
• URL deduplication: paste crawl output or a sitemap draft and strip repeated URLs before further processing.
• Log line deduplication: keep-last with case-sensitive matching removes repeated identical error lines while retaining the most recent occurrence.
• Keyword list cleanup: paste an auto-generated keyword list from multiple tools, strip duplicates, then sort alphabetically in the Multi-line Text Sorter.

The tool runs entirely in your browser; nothing you paste is uploaded or stored.