Duplicate Line Remover

Kept 3 distinct line(s) and removed 1 duplicate(s).

How it works

• Normalised CRLF and lone CR line endings to LF.
• Compared each line as-is.
• Compared lines case-sensitively.
• Kept the first occurrence of each distinct line.
• Rejoined the surviving lines in their original order.

• The result is idempotent: a second run finds no duplicates.

Methodology

1. Normalise CRLF and lone CR to LF.
2. Build a comparison key from each line, trimming and/or folding case as selected.
3. Keep the first or last line for each distinct key.
4. Rejoin the survivors in their original order.

Usage

1. Paste the list with duplicates.
2. Choose whether to keep the first or last occurrence.
3. Run the tool and copy the unique lines.

Examples

These are examples, not current values: they use fixed inputs, while the interactive tool computes live results in your browser.

Keep first

{
  "text": "apple\nbanana\napple\ncherry",
  "keep": "first",
  "caseSensitive": true,
  "trim": false
}

Keep last

{
  "text": "apple\nbanana\napple\ncherry",
  "keep": "last",
  "caseSensitive": true,
  "trim": false
}

Case-insensitive

{
  "text": "Alpha\nalpha\nBETA\nbeta",
  "keep": "first",
  "caseSensitive": false,
  "trim": false
}

Limitations

Deduplicating a list

Line-based de-duplication is useful for log extracts, keyword lists and exports. Deciding whether the first or the last occurrence wins matters when the same key carries different surrounding data.