Text Deduplicator

Kept 3 unique item(s) and removed 1 duplicate(s).

How it works

• Split the input into words.
• Compared items case-sensitively.
• Kept the first occurrence of each distinct item.
• Rejoined the surviving items in their original order.

• Word scope rejoins with single spaces, so original spacing is not preserved.
• The result is idempotent: a second run finds no duplicates.

Methodology

1. Split the input according to the chosen scope.
2. Build a comparison key, folding case when the comparison is case-insensitive.
3. Keep the first or last occurrence of each key.
4. Rejoin the survivors in their original order with the scope's separator.

Usage

1. Paste the text to de-duplicate.
2. Choose the scope (lines, words or commas) and keep rule.
3. Run the tool and copy the unique result.

Examples

These are examples, not current values: they use fixed inputs, while the interactive tool computes live results in your browser.

Unique words

{
  "text": "alpha beta alpha gamma",
  "scope": "words",
  "keep": "first",
  "caseSensitive": true
}

Unique comma items

{
  "text": "a, b, a, c",
  "scope": "commas",
  "keep": "first",
  "caseSensitive": true
}

Case-insensitive lines

{
  "text": "Alpha\nalpha\nbeta",
  "scope": "lines",
  "keep": "first",
  "caseSensitive": false
}

Limitations

Choosing a de-duplication scope

Deduplicating words is useful for keyword extraction; deduplicating comma items suits tag lists and CSV cells. Choosing the scope avoids treating a whole line as one item when you only want repeated tokens removed.