Duplicate Line Remover

Does deduplication preserve the original order of lines?

Yes. The algorithm walks the input top to bottom and keeps the first occurrence of each unique line while discarding subsequent duplicates. The relative order of kept lines matches their order in the input. This matters for log files, timeline data, and any list where position carries meaning. If you want alphabetically sorted output instead, sort the result separately (most editors have a sort-lines command) or paste pre-sorted input. For use cases where order is arbitrary, sorting first has a small performance benefit and makes diffs between runs more stable.

What is the difference between case-sensitive and case-insensitive mode?

In case-sensitive mode (default), 'User' and 'user' are treated as distinct lines and both are kept. In case-insensitive mode, they are treated as duplicates and only the first occurrence is kept, preserving its original casing. Choose based on your downstream system: email providers, DNS names, and most modern databases treat identifiers as case-insensitive, so deduping email lists or domain names case-insensitively makes sense. Unix filenames, URLs with query parameters, and content-comparison use cases are case-sensitive — dedupe preserving case to avoid collapsing distinct entries into one.

Can this handle CSV files with headers or JSON Lines format?

It treats all input as line-delimited text, so a CSV with a header row will dedupe correctly as long as no data row happens to exactly match the header (extremely unlikely). JSON Lines (one JSON object per line) also dedupes correctly — two lines with the same JSON object string will collapse. The limitation is that JSON objects with the same semantic content but different key order ('{a:1,b:2}' versus '{b:2,a:1}') are treated as different lines because the dedupe is string-based, not structure-aware. For structural JSON dedupe, pre-normalize the JSON with key sorting via the json-formatter tool before pasting here.

Why are my 'identical' lines not being removed?

Usually one of: invisible whitespace differences (trailing spaces, tab versus space indentation, BOM characters, non-breaking spaces instead of regular spaces), different line endings (Windows CRLF versus Unix LF), case differences if case-sensitive mode is active, or Unicode normalization differences (é as one character versus e plus combining accent). Turn on whitespace trimming to handle the first two, switch to case-insensitive for the third, and for Unicode issues the only fix is to normalize input through a Unicode NFC normalizer before pasting. The word-counter or text-case-converter tools can sometimes help surface hidden whitespace problems.

Is the data I paste sent to any server?

No. Deduplication runs entirely in JavaScript inside your browser tab using a native Set data structure. There is no fetch, no server call, no logging of content. The input never leaves the page. This matters when the list contains PII (email addresses, customer names, phone numbers), authentication material (SSH keys, API tokens, passwords), or any data subject to privacy regulations (GDPR, HIPAA, CCPA). You can verify this in your browser's developer tools — the Network tab will show zero activity during deduplication. For maximum paranoia, disconnect from Wi-Fi before pasting; the tool continues to work identically.