Discovery & ESI

Deduplication and Threading for Message Reviews: Cut Volume, Not Context

Email threading deduplication explained for review teams: hash-based dedup, inclusive emails, and why text messages need different matching rules.

By CourtBriefly Editorial TeamUpdated October 7, 2026 · 8 min read

Quick answer

Deduplication removes exact copies of the same item, usually by comparing hash values, so reviewers see each document once. Email threading groups replies and forwards into conversations and identifies inclusive emails that contain the whole branch. Both cut review volume. Text messages need different rules, because the same message exported from two phones rarely matches byte for byte, so matching must rely on normalized sender, time and text.

Deduplication suppresses identical copies; threading organizes conversations. They solve different problems.
Email dedup typically hashes selected fields because the same email differs byte-for-byte across mailboxes.
Dedupe at the family level and record every custodian who held a suppressed copy.
Text messages need tolerant, documented matching rules, with ambiguous pairs sent to a human.
Disclose your dedup and threading method in the ESI protocol before production.

In this guide

What is the difference between deduplication and threading?
How does hash-based deduplication work?
How does email threading work?
Why are text messages harder to deduplicate?
How do you deduplicate screenshots against exported messages?
A defensible workflow for a mixed message review
What should the ESI protocol say about dedup and threading?
Frequently asked questions

Email threading and deduplication are the two cheapest ways to shrink a review without losing information, and they are routinely confused. Deduplication suppresses exact copies so each item is reviewed once. Threading organizes replies and forwards into conversations so reviewers read them in order, often only once. Used together, they can take a sprawling collection down to a manageable set; used carelessly, they hide recipients, strip attachments from their emails, or merge two messages that only looked alike.

This guide explains how each technique works for email, why text and chat messages break the usual assumptions, and a defensible workflow for a mixed message review. It also includes sample ESI protocol language, because whatever method you use should be disclosed and agreed before production, not explained after a challenge.

What is the difference between deduplication and threading?

Deduplication answers “have we seen this exact item before?” Threading answers “where does this item sit in a conversation?” A collection can need both.

Technique	What it does	Main risk
Exact deduplication	Suppresses items with identical hash values	Losing track of who else held the copy
Near-duplicate grouping	Groups highly similar items for consistent coding	Treating a meaningful edit as noise
Email threading	Groups a conversation and flags inclusive emails	Missing a recipient or attachment present only in an earlier email
Message threading	Groups chats by conversation and participant set	Merging group and one-to-one threads

How does hash-based deduplication work?

A hash algorithm turns content into a fixed-length fingerprint. Identical content produces an identical hash; change one character and the hash changes completely. Deduplication compares those values and keeps one copy of each. For background, see what a hash value proves.

Files versus emails

For loose files, the hash is usually computed on the file itself. Emails are different: the same message stored in two mailboxes typically differs at the byte level because of mailbox-specific headers and storage. Review tools therefore commonly compute an email hash from selected fields, such as sender, recipients, subject, sent date, body and attachment hashes. Which fields are used determines what counts as a duplicate, so know your tool’s settings before you describe them to opposing counsel.

Global versus custodial

Custodial deduplication removes copies only within one custodian’s data. Global deduplication removes copies across all custodians and should populate a field listing every custodian who held the item. Without that field, global dedup quietly erases the fact that a particular person received a message, which may be the very fact in dispute.

Family-level deduplication

Dedupe emails and their attachments as a family. An attachment should not be suppressed because an identical standalone file exists elsewhere, or the produced email arrives without its attachment.

How does email threading work?

Threading reconstructs conversations from email header fields, such as the message identifier and reply references, supplemented by subject and text analysis. The tool then identifies inclusive emails, meaning messages whose content is not wholly contained in a later message, typically the last email in each branch plus any email with a unique attachment.

Reviewing only inclusive emails can cut reading time sharply, because each later message quotes the earlier ones. Three cautions apply:

Recipients change. Someone copied on an early message may be dropped later, or blind-copied and invisible in the quoted text. The earlier message’s metadata matters.
Quoted text can be altered. A sender can edit the quoted portion of a reply. The quote is not proof of what the original said.
Review is not production. Coding inclusive emails for efficiency is different from producing only inclusive emails. That second step needs agreement.

Why are text messages harder to deduplicate?

Text and chat exports lack the stable identifiers email carries, and the same message rarely looks identical from both sides of a conversation. Byte-level hashing that works for email often fails here.

Two phones, two timestamps. The sender’s device records when it sent; the recipient’s records when it arrived. They can differ, sometimes significantly if a phone was off.
Time zones. Each export may display local time for a different zone, or UTC.
Contact names differ. One phone shows “Mom,” the other a full name. Match on the number or account handle, never the display name.
Attachments get recompressed. A photo sent by MMS or a chat app may be resized in transit, so the two copies hash differently even though they show the same image.
Reactions and edits. A reaction may export as a separate text such as Liked followed by the quoted message, and edited or unsent messages export inconsistently across tools.
Channel fallback. A message can appear as both an app message and an SMS fallback in some exports.
Group versus one-to-one. The same people can have a group thread and a private thread. Merging them misrepresents who saw what.

How do you deduplicate screenshots against exported messages?

You usually can’t do it with hashes, and you often shouldn’t suppress the screenshots at all. A screenshot is an image of a conversation; an export is structured data from the device. They will never share a hash value, even when they show the same words.

The practical approach is to treat screenshots as a separate evidence type and link them to the export rather than delete them:

Extract the text from each screenshot, then have a reviewer correct it against the image.
Match the extracted text to messages in the export by sender, approximate time and wording.
Where a screenshot matches, note the corresponding export entry and Bates range, and use the export as the primary exhibit if the attorney prefers it.
Where a screenshot has no match, flag it. It may show a message deleted before the export was made, a different thread, or a different device, and each possibility needs follow-up with the client.
Keep every screenshot’s original file and fingerprint either way. A screenshot that duplicates an export today can become important if the export is later challenged.

This cross-referencing is also how you catch the case where the screenshot and the export disagree, which is a question for the attorney to resolve before anything is produced.

A defensible workflow for a mixed message review

The goal is fewer items to read with every source still traceable. This sequence works for email and messages together.

Preserve and fingerprint. Keep every source export untouched and hash it on arrival. Work only on copies.
Remove identical exports. If the client sent the same export file twice, file-level hashing catches it.
Normalize. Convert all timestamps to one stated time zone, standardize phone numbers and email addresses, and resolve each handle to a person in a separate key.
Deduplicate emails at the family level. Use field-based hashing, and populate an all-custodians field if deduplicating globally.
Match messages conservatively. Merge only high-confidence pairs, and record every source each merged message came from.
Thread. Group email by conversation and messages by participant set, keeping group and one-to-one threads apart.
Review in context. Read message threads in day-sized or similar units rather than isolated hits.
Keep a methodology note. Record tools, settings, match rules and counts before and after, so you can explain the process if asked.

A tool like Message Extractor handles the normalization and merge steps for common phone and app exports, placing everything into one thread model with duplicates merged and a SHA-256 digest of the output. Whatever you use, the human checks on ambiguous matches remain your team’s job.

What should the ESI protocol say about dedup and threading?

Disclose the method, preserve the ability to recover suppressed items, and agree on what production includes. Raise it at the conference; the ESI meet-and-confer checklist covers where it fits.

Sample protocol language

A producing party may deduplicate exact duplicates globally at the family level using hash values. The producing party will populate an All Custodians field identifying each custodian who possessed a suppressed duplicate. Near-duplicate analysis and email threading may be used to organize review but will not be used to withhold responsive, non-privileged documents, and all responsive non-inclusive emails will be produced unless the parties agree otherwise. For text and chat messages, duplicates collected from more than one device or account may be consolidated only where sender, conversation, message text and timestamp correspond, and the producing party will identify each source from which a consolidated message was collected.

For how threaded, deduplicated messages become a production set, see producing text messages in discovery, and for what a consolidated output looks like, see a sample of a merged, paginated message log. The rules a court will expect may vary by forum, so confirm local requirements with the attorney responsible for the production.

Frequently asked questions

What is the difference between global and custodial deduplication?

Custodial deduplication removes duplicates only within each custodian’s data, so the same email appears once per person who had it. Global deduplication removes duplicates across all custodians, keeping one copy and listing every custodian who held it in a separate field. Global saves more review time; custodial preserves a simpler picture of who had what. Agree on which you will use before production.

What is an inclusive email in email threading?

An inclusive email is one whose content is not fully contained in any later email in the thread, usually the last message in a branch or one with a unique attachment. Reviewing inclusive emails lets a team read every word of a conversation once. Non-inclusive emails still exist and may matter for recipients or timing, so production rules should be agreed separately.

Is near-duplicate detection the same as deduplication?

No. Near-duplicate detection groups documents whose text is highly similar, such as drafts of the same letter, so reviewers can code them consistently. It usually does not suppress anything, because small differences between drafts can be exactly what matters. Exact deduplication, by contrast, removes items whose hash values match, meaning the content is identical.

Can deduplication remove relevant evidence?

It shouldn’t if it is done properly. Exact deduplication removes only identical copies, and good practice keeps the originals, records every source of a suppressed copy, and dedupes at the family level so attachments stay with their emails. Problems arise when matching rules are too loose, which is a particular risk with text messages, where human review of uncertain matches is the safer approach.

Should you produce only inclusive emails?

That is a negotiated point, not a default. Producing only inclusive emails reduces volume, but the requesting party may want every message to see exact recipients, timing and metadata for each. Many teams use threading for review efficiency while producing all non-privileged responsive emails. Whatever you choose, put it in the ESI protocol so neither side is surprised.