This Month's Newsletter
Selecting an E-Discovery Vendor:
A Best Value Approach
It looks like e-discovery is here to stay. In fact, it is even becoming increasingly common. As such, it is only a matter of time until every litigator faces their first e-discovery project. When it comes, will they seek help from an outside vendor or try and muddle through on their own? If they seek help, how will they select a vendor?
The uninitiated are not the only ones facing the vendor selection problem, however. The rapidly changing technology, new requirements and the increasing number of service providers means that even an experienced practitioner with established resources can face the same question.
At first, selecting a vendor might seem simple. After all, why not select the low bidder?
While finding the lowest cost is certainly the objective, would one advise a client to find their attorney simply based on the lowest rate? Or, are there other factors to consider when calculating the lowest cost?
Even if vendors offer the same rate, do they all offer the same services as part of that rate? In addition, do they all offer the same level of quality for the services provided?
Indeed, there are many features that vendors can offer. Consequently, calculating the lowest cost means looking beyond the rate, comparing their offerings and normalizing their rates in order to determine the lowest cost.
When comparing vendor prices, their offerings can likely be separated into four areas: data preparation, volume reduction, productivity, and other considerations. The distinguishing features that buyers may want to consider in each area are discussed in the sections that follow.
Data Preparation
Whether you are using your own ESI management system or someone else’s, the data must be prepared for usage. Preparing the data for review is probably the most widely identifiable phase in the e-discovery process. It involves many common tasks like converting the native format files into PDF or TIFF images and populating the database fields. The process also has many nuances that can significantly influence case outcomes.
For example, selecting the data files for processing is often performed on the basis of criteria such as custodian, date stamps and file types. When the selection of file type is based on the document extension, important data files can be omitted if the user has changed the file’s extension in order to obscure its significance.
This kind of data hiding can be overcome by using header or signature analysis in order to properly identify desired file types. So, one of the features that should be evaluated along with the vendor’s price is the technique for selecting file types?
Another nuance involves the preparation of searchable text. If the text is simply obtained by recognizing characters in the imaged (TIFF or PDF) documents, the initial data could have a significant error rate that will reduce the effectiveness of subsequent searches. The kinds of errors include interpretation of the characters as well as the entire omission of characters.
Numerous factors can influence OCR accuracy rate. It can depend on the manner in which the initial images are derived such as direct from native files or scanned from printed pages. In addition, the resolution of the images can effect recognition accuracy. Other factors are the character size on the image and the effectiveness of the OCR tool. Frankly, there are numerous factors. These are just a few.
Text extraction from native files can be more effective, although it can miss text contained within embedded images. While imaging and then OCRing can eliminate this problem, OCR clearly introduces another set of issues.
When search effectiveness is reduced the chance of missing significant documents is increased. Of course, using fuzzy logic in search criteria can minimize this risk; however, it will also increase the size of document sets selected for review with corresponding increases in costs. So, another nuance worthy of consideration when evaluating a vendor’s price is the technique that is used by the vendor to obtain the searchable text and the accuracy rate it will have.
Another nuance that can be used to differentiate vendors is metadata. Metadata exists in two basic forms, system metadata and application metadata.
While vendors typically include metadata fields in the database structure, many also are omitted. For example, not all of the file system data stamps are included. Often the entry modified date stamp from a New Technology File System (NTFS) storage device is omitted.
Similarly, many attributes of application metadata are omitted. Part of the problem is the inconsistent structure of metadata fields between different document types and even similar document types. Furthermore, this is just for the standard metadata fields and does not even consider the custom fields that can be created by certain applications.
While all of the metadata exists in the native files, to what extent will that data be accessible in the user’s document system? If it is not easily accessible then it can be even more important that it be revealed by the vendor. So, another nuance worthy of consideration is the accessibility of metadata and the extent to which the vendor will reveal that data when the user’s system cannot access it.
Volume Reduction
The most expensive part of the e-discovery process is the review phase. It has been estimated that the review phase costs 20 to 40 times more per gigabyte than the processing phase. So, if processing typically runs between $500 and $1,000 per gigabyte, the review effort will cost between $10,000 and $40,000 per gigabyte. Amazingly, some have estimated that review can cost even more.
If this seems hard to believe, consider that 1 gigabyte is likely to have between 50,000 and 70,000 printable pages of textual data. If so, how many hours will it take to review that many pages? More importantly, how much will it cost?
Even at $100 per hour and about 2,000 pages per day, it will take someone 25 days to review 50,000 pages. Thus, the client’s costs would be about $20,000 per gigabyte or nearly $2 million to manually review a 100 gigabytes of data, which is fairly common data population in current e-discovery projects.
Naturally, if the gigabyte contains a large volume of image data, instead of textual data, then it would likely contain fewer printable pages and take less time to review. In any event, anything that can be done to reduce the review population can have a significant impact on the review costs as well as overall e-discovery costs. So, if the e-discovery vendor includes volume reduction services in its offering that can be a significant discriminator between the two prices.
For example, assume one vendor provides only data preparation at a price of $500 per gigabyte while another provides the same data preparation services along with some service that is likely to reduce the review volume by ten percent. The dilemma is that the second vendor’s price is $1,000 per gigabyte.
If the data population is 100 gigabytes the first vendor will cost $50,000 for data preparation which is half of the second vendor’s $100,000. Nonetheless, the second vendor is the better deal because, after considering review costs into the calculation and the savings that will be recognized from the reduced review volume, the overall costs are $50,000 less than if the first vendor was selected.
Clearly, the above example is a good illustration of why selecting an e-discovery vendor is not just an exercise in picking the low rate and why price normalization is essential to making a choice that will result in the lowest overall e-discovery cost.
It also is a good example of how important volume reduction can be to the e-discovery project. If the second vendor’s process could reduce the population by half than the second vendor could charge 10 times the rate of the first vendor and still have overall project costs less than the first vendor.
Of course, if the lawyer’s own ESI management system provides volume reduction capability then the capability of the vendor in this regard becomes less significant. The reality, however, is that most of the installed base of document management systems used by litigators are not very feature rich with volume reduction capability. While they may include some level of search capability they typically do not provide identification of known file types or deduplication capability. All of these are key volume reduction features that are discussed in the sections that follow.
Using Search to Reduce Volume
Review volume reduction frequently begins during the data preparation phase when potentially relevant documents are selected for processing from the overall document population based on date stamps, custodians and document types. Search results further reduce the review population. If the searches can be run prior to processing, then they can even reduce the processing population and its related costs.
Besides volume reduction in general, the above example clearly illustrated the importance of electronic search proficiency. It is just not practical to spend $2 million performing a manual document review on 100 gigabytes of data. Furthermore, not only is it not economically practical, a manual review of that much data does not ensure a higher accuracy rate.
Using electronic search technology for volume reduction is not limited to entering a few terms into a search engine. Indeed, crafting the terms can require linguistic analysis as well as intelligence in crafting the correct collection of discriminators like Boolean connectors and proximity locators. After all, the objective is to not only locate the critical documents but to eliminate the irrelevant or less significant ones as well.
So, while functions like fuzzy logic can be essential in capturing the significant documents, it must be carefully used if volume reduction is to be accomplished. Otherwise the volume may be increased instead of reduced.
Of course, for search technology to even be effective two conditions must exist. First the data must be searchable. Second, the searchable text must be representative of the original documents. Both of these conditions are criteria for discriminating between vendors.
With respect to the first one, searchable text, what will the vendor actually use to conduct its searches? If it is the native files themselves then how will they know which ones are searchable and which ones are not?
For example, PDF documents may be thought to be searchable but what if they are not? What if they simply contain images of documents that are not searchable? The same could be said of web pages, spreadsheets and word processing documents. What if they contain images instead of text? What if the files have been protected with encryption so that they are not searchable? How will such situations be identified and how will those documents be searched?
If the vendor is not using the native documents for searching or if the native documents are not searchable, the second discriminator, the representation of the searchable text, comes into play. Some vendors may find it simpler to image the documents and optically recognize their text afterwards. This solves the problems of the non-searchable file as well as the searchable file containing non-searchable data; however, it introduces another issue—how accurate is the recognized data?
As indicated previously, if the recognized data is not representative of the original documents as a result of misinterpretation of characters, significant documents could be missed by search criteria. Fuzziness could be added to compensate for the inaccuracy problem but then this would increase data volumes rather than reduce them.
Even automated correction mechanisms can be problematic if the errors are contained in the original document. After all those errors could be basis for subsequently locating a document within the population. If that unique characteristic is missing then it will no longer be locatable.
So, some error requiring fuzzy logic could be required, although those instances could be limited to more informal communications like e-mail and text messages. If so, the error rate issue could have different tolerances to different document types found in the population. It is also worth noting that the error rate concern applies to every kind of search no matter what the search engine. Context search is not going to minimize the problem. After all, context search is a sophisticated index based technique. So, this is not something that will be less significant for context based searches than keyword based searches. In any event, the consequence of this reality is worthy of consideration and provides a clear basis for discrimination between vendor solutions and rates.
Removal of Known File Types
Another method of volume reduction is known as the removal of known file types or de-NISTing. The de-NISTing name comes from the use of a database of known software files published by the National Institute of Standards and Technology (NIST) called the National Software Reference Library (NSRL).
The NSRL is, “a repository of known software, file profiles, and file signatures for use by law enforcement and other organizations in computer forensics investigations.” The data from which the NSRL is created is obtained through purchase or by donation from the original software publishers. The database contains information like the application name, the file name and its signature (digital hash). E-discovery vendors use the database to identify those files in their own population of documents that can be excluded from further consideration.
For example, the file types of interest in a particular matter may include text files and spreadsheets. Within a particular spreadsheet application there will be text files related to licensing and installation as well as spreadsheet files themselves that are part of a demonstration or tutorial library contained within the application. In addition there may be PDF files that are reference manuals about the application’s operation and usage. With the NSRL, these files can be identified and excluded from further consideration in an e-discovery project.
While most of the files contained within an application are usually program executables and other software files that would never be selected for consideration in most e-discovery projects, a small number of files could meet the selection parameters. When these few files are multiplied by the number of software applications on a custodian’s computer hard drive, the number of excluded files can be significant. So, the ability to identify and remove the known files from an e-discovery data population is another discriminator that can be used to evaluate a vendor’s rate for overall lowest cost.
Removing Duplicates
The final method of volume reduction to be discussed is deduplication. This can be a rather complex subject with significant consequences for determining volume reduction. Furthermore, their ultimate success may depend on other facets such as production capability.
For example, unless the parties have agreed to produce only the unique document instances, how can deduplication reduce the review effort if the deduplicated results cannot be exploded back to all instances in the population?
One of the ways that deduplication can be performed without providing exploded production is when there are multiple instances of the same data for the custodian. This can occur with backup tapes for example where the same data is captured in each period’s full backup.
In that instance, there may be multiple copies of the same custodian’s data and clearly only one need be produced. Some vendors refer to deduplication within the same custodian as vertical deduplication.
The duplication issue becomes more complicated, however, when multiple custodians have the same document. For example, when both sender and recipient have the same e-mail is there any need to review both even though there may be a need to produce both? Some vendors refer to deduplication across custodians as horizontal deduplication.
While horizontal deduplication will provide a smaller review set than vertical deduplication it can still have limitations, particularly when compound documents like e-mails are involved. For example consider the case where an e-mail is sent to a distribution list. After sending the e-mail the sender realizes that the list was incomplete and then sends the same e-mail to some other recipients that were not previously included. If all things about the e-mail were identical except the recipient list and the sent date and time, is there really any need to review it separately? More than likely this e-mail will not be excluded through either horizontal or vertical deduplication using any number of deduplication methods because of the different recipient list and sent date and time.
In fact, other facets not visible to the normal user would also be different and also prevent its exclusion through vertical or horizontal deduplication. Indeed, the hidden metadata of a compound document, like e-mail, provides several challenges for deduplication. For example, consider the case where an e-mail is sent from Atlanta to recipients in Los Angeles and New York. Also, consider that all three custodians are significant to the case and their e-mails are collected and reviewed.
Although the content of the e-mail will be identical for all three parties, they could not be deduplicated through vertical or horizontal deduplication. As a result, all three would appear in the review population.
The problem is that despite the message and even all of the other visible parts of the e-mail are the same, the message headers will be different for each. In addition to other data, the message header captures time and date stamps for each server that it passes as it winds its way through the internet from sender to recipient.
On the sender’s side it will not have traveled the internet. So, it will not have any date and time stamps. While the recipients will have date and time stamps for the servers they will likely be different, since the path from Atlanta to Los Angeles is likely different than the path from Atlanta to New York. As a result of the above, when each of the messages is hashed, it will have a different value.
With granular deduplication this problem can be solved. Under granular deduplication only the fields of interest are considered when computing the hash value. Thus, for determining the smallest review population the calculation can be limited to just the message body.
The prospect for granular deduplication can occur in other compound documents besides e-mail such as office documents. Spreadsheets, for example, can capture metadata about the last viewed or printed event. This update would make a change to the file that would make it different from other instances even though the rest of the document is unchanged.
When performing a privilege review, for example, what difference would the last print date or viewed date matter? If the deduplication test could be limited to content other than this kind of metadata the review population could be further narrowed.
Clearly, there are differences between vertical, horizontal and granular backup. Understanding exactly what kind of deduplication the vendor will perform and understanding its consequence on the review population is essential to determining lowest cost.
Even understanding the particular method used for deduplication can be significant. If the method is based on one of the accepted algorithms like MD5 or SHA1 then fine but what if the vendor has their own algorithm? If so, what is it? How does it work? How effective is it? Is it even meaningful? Will it survive challenge in the event a production problem arises?
As good as it is, deduplication is not without some practical challenges. For example, once the reduced population has been reviewed what will be produced? Will it only be the unique document versions or will all instances be produced? Unless the parties have agreed to only produce the uniques then all instances will be produced. If so, how will the deduped population be exploded back to all instances for production? Even if only the unique instances are produced will a list of all the duplicate documents and their locations be produced and how will that list be prepared?
If the deduplication has been achieved through vertical deduplication this is probably not an issue if all that has happened is that the same document has been removed from repetitive storage media like backup tapes. If the deduplication has been horizontal or granular the production problem and the explosion of the reviewed uniques to the total population is more evident.
The problem is easily solved with software logic, however. It is simple enough to locate matching documents in the global population and code them identically for either production or withholding. Of course, the problem can be complicated when the explosion involves redacted documents, although it is merely a complication. The solution is similar as non-redacted documents but with a few more twists.
The production issue is not the only challenge, however. Another problem area that must be solved in order for the deduplication promise to have maximum benefit, involves privileged documents when they exist in compound documents like e-mail. After all it is possible that some components of a compound document could be duplicates that will be eliminated if the smallest review population is to be derived.
For example, consider the situation where an attorney sends an e-mail instructing the client to perform the tasks listed in the attachment for the current litigation. Such a communication would likely be withheld as a privileged document.
Also, consider that the attachment is separated from the e-mail and saved to the recipient’s hard drive. The deduplication process includes the stored version in the review population while eliminating the e-mail attachment version from the review population as a duplicate.
When the e-mail is reviewed it will likely be marked for privilege and withheld. When separated from the e-mail the attachment is likely to have lost its significance and could be marked for production. When the actual production is performed, however, the production logic could be to withhold all components of a compound document if any element is marked as privileged. If so, the e-mail and its attachment would not be produced; however, what happens to the attachment version that has been saved to the custodian’s hard drive that has been marked for production?
This issue, too, can be solved with software coding logic. For example, the actual production of documents could be prohibited for documents that also exist in compound documents like e-mails. Their production could be prohibited until all elements of the compound documents in which they also exist have been reviewed. If any component of a compound document in which they exist is marked as privilege then it is withheld.
Of course, the above are just some possible solutions. There are likely many others. In any event, they clearly illustrate that the e-discovery challenge is complex and the nuances need to be understood when evaluating vendor prices. Certainly, the problems could be simplified by avoiding issues like volume reduction but that would also result in significant costs to the client. Equally bad is that the greater volume also increases the chance of error.
Review Productivity
The increased volume issues inherent in e-discovery are not the only ones worthy of attention. Since the review effort itself is so costly the entire process is worthy of consideration. If it takes 8 hours to review 2,000 document pages then that is about 25 seconds a page or about 1.75 minutes a document if there are an average of 4 pages per document. By itself that seems rather fast for something where a reviewer may actually have to read, consider and then mark each document accordingly.
If the reviewer could improve that rate by even 30 seconds a document that would save about 250 minutes per day or about $400 per gigabyte for a $100 per hour reviewer. When extended to the entire 100 gigabyte population that is about $40,000. That savings alone would just about pay for the cost of data processing. Naturally, it would be less if the volume reduction techniques are able to also cut the population.
While many litigation firms have already invested in their own ESI platform, they may still want to consider whether their e-discovery vendor offers a solution with better productivity savings. After all, the overall goal is to reduce discovery costs. While a litigator’s own system may host documents, will it actually facilitate the fastest and most economical review mechanism with the lowest overall discovery cost? So, when evaluating an e-discovery vendor’s price, litigators should consider other attributes like the vendor’s productivity features and project planning and management capabilities.
While the number of productivity features is limited only by the imagination of software developers, some of the more common methods involve multi-document marking, like document grouping, and near duplicates, on-line redaction, privilege log creation and virtual document swap.
The multi-document marking feature is more than grouping like documents and then iterating through them to apply identical coding. Rather, it is applying the same coding to a group of documents at one time. Conceivably this could provide the same savings as deduplication if the like documents were grouped based on their digital fingerprints. If so, it could also avoid the exploded production problems that are associated with deduplication, if the review population contained all instances. Even so, there is probably still some overhead associated with such an approach that would be totally avoided by taking the deduplication approach. For example, just iterating through the population and grouping on the identical documents would be extra steps that take extra time and computing horsepower. In addition, the multi-document marking on identical documents will not solve the issue of privilege documents within compound documents. Nonetheless, the multi-document marking feature is quite powerful when applied outside of the deduplication issue where documents of like characteristic have been selected.
Like document grouping involves the presentation of similar document types, like invoices, to have them reviewed as a group. Although it may still require individual examination, there are efficiencies that can be gained by having similar documents grouped together and presented to the reviewer one after the other. The condition for determining whether to produce or withhold a document of a certain type may always exist in the same location or have a similar characteristic. Thus grouping these together and having a reviewer iterate through them will likely be faster than just having them appear randomly in a document population to different reviewers.
Naturally, not all document types may lend themselves to this technique but, for those that do, it could result in considerable savings in time and costs. Furthermore, if a filter automatically removes them from the population as coded there might not even be any need for the reviewer to navigate through the group.
Near duplicates is another way of looking at similar document grouping; however, the similarities are more than the same type document. Indeed, near duplicates are likely different versions of the same document or perhaps a string of e-mail replies on the same subject.
Some document systems do not provide on-line redaction capability. Rather this is performed off-line through other systems and then the redacted document returned to the population. On-line redaction, streamlines this effort and reduces the chance of errors and inefficiencies in communication.
Another attractive cost cutting feature is the virtual document swap. After all preparing load files and exchanging data is just another unnecessary drain on the litigation budget. A virtual document swap takes the reviewed and cleared documents and makes them available to the opposing side without incurring the additional costs of load file preparation, exchange and data loading.
Even if the litigator’s platform were capable of this, a neutral expert would be the likely choice for such a feature. So, it may be something to consider when selecting an e-discovery vendor.
Better management of the project is another way to squeak out cost savings. Better management often depends on better management information, however. Information about the types of documents in the population and their various characteristics can better help organize and plan the effort. Once the project is underway, better information about progress and productivity can help redirect resources on the fly in order to improve overall performance.
Other intangibles
The subjects covered previously are not the only ones that may can be used to discriminate between vendor prices. Indeed, there are many others worthy of at least some consideration like security, storage, chain of custody, traceability, testifying experience along with many others.
While each of these can be significant they may not have direct effects on project costs. Two others that arguably could have direct effects on project costs are processing speed and search engine technology.
Speed
Schedule is often a significant factor in any litigation. It is no less significant in e-discovery. It might even be more important than in other cases simply because of the significantly larger volume of ESI.
As such, many vendors tout the speed at which they can process data. Before using this facet to differentiate vendors, it may be worth understanding the relevance of their claim to the case. After all, the speed of processing documents can depend on their type. For example, plain text files will process faster than other types of text based files like Microsoft Word documents. Spreadsheets will process even slower.
In addition, what happens if the documents are protected in some form? Protection from changing the document content will be easier to process than protection from even opening the document.
Sometimes an increased speed is achieved through other shortcuts. For example, it may be faster to deduplicate documents using a less sophisticated method than MD5 or SHA1.
Similarly, it may be faster to jettison problem documents to an exception status than make the extra efforts needed to extract text, extract metadata or convert to image formats. If so, how skewed are the results and how are the exceptions handled and what might the exception rates approximate? In fact, what is the basis for the claim? Is it the result of an actual project? Or is it merely the extrapolation of a test population? If it is an extrapolation, a linear extension is not likely realizable.
Search
The large data volumes typically encountered in e-discovery projects have made the use of computerized search technology essential. The old manual methods are just not practical.
As professionals have adopted computerized search techniques they have also learned that, while essential, the technology is not without its shortfalls. In fact, while it is seemingly simple the technology encompasses several complex subject matters that if misused can case real problems for its users. For example, if the search criteria are too inclusive then dispositioning the results can be prohibitively time consuming and expensive. On the other hand, if they are too narrow or ill-conceived the desired documents can be totally missed.
For some, the silver bullet to these issues has been context search. Simply stated context search adds relevance to the search criteria. For example, when one is searching for jaguars are they searching for football teams and their players, automobiles or large cats?
Context search solves this problem by embellishing on the traditional keyword index search model. More specifically, the index process of a context search engine collects more data about a document’s terms. The additional data along with synonym expansion, fuzzy logic and stemming technology are used in an algorithm to identify and usually rank the documents having the best match to the likely search objectives.
So, the context search technology bundles numerous advanced techniques in a simple user interface. Remarkably its simplistic black box approach can also be its shortcoming. After all, all of these additional components such as the additional data captured during the indexing process, the synonym library and algorithm are usually closely guarded secrets. Furthermore, they cannot be modified by the user. For example, relevancy is quite subjective. What might be relevant or even more relevant to one person is irrelevant or less relevant to another. Yet, that determination of relevancy is controlled by the algorithm used in the context search engine.
In the final analysis, context search is a sophisticated keyword search. The real difference is the extent to which features like synonym usage, relevancy ranking and concept discrimination have been programmatically scripted into the search engine functionality. The same capability is available with traditional keyword search tools but the implementation is totally dependent on the user’s skill to incorporate those features in the keyword search query.
Since the user cannot modify the algorithm’s used by the context search tool, it is possible for sophisticated users of sophisticated keyword search tools to even exceed the capability of context tools. After all, relevance and significance are subjective determinations.
What context tools offer is a sophisticated capability for less sophisticated users. What they do not offer is a silver bullet solution. They have limits. Furthermore, users are likely not to know how to assess those limits since their machinery is hidden “under the hood”. Of course, sometimes this can be useful when negotiating search protocols, since one party may be more motivated to impede the search process than to promote it. In those cases, a context search engine may make it more difficult for such an opponent to game the search process.
Conclusion
As e-discovery continues to become more widespread and more technical, practitioners will likely face the need for vendor assistance. When the time comes, how should practitioners chose a vendor? Is it just a matter of lowest cost?
While low cost is the obvious objective, is that necessarily achieved with the lowest vendor price? Possibly not.
When viewing the entire litigation lifecycle, there are numerous places where practitioners can cut costs. While processing is likely the most widespread activity, those activities related to the review function are where the biggest bang for the buck can be realized. After all the review function is the most expensive activity. Furthermore, it is about 20 to 40 times more expensive than data processing. So, even small savings in the review activity can have overall project savings that are large enough to offset the entire data processing phase.
Although it is possible that service buyers will have features directed toward review cost reduction, the reality is that vendors and their systems will have even greater opportunities. The most common area for review effort savings are in volume reduction and review effort productivity. So, service buyers should carefully investigate vendor offerings in these areas and know how to factor in those savings when evaluating vendor prices.