Back in 2017, we partnered with Emily Cobb from Ropes & Gray, LLC and the team at ACEDS to host a webinar covering the ins and outs of successful document review. It was a thoughtful, helpful discussion on strategies and tips for improving the document review process, and we covered a lot of ground during the one-hour program, from modern approaches to managed review, technology-assisted review (TAR), putting together a review team and more.
Attendees posed a lot of great questions and we shared them on the BIA Blog. Since managed review tools and strategies are always changing for the better, we thought it would be beneficial to refresh and re-share the Q&A to keep the discussion fresh and ongoing. We hope you find this information helpful!
1. What is the average rate of legal document review, per reviewer, on an hourly basis?
It all depends on the complexity of the legal document review protocol, really. Reviews with simple protocols and easy “yes” or “no” questions will go very quickly, but if your review project is highly complex, then the review rate will be slower.
For example, the average rate of legal document review for emails only, if you’re only coding for responsiveness, could be anywhere from 70-80 documents per hour. However, if your reviewers are sifting through formulas in spreadsheets, you can expect them to spend a lot more time on each document, not only to review the information but to open the document in its native software. While some review platforms provide a decent spreadsheet view, the best way to view all the information in a spreadsheet is still to open it in Excel or Google Sheets. That means a reviewer may only look at 20-30 documents in an hour.
In general, assuming that reviewers are looking at a mix of documents that include some spreadsheets, most reviewers average 40-50 documents per hour.
2. When a law firm associate is overseeing an eDiscovery/managed legal document review project, what is the role of the project manager at the vendor or corporate client?
Put simply, the role of the project manager is to work with a law firm associate to keep him or her informed on the status of each step of the project, including document collection, processing, search results, review and production. In addition to providing reports and metrics on the case, the project manager is also there to troubleshoot any issues or facilitate things like complicated search term requests.
A project manager at the corporate client would be in a similar position. There’s usually a point person at the corporation who takes the lead in facilitating data collections. During the webinar, our colleague and fellow presenter Emily Cobb pointed out that it’s the law firm’s neck on the line in terms of what’s reported out. As such, it’s the law firm associate that has the final say on most substantive issues.
3. Does outside counsel need to do 100% of the quality control of documents reviewed by contract attorneys or the managed review vendor?
Generally speaking, no. Of course, that assumes that you have a reasonable level of confidence in the process, the people and the vendor. Whether done in-house with contract attorneys or outsourced to a vendor, quality results depend on a quality process from the outset. It helps to have an ongoing dialogue between the review team and counsel so there is a process to track questions asked and answered. This promotes consistency and quality in the review product.
Quality control in document review typically involves random samplings throughout the review process. If the review is being handled by a vendor’s review team or contract attorneys, they usually have their own QC process, in addition to what outside counsel will do.
We suggest that the samplings happen more often and are more encompassing at the beginning of the process, looking at the results on an individual reviewer basis to get a sense of how accurately each person is coding. Also look to see if any problems are widespread, as this could mean the review protocol was not properly explained or understood.
It’s also good to look at overturn reports, which show when document coding was overturned by the QC person. If there are patterns there – such as certain types of documents that get changed often, or if one reviewer’s coding calls are overturned more regularly – you can gain insight into any potential issues and make adjustments as needed.
With a strong focus on quality, combined with both general and individual feedback addressing any quality concerns from the outset, you can help ensure that the overall process is a success. As that proceeds and you become comfortable with your reviewers and the protocols for that review project, the QC process can be scaled back a bit, but should still include random samplings of the entire document set throughout the review.
That said, there are certain documents for which we do recommend to have a 100% quality-check review, whether it’s done by outside counsel or senior level reviewers at your vendor (or a combination of both). At BIA, for example, we always make sure there is a second set of eyes from our team on all documents coded as privileged or “hot.”
4. How has the quality control process matured over time with the “modern” approach? How can we best leverage technology to track errors, etc.?
Technology now allows us to more easily locate items that need to be quality controlled. For example, we utilize TAR technologies not just in review, but also in our QC processes, helping to quickly highlight any quality concerns. In the past, we would have had to go through the entire document review process before there were insights to pull. Now, we can quickly and easily glean valuable insights, such as specific areas of documents that need to be quality controlled.
5. What kind of supervision do you give to a first or second-year associate managing an eDiscovery/managed legal document review project?
We do give more training and support to people who are new to the process. We believe it is BIA’s responsibility to our clients to make sure all associates – and really everyone involved – fully understand and are comfortable with the proven processes that we have established. We don’t want anyone to fail.
6. Why do firms not utilize eDiscovery staff attorneys more?
This is an interesting question. Before BIA, I (Barry) worked at a law firm that managed document review projects with more than 160 contract attorneys on multiple projects. What we found was that having so many contract attorneys on multiple projects kept us from building institutional knowledge for a case or client. We changed that process to employ 10-11 staff attorneys and recruited out of our existing talent pool of 160+ contract attorneys.
Staff attorneys cost less because they’re not on the partner track at a firm, but they still bill higher than contract attorneys – so there are two sides at play. But in general, staff attorneys build cost-effective institutional knowledge, and they provide consistent coding.
That’s the same philosophy that we use at BIA with our Managed Document Review Services team, which provides the benefits of staff attorneys – including being cost-effective and maintaining institutional knowledge – with the ability for our clients to use our attorneys as needed like one would with a contract attorney review team.
7. Are there any quality issues with the per-doc model’s incentive to get through legal document review quickly?
No, quite the opposite. With the per-document pricing model, the incentive is directly built into the model to get it right the first time around.
BIA’s preferred practice is to bill per document versus per hour, as we find it’s not only more predictable for the client, but easier for everyone to manage. We touched on this somewhat during the webinar, and it is discussed in more depth here.
Simply put, legal document review must be done accurately, or, regardless of how quickly it was done or how it was billed, it will have to be done again. Our per-doc model of review puts that burden where it should be – on us. If we don’t maintain the highest quality, then documents will need to be re-reviewed, which negatively impacts our profit on the project. Thus, inherent in the very model is the incentive for quality from the outset.
BIA’s well-designed process allows for reviewers to take the necessary time to code documents correctly the first time and includes both team management and our comprehensive quality control process all in one simple price. We are convinced of this model’s effectiveness for several reasons, but our favorite is that not a single BIA client utilizing this model has ever looked back.
8. If the other party does not specify format can you produce in the format you deem reasonable?
Per the federal rules, the answer is yes. However, what one side deems reasonable isn’t necessarily what the other side will agree upon. It’s good to confirm – more than once – what the specified format will be, just to avoid any back-and-forth in court. It’s not worth the money or time on either side to argue about the format. We suggest agreeing on an ESI production protocol at the outset of a matter to minimize ambiguity when completing productions.
9. How do you measure, monitor and track productivity and quality?
This is a good sum-up question. At BIA, we look at each individual reviewer’s rate of review (documents per hour) and then measure speed, accuracy, the difficulty of the review material, amount of errors per set and time spent reviewing. Conducting these measurements gives us a good indication of how the review is going and points out areas for improvement, be that for an individual or an entire team.
Want to streamline your document review? Check out our webinar, Document Review: The EDRM’s Final Frontier, that discusses how to approach, plan for and execute a successful document review.
Want to learn about using analytics in document review? In our recent Practical Uses of Brainspace webinar, we discuss how we leverage their technology to deliver unparalleled accuracy and cost savings to our Managed Document Review offering.