By Barry Schwartz and Lisa Prowse
Last month, we partnered with Emily Cobb from Ropes & Gray, LLC and the team at ACEDS to host a webinar covering the ins and outs of successful document review. It was a thoughtful, helpful discussion on strategies and tips for improving the document review process, and we covered a lot of ground during the one-hour program, from modern approaches to managed review, technology-assisted review (TAR), putting together a review team and more.
Attendees posed a lot of great questions, and we wanted to share our answers here and keep the discussion going…
1. What is the average rate of review, per reviewer, on an hourly basis?
It all depends on the complexity of review protocol, really. Reviews with simple protocol, such as questions that can be answered by an easy “yes” or “no,” will go very quickly, but if your review project is highly complex, then the review rate would be slower.
For example, the average rate of review for emails only, if you’re only coding for responsiveness, could be anywhere from 70-80 documents per hour. However, if your reviewers are sifting through formulas in spreadsheets, you can expect them to spend a lot more time on each document, not only to review the information but to open the document in its native software. While some review platforms provide a decent spreadsheet view, the best way to view all the information in a spreadsheet is still to open it in Excel or Google Sheets. That means a reviewer may only look at 20-30 documents in an hour.
In general, assuming that reviewers are looking at a mix of documents that include some spreadsheets, most reviewers average 40-50 documents per hour during review.
2. When a law firm associate is overseeing an eDiscovery/managed review project, what is the role of the project manager at the vendor or corporate client?
Put simply, the role of the project manager is to work with a law firm associate to keep him or her informed on the status of each step of the project, including document collection, processing, search results, review and production. In addition to providing reports and metrics on the case, the project manager is also there to troubleshoot any issues or facilitate things like complicated search term requests.
A project manager at the corporate client would be in a similar position. There’s usually a point person at the corporation to take the lead in facilitating data collections. During the webinar, our colleague and fellow presenter Emily Cobb pointed out that it’s the law firm’s neck on the line in terms of what’s reported out. As such, it’s the law firm associate that has the final say on most substantive issues.
We also did a previous webinar on the role of the eDiscovery project manager, which goes more in-depth, and can be seen here.
3. Does outside counsel need to do 100% of the quality control of documents reviewed by contract attorneys or the managed review vendor?
In a perfect world, someone may quality check (QC) every single document. But in real life, the QC process is not 100%. It typically involves random samplings throughout the review process. If the review is being handled by a vendor’s review team or contract attorneys, they usually have their own QC process, in addition to what outside counsel will do.
We suggest that the samplings happen more often and are more encompassing at the beginning of the process, looking at the results on an individual reviewer basis to get a sense of how accurately each person is coding. Also look to see if any problems are widespread, which could mean that the review protocol was not properly explained or understood.
After you are comfortable with your reviewers, the QC process should include random samplings of the entire document set.
That said, there are certain documents that we do recommend have a 100% review. At BIA, for example, we always make sure there is a second set of eyes on all documents coded as privileged or “hot.”
It’s also good to look at overturn reports, which shows when document coding was overturned by the QC person. If there are patterns there – such as certain types of documents that get changed often, or if one reviewer’s coding calls are overturned more regularly – you can gain insight into any potential issues and make adjustments as needed.
4. How has the quality control process matured over time with the “modern” approach (as described in the webinar)? How can we best leverage technology to track errors, etc.?
Technology now allows us to more easily locate items that need to be quality controlled. In the past, we would have had to go through the entire document review process before there were insights to pull. Now, we can glean insights, such as specific areas of documents that need to be quality controlled, easily and quickly during the process.
5. What kind of supervision do you give to a first or second year associate managing an eDiscovery/managed review project?
We do give more training and support to people who are new to the process. We see that as part of what we do. It’s worth it to both us and our clients to make sure an associate – and everyone involved – is comfortable with how we work and the proven processes that have been established. We don’t want anyone to fail.
6. Why do firms not utilize eDiscovery staff attorneys more?
This is an interesting question. Before BIA, I (Barry) worked at a law firm that managed document review projects with more than 160 contract attorneys on multiple projects. What we found was that having so many contract attorneys on multiple projects kept us from building institutional knowledge for a case or client. We changed that process to employ 10-11 staff attorneys and recruited out of our existing talent pool of 160+ contract attorneys.
Staff attorneys cost less because they’re not on the partner track at a firm, but they still bill higher than contract attorneys – so there are two sides at play. But in general, staff attorneys build cost-effective institutional knowledge, and they provide consistent coding.
That’s the same philosophy that we use at BIA with our Managed Document Review Services team, which provides the benefits of staff attorneys – including being cost-effective and maintaining institutional knowledge – with the ability for our clients to use our attorneys as needed, like one would with a contract attorney review team.
7. Are there any quality issues with the per-doc model’s incentive to get through review quickly?
BIA uses a pricing model for review of billing per document versus per hour. We touched on this somewhat during the webinar, and it is discussed in more depth here. One big reason we have embraced this pricing model is because speed should not be the main goal of review. Review must be done accurately, or else, regardless of how quickly it was done, it would have to be done again. At BIA, our per-doc model of review allows for reviewers to take the necessary time to code documents correctly the first time, and it includes our comprehensive quality control process.
8. If the other party does not specify format can you produce in the format you deem reasonable?
Per the federal rules, the answer is yes. However, what one side deems reasonable isn’t necessarily what the other side will agree upon. It’s good to confirm – more than once – what the specified format will be, just to avoid any back-and-forth in court. It’s not worth the money or time on either side to argue about format.
9. How do you measure, monitor and track productivity and quality?
This is a good sum-up question. At BIA, we look at each individual reviewer’s rate of review (documents per hour) and then measure speed, accuracy, difficulty of review material, amount of errors per set and time spent reviewing. Conducting these measurements gives us a good indication of how the review is going and points out areas for improvement.
Thanks to all who joined us for our recent webinar. And if you missed it, you can check it out here. Stay tuned to our blog for more insights into eDiscovery, and look for our next webinar with ACEDS in January 2018!