Strategic Plan > Index to Action Papers > Evaluation of Search Quality

Evaluation of Search Quality


In order to enhance patent quality, the USPTO will implement an improved Quality Assurance Program.  One important component of the improved program is enhancing the evaluation for the quality of the search product. 

Background Information:

The USPTO recognizes a robust quality assurance program is critical to maintaining the public's desired level of patent product quality.  The public's interest in quality has been repeatedly and strongly voiced.

Options Considered:

The adequacy of prior art searches performed by USPTO examiners is evaluated as part of the various quality reviews performed at the USPTO.  These reviews include allowance reviews by the Office of Patent Quality Review (OPQR), "second pair of eyes" reviews performed by Quality Assurance Specialists (QASs), in-process reviews (IPRs) performed in each Technology Center, and day-to-day monitoring of examiner's work product. 

It is being proposed that searches be performed by other parties in addition to examiners.  It will be necessary to evaluate the adequacy of searches of all parties providing this work product to ensure that the searches are complete and of the highest quality.  The monitoring of search quality will be integrated into the regular Quality Assurance Program.  Data collection on search quality would occur at two data collection points as part of the standard review process:

1) Day-to-day review of searches by examiners as they begin examination; and
2) In-process review of applications.

This option addresses only measurement of search quality.  The establishment of the search criteria and actions to be taken after search evaluations are completed are both addressed in other papers.  The criteria for the review of search quality at data collection points 1 and 2 will be based on the search criteria established.

USPTO Recommended Course of Action:

Incorporate an evaluation of search quality into the work product review process.  The evaluation of search quality is now included in all expanded work product reviews. Specifically, the adequateness of the search is being evaluated.  This is being accomplished by the QAS and SPE in their expanded IPR and allowance reviews.

Proof of Concept:

The evaluation of search quality will be tested within the expanded IPR projects, and where appropriate, will include exploitation of search results (Work Sharing 1 (P-36)) and PCT competitive sourcing (Flexibility 2 (P-28)).  Data gathered during the first year of these projects will be used in establishing search criteria during subsequent years.

Estimated Benefits:

Quality:  Quality of substantive examination will be enhanced since examiners will have better training programs based on improved feedback to examiners, as well and certified search services once the multi-track examination process (Pendency 2 (P-01)) is implemented. 

Cost-effectiveness:  In the new multi-track process there will be a significant need to make sure all examiners are ready for re-certification.  The Quality Assurance Program is a relatively low cost way to assess general and individualized examiner training needs and to maintain assurance of the desired knowledge, skills, and abilities for all examiners.
Implementation Schedule
Work Breakdown StructureTask NameStartFinishProject Lead
7Quality 1/Quality 3/Qualtiy5: P-17 - Expansion of Patent Application Work Product Reviews06/03/0212/01/03A. MacDonald
7.2Develop action plan for incorporating QR into TC (P-17)06/03/0209/29/03 
7.2.1Obtain names of QR examiners and art examined (P-17)06/07/0207/02/02 
7.2.2Notify TC to close all open QR cases (P-17)06/14/0206/14/02 
7.2.3Develop QAS position description (P-17)06/06/0206/06/02 
7.2.4Develop QAS PAP (P-17)06/03/0206/03/02 
7.2.5Develop Administrator position description (P-17)06/03/0209/30/02 
7.2.6Develop Administrator PAP (P-17)06/03/0209/30/02 
7.2.7Select TC Lead QAS (P-17)07/01/0208/02/02 
7.2.8Announce OPQR functions will be integrated into the TCs - notify employees (P-17)07/11/0207/11/02 
7.2.9Notify POPA and NTEU 243 of QR incorporation into TCs (P-17)07/10/0207/10/02 
7.2.11Notify D. Bertsch for planning space and equipment move for QR reviewers (P-17)06/17/0206/17/02 
7.2.13Submit reassignment SF-52s to OHR (P-17)08/30/0208/30/02 
7.2.14Transfer OPQR personnel to QAS positions (P-17)10/01/0210/01/02 
7.2.15Expand OPQR Reviews (P-17)09/03/0309/03/03 
7.2.16Announce new "Review QAS" positions (P-17)09/03/0201/27/03 
7.2.17Panel the "Review QAS" positions (P-17)02/03/0303/04/03 
7.2.18Interview and fill the "Review QAS" positions (P-17)03/07/0305/29/03 
7.2.19Announce additional "Training QAS position (P-17)09/03/0201/27/03 
7.2.20Panel the "Training QAS" positions (P-17)02/03/0303/04/03 
7.2.21Interview and fill "Training QAS" positions (P-17)03/11/0304/03/03 
7.2.22Training for managers, especially new QAS (P-17)04/04/0309/29/03 
7.2.23Transfer current QAS positions to "Training QAS" positions (P-17)10/01/0210/01/02 
7.2.24Transfer current SPREs to new "Special Program QAS" positions (P-17)10/01/0210/01/02 
7.2.25Announce realignment of OPQR to Patents (P-17)07/10/0207/10/02 
7.2.26Identify which positions and who will be reassigned where (P-17)07/08/0207/10/02 
7.3Analyze customer responses on new quality review and integrate feedback into program (P-17)06/03/0207/05/02 
7.4Finalize action plan for pilot program (P-17)06/03/0208/14/02 
7.5Prepare OPQR for transition (P-17)06/17/0209/13/02 
7.6Establish quality metrics (P-17)06/03/0209/30/03 
7.6.1Allowance/IPR review form (P-17)06/03/0201/02/03 
7.6.2X-case review form (P-17)06/03/0201/02/03 
7.6.3Case review appeal process development (P-17)06/03/0201/02/03 
7.6.4Statistical validity analysis of sampling (P-17)06/03/0201/02/03 
7.6.5Establish FY03 quality targets (Reopenings) (P-17)06/03/0201/02/03 
7.6.6Baselining IPR Metrics (P-17)10/01/0209/30/03 
7.7Initiate pilot program by integrating quality review positions into TCs with highest error rates (P-17)03/03/0309/30/03 
7.8Quality evaluations (P-17)01/02/0309/30/03 
7.8.2Second quarter results (P-17)03/28/0303/28/03 
7.8.3Perform workgroup IPRs (P-17)02/03/0309/30/03 
7.8.4Second pair of eyes (P-17)03/05/0308/29/03 
7.9Review results (P-17)10/01/0310/31/03 
7.10Evaluate results and incorporate lessons learned (P-17)11/04/0312/01/03 
7.11Define PALM requirements (P-17)06/03/0208/27/02 
7.12Baseline PALM project (P-17)06/03/0206/28/02 
7.13Enhance PALM (P-17)03/06/0308/20/03 
7.14Complete implementation of PALM automation requirements for application work product reviews (P-17)08/22/0308/22/03 
KEY: e Biz=online business system fees=fees forms=formshelp=help laws and regs=laws/regulations definition=definition (glossary)

Is there a question about what the USPTO can or cannot do that you cannot find an answer for? Send questions about USPTO programs and services to the USPTO Contact Center (UCC). You can suggest USPTO webpages or material you would like featured on this section by E-mail to the While we cannot promise to accommodate all requests, your suggestions will be considered and may lead to other improvements on the website.