| Strategic Plan > Index to Action Papers > Enhance Current Quality Assurance Program by Integrating Reviews to Cover all Stages of Examination |
|
|
|
Action:
In order to enhance patent quality, the USPTO has implemented an improved Quality Assurance Program. The improved program integrates the various quality review programs into a single Quality Assurance Program in each Technology Center (TC), expands the quality review of work products, and improves the connection between the review's data collection and subsequent training development.
Background Information:
The USPTO recognizes that a robust quality assurance program is critical to maintaining the public's desired level of patent product quality. The public's interest in quality has been repeatedly and strongly voiced. The USPTO has repeatedly in the past expanded reviews of examiners' work products in order to obtain quality measurements and to enhance patent product quality.
Options Considered:
Option 1
- Integration of the Allowance Reviews of the Office of Patent Quality Review (OPQR) into the Technology Centers:
The USPTO established its OPQR end-checking process more than 25 years ago. In that time the measured error rates have oscillated between three to seven percent. More importantly, there has been no significant quality improvement since the establishment of OPQR. While the randomly sampled OPQR measures are statistically valid, they have proven difficult to use when building training programs. Particularly difficult has been the development of training programs to help individual examiners whose needs do not coincide with the Technology Center averages determined by OPQR statistical measures. Further, OPQR is a stand-alone organization at the end of the examination pipeline. This has made it difficult to modify the OPQR process to better connect their quality measurements into training program development.
The USPTO will integrate the allowance reviews currently performed by OPQR into the TCs by expanding the In-Process Review (IPR) program to include allowance reviews. The allowance reviews will be performed by Quality Assurance Supervisors (QASs).
Option 2
- Expansion of QAS IPR of Work Product in Patent Applications:
The USPTO established its current IPR of examiners' work products using a review standard based upon applicant/attorney feedback. The QASs for each TC review a random sample of 380 completed and mailed Office actions per fiscal year. A further sample from the reviewed pool is used for follow up with applicant's representative to get their judgment of the quality. The results of the random sample are used to determine general TC-wide training needs from which the QASs develop training courses. In addition to random IPRs, TCs can perform focused IPRs (e.g., an examiner may request a focused IPR to gather feedback in preparation for the signatory authority program, or a TC may do focused reviews of all primary examiners, or a particular technology area). In a focused IPR, feedback is used to determine examiner specific training needs or, alternatively, that the examiner's work product is of high quality and he or she can help train others.
QASs will continue to perform their current duties which comprise application quality review, training development, help panels, classroom teaching, one-on-one help sessions, special case review requests, and "second pair of eyes" reviews. In addition, the Office will expand the QASs' IPR of work products within the TCs to cover all stages of examination from first action to allowance. (Also see Option 5 below on search quality reviews.)
IPR results will be incorporated into the quality metrics for the USPTO. In addition to the IPR program there will continue to be other quality related programs. These additional programs include:
§ Day-to-day work product reviews.
§ "Second pair of eyes" reviews (option 4 below).
§ "Patent action survey" results collected from practitioners (see Quality 6 (P-39)).
§ Appeal conferences (with certified appeal conferees).
Data collected during IPRs will flow into numerous processes including:
§ Training development.
§ Examiner certification and re-certification.
§ Selection of SPEs, QASs, and Examiner Trainers.
§ Calculation of Technology Center quality measurements such as reopening rate.
§ Calculation of overall USPTO quality measurements.
The steps to implement this expanded IPR of primary examiner work products will comprise:
§ Increasing the resources available for examiner-focused IPRs.
§ Performing a focused IPR annually for every primary examiner.
Option 3
- Performance of Supervisory Patent Examiner Internal Process Reviews (SPE-IPR) quality reviews:
The USPTO established its annual performance appraisal plan for primary examiners several decades ago. A key component of each appraisal is feedback about quality given to the primary by their SPE. As a basis for the quality feedback, the direct SPE is required to review at least four applications per year for each of their primary examiners. These reviews also serve to keep the SPE directly connected to the work product produced in their technology area and attuned to the skills of people working within the area.
Option 4
- Expansion of Universal "Second Pair of Eyes" Review of Work Products:
In March 2000, the USPTO established the "second pair of eyes" review for business method patents in Class 705. It is a "universal" review of all allowances in Class 705 with each allowance taking about one hour. The purpose of this review is for the reviewer to quickly flag issues that need further consideration by the examiner and/or the examiner's supervisor.
In fiscal year 2001, the "second pair of eyes" review in Class 705 resulted in a reduction of the Class 705 reopening percentage by the OPQR to zero percent. For fiscal year 2002, only one 705 application was reopened as a result of review by the OPQR. The results of the "second pair of eyes" reviews are used to develop training in Class 705.
The USPTO will expand universal "second pair of eyes" reviews beyond allowances in Class 705 to other art areas with unacceptably high reopening rates. Also, the effectiveness of "second pair of eyes" reviews at earlier stages of the examination process will be evaluated. The results of these early stage reviews will be used to check the quality of work products during examination and to determine examiner training needs.
Option 5
- Evaluation of Quality of Search:
The adequacy of prior art searches performed by USPTO examiners is evaluated as part of the various quality reviews performed at the USPTO. These reviews include allowance reviews by the OPQR, "second pair of eyes" reviews performed by QASs, IPRs performed in each Technology Center, and day-to-day monitoring of examiner's work product.
It is being proposed that searches be performed by other parties in addition to examiners. It will be necessary to evaluate the adequacy of searches of all parties providing this work product to ensure the delivery of high quality searches. The monitoring of search quality will be integrated into the regular Quality Assurance Program. Data collection on search quality would occur at two data collection points as part of the standard review process:
1) Day-to-day review of searches by examiners as they begin examination; and
2) In-process review of applications (see Options 2 and 3).This option addresses only measurement of search quality. The establishment of the search criteria and actions to be taken after search evaluations are completed are both addressed in other papers. The criteria for the review of search quality at data collection points 1 and 2 will be based on the search criteria that are established (see related papers, for example Quality 8 (P-07-01)).
Option 6
- QAS Review:
Evaluation will be done by the QASs and others (such as reviewing SPEs) who will feed results to the QASs for addition to the quality databases. Measurement will be ongoing. QASs and reviewers will be measuring all year, with the data available for observation at any time. Data will be formally reported on a quarterly basis to specific designees (home SPE, work group SPEs, Group Directors, Commissioner, etc). Measurements will be loaded into databases on a regular basis by a program analyst as the reviews are completed. The QASs will be trained to do root cause analysis to make best use of the data.
During allowance reviews, measurements will be made of the following conventional item:
§ Reopenings when reviewer finds an unpatentable claim (replaces OPQR reopenings). This measurement will be made in accordance with current OPQR guidelines. This will allow baselines and benchmarks to be set early in the process using past performance. Additionally, the quality of the search will be evaluated during each allowance review (see Option 5).
During reviews of other actions, measurements will be made of the following conventional items:
§ Was the rejection clearly wrong? (35 USC 101, 102, 103)
§ Were claim limitations correctly matched to the prior art? (35 USC 102, 103)
§ Was the examiner's position properly supported? (35 USC 101)
§ Were the differences stated? (35 USC 103)
§ Was the modification stated/explained? (35 USC 103)
§ Was the motivation stated? (35 USC 103)
These measurements are identical or minor variations to current IPR measures. This will allow baselines and benchmarks to be set early in the process using past performance. Additionally, the quality of the search will be evaluated during each allowance review (see Option 5).
During reviews of all actions, measurements will be made of the following new item:
§ Is the file wrapper complete?
As this is a new item to be measured, benchmarks and baselines will be hard to establish quickly. However, a few sub-portions of this will be continuations of portions of the current OPQR allowance review, such as completeness of "Reasons for Allowance" and use of "Proper Motivation Statements." Also, file wrapper completeness review guidelines will need to be established.
USPTO Recommended Course of Action:
Details:
Fully implemented an integrated Quality Assurance Program (Options 1-6):
· Integrated OPQR functions into the TCs.
· Expanded the In-Process Review of work products to cover all stages of examination.
· Supplemented annual direct supervisor review of primary examiner work products with work group SPE-IPR quality reviews.
· Expanded the "second pair of eyes" review of work products.
· Incorporated an evaluation of search quality into the work product review process.
· Establishing quality metrics for the expanded Quality Assurance Program.
Proof of Concept
A series of quality projects have begun in fiscal year 2003. These quality projects have targeted those USPTO areas that most exhibit the problem that the change is designed to address.
Current USPTO quality reviews generate organizational measures, but do not provide measures that can be used to identify the quality needs of an individual examiner. The major goal of the expansion of reviews has been to identify quality trends and individual primary examiners with significant quality concerns and to provide sufficient help to address those concerns. This concept of quality improvement is being expanded throughout the Corps and specifically one TC with the greatest quality concerns based on the current USPTO quality reviews. While each TC has implemented expanded reviews, one TC has been provided with additional resources to independently review an added number of applications for every primary examiner in the TC. The key measurement for this project will be the level of improvement during subsequent reviews after quality concerns are identified and feedback and training has been provided to address those concerns. The milestones for the first year will be the performance of the reviews and the addressing of those concerns.
The concept of "second pair of eyes" review has already been proven through its successful use with business method patent applications over the last two years. During fiscal years 2001 and 2002, the OPQR reopened only one business method application as a result of their reviews. The use of "second pair of eyes" reviews will be expanded to other selected technologies where a need is identified.
The evaluation of search quality is being tested within the expanded IPR projects and, where appropriate, will include exploitation of search results (Work Sharing 1 (P-36) and PCT outsourcing (Flexibility 2 (P-28)). Data gathered during the first year of these projects will be used in establishing search criteria during subsequent years.
Currently, the QAS's duties comprise application quality review, training development, help panels, classroom teaching, one-on-one help sessions, special case review requests, and "second pair of eyes" reviews. Current shortages of QASs are covered by SPEs both full and part-time.
In addition to "Review QASs" and "Training QASs", each Technology Center's Quality Assurance Shop will absorb the Technology Center's current Special Programs function. The current Special Programs Examiners (SPREs) will move to "Special Programs QAS" positions in the QAS shop. Each TC will have "Special Programs QASs" equal in numbers to their current allocation of "Special Programs Examiners" (SPREs).
Estimated Benefits:
Quality: Quality of substantive examination will be enhanced since examiners will have better training programs based on improved feedback to examiners.
Cost-effectiveness: In the new multi-track process (Pendency 2 (P-01)) there will be a significant need to make sure all examiners are ready for re-certification. The Quality Assurance Program is a relatively low cost way to assess general and individualized examiner training needs and to maintain assurance of the desired knowledge, skills, and abilities for all examiners.
Implementation Plan:
The expansion of patent application work product reviews will soon yield a Quality Assurance Shop in each TC that will comprise three distinct types of Quality Assurance Supervisors. The numbers of each type will differ among TCs based on workloads (number of primaries, hires, etc).
1) "Review QASs" to perform (see Options 2, 4, and 5):
A) Primary examiner allowance reviews and in-process reviews,
B) Non-primary allowance reviews and IPRs,
C) "Second pair of eyes" reviews,
D) The IPR of the quality of prior art searches, and
E) Reexamination and Reissue final quality reviews.
2) "Training QASs" to perform:
A) Primary examiner training (including one-on-one help sessions),
B) Non-primary training and new hire Academy follow-up,
C) Special case reviews (e.g. help panels), and
D) Oversight of SPE-based IPRs (see Option 3).
3) "Special Program QASs" (replaces SPREs) to perform:
A) Reexamination preliminary reviews,
B) Reissue preliminary reviews,
C) Interference reviews, and
D) Petition processing.
Additionally each discipline (electrical, chemical, mechanical) will have:
-one Discipline Quality Lead (QL) [or Senior Review QAS]; and
-one program analyst (two for electrical as this = 1/2 of corps).
The three QLs have been selected by and report to the head of Patent Operations in order to preserve the independence of quality reviews and measurements. The QL will serve as the rating official for the "Review QASs" in their discipline and for the program analyst(s). Additionally, each QL will perform "Review QAS" application reviews in each Technology Center within their discipline. The QLs will be responsible for maintaining uniformity in the review processes.
| Work Breakdown Structure | Task Name | Start | Finish | Project Lead |
|---|---|---|---|---|
| 7 | Quality 1/Quality 3/Qualtiy5: P-17 - Expansion of Patent Application Work Product Reviews | 06/03/02 | 12/01/03 | A. MacDonald |
| 7.2 | Develop action plan for incorporating QR into TC (P-17) | 06/03/02 | 09/29/03 | |
| 7.2.1 | Obtain names of QR examiners and art examined (P-17) | 06/07/02 | 07/02/02 | |
| 7.2.2 | Notify TC to close all open QR cases (P-17) | 06/14/02 | 06/14/02 | |
| 7.2.3 | Develop QAS position description (P-17) | 06/06/02 | 06/06/02 | |
| 7.2.4 | Develop QAS PAP (P-17) | 06/03/02 | 06/03/02 | |
| 7.2.5 | Develop Administrator position description (P-17) | 06/03/02 | 09/30/02 | |
| 7.2.6 | Develop Administrator PAP (P-17) | 06/03/02 | 09/30/02 | |
| 7.2.7 | Select TC Lead QAS (P-17) | 07/01/02 | 08/02/02 | |
| 7.2.8 | Announce OPQR functions will be integrated into the TCs - notify employees (P-17) | 07/11/02 | 07/11/02 | |
| 7.2.9 | Notify POPA and NTEU 243 of QR incorporation into TCs (P-17) | 07/10/02 | 07/10/02 | |
| 7.2.11 | Notify D. Bertsch for planning space and equipment move for QR reviewers (P-17) | 06/17/02 | 06/17/02 | |
| 7.2.13 | Submit reassignment SF-52s to OHR (P-17) | 08/30/02 | 08/30/02 | |
| 7.2.14 | Transfer OPQR personnel to QAS positions (P-17) | 10/01/02 | 10/01/02 | |
| 7.2.15 | Expand OPQR Reviews (P-17) | 09/03/03 | 09/03/03 | |
| 7.2.16 | Announce new "Review QAS" positions (P-17) | 09/03/02 | 01/27/03 | |
| 7.2.17 | Panel the "Review QAS" positions (P-17) | 02/03/03 | 03/04/03 | |
| 7.2.18 | Interview and fill the "Review QAS" positions (P-17) | 03/07/03 | 05/29/03 | |
| 7.2.19 | Announce additional "Training QAS position (P-17) | 09/03/02 | 01/27/03 | |
| 7.2.20 | Panel the "Training QAS" positions (P-17) | 02/03/03 | 03/04/03 | |
| 7.2.21 | Interview and fill "Training QAS" positions (P-17) | 03/11/03 | 04/03/03 | |
| 7.2.22 | Training for managers, especially new QAS (P-17) | 04/04/03 | 09/29/03 | |
| 7.2.23 | Transfer current QAS positions to "Training QAS" positions (P-17) | 10/01/02 | 10/01/02 | |
| 7.2.24 | Transfer current SPREs to new "Special Program QAS" positions (P-17) | 10/01/02 | 10/01/02 | |
| 7.2.25 | Announce realignment of OPQR to Patents (P-17) | 07/10/02 | 07/10/02 | |
| 7.2.26 | Identify which positions and who will be reassigned where (P-17) | 07/08/02 | 07/10/02 | |
| 7.3 | Analyze customer responses on new quality review and integrate feedback into program (P-17) | 06/03/02 | 07/05/02 | |
| 7.4 | Finalize action plan for pilot program (P-17) | 06/03/02 | 08/14/02 | |
| 7.5 | Prepare OPQR for transition (P-17) | 06/17/02 | 09/13/02 | |
| 7.6 | Establish quality metrics (P-17) | 06/03/02 | 09/30/03 | |
| 7.6.1 | Allowance/IPR review form (P-17) | 06/03/02 | 01/02/03 | |
| 7.6.2 | X-case review form (P-17) | 06/03/02 | 01/02/03 | |
| 7.6.3 | Case review appeal process development (P-17) | 06/03/02 | 01/02/03 | |
| 7.6.4 | Statistical validity analysis of sampling (P-17) | 06/03/02 | 01/02/03 | |
| 7.6.5 | Establish FY03 quality targets (Reopenings) (P-17) | 06/03/02 | 01/02/03 | |
| 7.6.6 | Baselining IPR Metrics (P-17) | 10/01/02 | 09/30/03 | |
| 7.7 | Initiate pilot program by integrating quality review positions into TCs with highest error rates (P-17) | 03/03/03 | 09/30/03 | |
| 7.8 | Quality evaluations (P-17) | 01/02/03 | 09/30/03 | |
| 7.8.2 | Second quarter results (P-17) | 03/28/03 | 03/28/03 | |
| 7.8.3 | Perform workgroup IPRs (P-17) | 02/03/03 | 09/30/03 | |
| 7.8.4 | Second pair of eyes (P-17) | 03/05/03 | 08/29/03 | |
| 7.9 | Review results (P-17) | 10/01/03 | 10/31/03 | |
| 7.10 | Evaluate results and incorporate lessons learned (P-17) | 11/04/03 | 12/01/03 | |
| 7.11 | Define PALM requirements (P-17) | 06/03/02 | 08/27/02 | |
| 7.12 | Baseline PALM project (P-17) | 06/03/02 | 06/28/02 | |
| 7.13 | Enhance PALM (P-17) | 03/06/03 | 08/20/03 | |
| 7.14 | Complete implementation of PALM automation requirements for application work product reviews (P-17) | 08/22/03 | 08/22/03 |
Is there a question about what the USPTO can or cannot do that you cannot find an answer for? Send questions about USPTO programs and services to the USPTO Contact Center (UCC). You can suggest USPTO webpages or material you would like featured on this section by E-mail to the webmaster@uspto.gov. While we cannot promise to accommodate all requests, your suggestions will be considered and may lead to other improvements on the website. |
|
|
=online
business system 
=fees 
=forms
=help 
=laws/regulations 