Quality metrics

At the start of each fiscal year, a goal for the total number of OPQA random reviews is set based on the USPTO’s data needs and available resources. In recent years, the goal has been to randomly review approximately 12,000 work products in a given fiscal year. This yearly volume of reviews provides sufficient data to identify corps wide trends, provide TC level insight for select topics, and allows the USPTO to answer many inquiries from our stakeholders in a timely manner.

The random sample of work products reviewed by OPQA is representative of the population of work products the Patents organization completes and mails, both by Office action type (i.e., non-final rejection, final rejection, and allowance) as well as by technology-type. To accomplish this, OPQA randomizes all Office actions mailed within a previous seven day period to create a randomized pull list. Reviewers from OPQA are then assigned Office actions for review from the randomized pull list based on their assigned technology area.

OPQA reviews work products based on the statutory compliance standard. Examiner Office actions are reviewed for statutory compliance by evaluating whether the Office action includes appropriate and clear determinations for every pending claim based on the four patentability statutes:

·        35 U.S.C. §102 - Novelty;

·        35 U.S.C. §103 - Non-Obviousness;

·        35 U.S.C. §112 – Specification (Enablement, Written Description, Definiteness);

·        35 U.S.C. §101 – Inventions Patentable (Subject Matter Eligibility, Utility).

To comply with the statutory compliance standard, rejections must be correct and must, at a minimum, appropriately (1) identify the claim and relevant statute and (2) set forth sufficient evidence to put a person skilled in the art on notice as to why the claim is considered unpatentable. In addition, failure to make a rejection under a statutory basis that should have been made (i.e., omitted rejection) results in the work product not complying with the statutory compliance standard.

The Master Review Form (MRF) is a modular review form used by reviewers in OPQA and the TCs to evaluate work products for statutory compliance as well as to evaluate quality in other areas of prosecution (e.g. search, response to arguments, etc.). The MRF includes over 20 sections and a 330+ questions library. In addition to the sections focused on statutory compliance and other quality related information, the MRF was updated in fiscal year (FY) 2020 to include “Characteristics” sections throughout the form to capture more specific information about use of best practices and to provide better insight into Office action clarity than in the past. The Characteristics sections also serve as the basis for designating of work products as an accolade. OPQA reviewers designate select high quality work products as an accolade for including best practices that go beyond the requirements of the Manual of Patent Examining Procedure (MPEP).

View a copy of the current Master Review Form.

In evaluating work products under the statutory compliance standard, OPQA reviewers evaluate every rejected claim in an Office action to ensure that the rejection of the claim was proper relative to each statute under which the claim is rejected.

By evaluating all claims under each statute, the USPTO is in fact performing many statutory compliance evaluations when reviewing an individual Office action. If the review finds that any single claim has an improper determination under any statute (e.g., an improper rejection or an improper omitted rejection) then the entire Office action is identified as non-compliant regardless of how many proper determinations were made or whether the non-compliance is the result of an independent claim or dependent claim.

If all the claims treated in the Office action are treated correctly under every statute then the case is found to be compliant. Any Office action where there was at least one claim found to be non-compliant is ultimately verified by an OPQA supervisor and sent to the relevant TC for consideration. If the receiving TC disagrees with an OPQA finding of noncompliance, they may file a rebuttal and OPQA has a process to reconsider the finding of noncompliance based on the arguments presented. The appropriate course of correction for work products ultimately finalized by OPQA as noncompliant is then left to the TC. For allowances specifically, this includes correction of the Office action prior to issuing an allowance of the patent application.  

The random sample of Office actions reviewed by OPQA for statutory compliance generates an overall compliance metric for each statute. The metrics are calculated by dividing the total number of Office actions that properly evaluated all pending claims in light of the patentability statutes by the total number of Office actions reviewed. The statutory compliance metrics are then expressed as a percentage of Office actions reviewed that properly handled all claims in the application and can be further broken by relevant statute.

The compliance metrics generated by OPQA are validated in part through TC feedback as part of the rebuttal process, by comparisons to quality reviews performed in the TCs and data collected from external perception surveys. In addition, OPQA does not have any targets or incentives based on the findings of noncompliance, but does monitor the consistency of OPQA reviews as an additional validation.

The granularity of data obtained by reviewing all claims in an Office action for statutory compliance provides meaningful feedback to TC management and quality assurance specialists and facilitates the identification of quality trends, training opportunities, as well as an evaluation of recent training at the corps level and below.

This section includes a summary of data collected in the MRF as a result of the OPQA random reviews over the last fiscal year.

MRF Data Summary Table – Fiscal Year (FY) 2023

In addition to the random reviews that underpin the statutory compliance metric, OPQA conducts numerous other reviews throughout each fiscal year, often in partnership with the TCs. This allows for investigation of specific quality issues relevant to our stakeholder community as well as to the specific needs of each TC.

The USPTO leverages various data sources to monitor for consistency. This includes evaluating certain types of transactions in the Patent Application Location and Monitoring (PALM) system as well as the Master Review Form (MRF) to identify trends and examiner behaviors indicative of either best practices or potential quality concerns.

In addition to providing consistency data to supervisors, examiners are provided with on-demand access to their own examiner statistics for a better understanding of their prosecution trends and decision-making compared to the average in their art unit, TC, and the overall corps.

Internal Stakeholder Perception Survey: the internal survey is sent to 750 randomly selected patent examiners on a semi-annual basis.

External Stakeholder Perception Survey: the external survey is sent to 3,000 of our frequent-filing customers on a semi-annual basis. A rotating panel effect is used to minimize respondent burden but still facilitate comparisons across surveys. Respondents are stratified into panels and asked to participate in two consecutive survey periods, with two unique panels participating in each survey. Each collection period, 50% of the respondents (or one panel) are rotated out and replaced with a new panel. Customers rotating out of the survey are not selected for further participation until the sample frame is refreshed and new panel assignments are made, which occurs approximately every three years.

To maintain objectiveness and anonymity of respondents, the surveys are administered and analyzed by a third-party contractor that performs the work under the guidance of the USPTO’s Chief Statistician.

The internal quality survey administered to patent examiners focuses on internal and external factors impacting examiners’ ability to provide high-quality patent examination. Internal factors address examiner satisfaction with topics such as patent examination tools, training, and the coaching and mentoring they receive. External factors address examiner satisfaction with incoming patent applications and applicant interactions during prosecution.

The external quality survey is designed to provide customer perspectives on the correctness, clarity, and consistency of rejections made by examiners. The survey also gathers perceptions about examiners’ adherence to rules and procedures and satisfaction with search and prior art. While the survey questions remain static to facilitate longitudinal analyses, a single open-ended question is incorporated during each enumeration to explore current topics of interest to the USPTO, such as specific effects of recent quality efforts or considerations for pending quality initiatives.

External

2023
External Quality Perceptions Survey – Fiscal Year 2023, Quarter 4
External Quality Perceptions Survey – Fiscal Year 2023, Quarter 2


Internal

2023
Internal Quality Perceptions Survey – Fiscal Year 2023, Quarter 4

MRF Data Summary Table – Fiscal Year (FY) 2022
MRF Data Summary Table – Fiscal Year (FY) 2021