Pilot for United States Patent and Trademarks Office (USPTO) 30-60-90 Day Plan Technical Proposal

Pilot for United States Patent and Trademarks Office (USPTO) 30-60-90 Day Plan Technical Proposal Table of Contents

1 Executive Summary ...... 1 2 Technical Approach ...... 3 2.1 Assessment (Sprint 0)...... 3 2.2 Setup OpenCloudCX (Sprint 1) ...... 4 2.3 Integrate Pipeline with OpenCloudCX (Sprint 2) ...... 5 2.4 Productionize OpenCloudCX (Sprint 3) ...... 7 2.5 Enable Immune Deployment System (IDS) (Sprint 4) ...... 8 2.6 Agile Team Production use of OpenCloudCX (Sprint 5) ...... 9 2.7 Beyond the Pilot (Sprint n+) ...... 10 3 Management Approach ...... 11 3.1 The Pilot Team ...... 11 3.2 Metrics for Success ...... 12

1 Executive Summary Seven years ago, a Cloud First Policy was created to accelerate the pace of government adoption of Cloud capabilities. During this timeframe, many departments within agencies moved to the cloud under very little governance and direction, which led agencies and even departments to use multiple cloud providers. Many government agencies have been successful in implementing a DevSecOps environment and many departments within the agency use different cloud providers — such as Google, Amazon Web Services or Azure — but they are running into a multi-cloud deployment nightmare trying to manage deployments into multiple cloud providers. Deployment pipelines are siloed and not reusable across the enterprise, which burdens additional cost for the agency and limits the cross-department reuse of microservices, applications, and software efforts due to the “stickiness” services offered by each cloud provider. There are additional benefits to having multiple cloud providers, such as enhanced redundancy or business continuity planning are a few, but we’ve seen that most agencies have multiple clouds because of the organic use of a cloud provider that best fits their business case and mission. This strategy is best for the department but not for the overall Enterprise efforts for the Agency. In 2019, the Federal Cloud Computing Strategy, Cloud Smart, introduced a path forward for agencies to migrate to a safe and secure cloud infrastructure. This “BY UPDATING AN OUTDATED new strategy will support agencies to POLICY LOUD MART EMBRACES achieve additional savings, security, and , C S will deliver faster services. BEST PRACTICES FROM BOTH THE FEDERAL GOVERNMENT AND THE RIVA is agnostic to COTs, open source PRIVATE SECTOR, ENSURING and GOTs solutions, but we do evaluate and provide guidance to our customers AGENCIES HAVE CAPABILITY TO about the Pros and Cons for each LEVERAGE LEADING SOLUTIONS TO solution and develop a “best fit” strategy BETTER SERVE AGENCY MISSION, for each customer. DRIVE IMPROVED CITIZEN SERVICES With federal cloud adoption increasing AND INCREASE CYBER SECURITY.” dramatically in the last 5 years. Many — Suzette Kent, Federal Chief Information Officer agencies have implementation across multiple clouds with no centralized strategy in how to efficiently manage deployments to the cloud via an optimized and maintainable Continuous Delivery/Continuous Integration approach. OpenCloudCX aims to solve this problem. RIVA Solutions, LLC (RIVA) developed the OpenCloudCX Multi-Cloud solution based on an open source solution originally developed using Netflix’s Spinnaker CI/CD solution. Furthermore, we have integrated Hygieia, which is two self-contained dashboards – one for engineers and another for executives – that visually depict CI/CD pipelines. We are able to provide senior leadership the ability to understand the DevSecOps state of maturity and risk across a wide range of product portfolios. OpenCloudCX is an open source, multi-cloud continuous delivery platform for releasing software changes with

Pilot for United States Patent and Trademarks Office (USPTO) 30-60-90 Day Plan Technical Proposal high velocity and confidence. Furthermore, because the key building blocks of our solution are based on Open Source technology, there is no worry of vendor lock-in that exists with proprietary software solutions. What differentiates OpenCloudCX from other Multi-Cloud solutions is how OpenCloudCX abstracts the cloud operations from its users by allowing users to create custom views of the cloud. Providing a custom view of the cloud enforces best practices that help development teams draw from the lessons codified in OpenCloudCX. This enable agencies with multiple cloud providers the ability to seamlessly integrate OpenCloudCX into their existing CI/CD pipelines and abstract out the cloud provider operations and allow agencies to adopt best practices by incorporating security, reuse existing deployments, and standards cloud deployments for the Enterprise. RIVA has invested in this solution for clients, extending Spinnaker to integrate it with our existing open source DevSecOps platform to provide a fast, repeatable, open source multi-cloud deployment capability. A core differentiating component of OpenCloudCX is the deep learning models that determine the normal state of deployments and provides anomaly alerts when development teams deviate from a successful deployment in our Immune Deployment System (IDS) which uses unsupervised and supervised machine learning and AI to understand all about your multi-cloud deployments. RIVA is an 8(a) small disadvantaged business (SDB) (graduation date January 18, 2026) has a long and successful management history supporting USPTO operations, including five current contracts supporting the Office of the Chief Information on Officer (OCIO), Office of the Chief Financial Officer (OCFO), and Office of Human Resources (OHR) for deep mission understanding and best practice sharing. We bring a deep understanding of the of the USPTO environment, culture, and mission in addition to valuable experience on multiple USPTO contracts providing similar services. We offer the USPTO a transparent, reliable team that focuses on customer delivery and quality services. RIVA is the right company to start small and scale to win!

Pilot for United States Patent and Trademarks Office (USPTO) 30-60-90 Day Plan Technical Proposal 2 Technical Approach RIVA’s experience with OpenCloudCX implementations has taught us that each organization’s journey is unique, and complex. Success relies on understanding the current people, process, and technologies. We start with an understanding our USPTO’s executive vision and benefits a multi-cloud continuous deployment capability will bring to them. We will decompose that vision into prioritized requirements as a Roadmap. The Roadmap will detail which applications and agile teams will transition to OpenCloudCX. We define these requirements as user stories and ensure that they are refined into actionable tasks that rely on USPTO’s DevSecOps and Cloud Maturity. We use the requirements to get from “as is” to the “to be” state. Below shows a model for our approach, as we transition from an assessment to execution, incrementally growing the “body of knowledge,” buy-in, and maturing USPTO’s multi-cloud governance model. RIVA follows Agile Scrum to integrate OpenCloudCX in order to respond quickly and accurately to migrate USPTO to a multi-cloud continuous delivery platform.

Figure 1-RIVA’s OpenCloudCX Engagement Process for 30/60/90 day approach at USPTO .

2.1 Assessment (Sprint 0) The assessment phase is our first 2-week Sprint, Sprint 0, where we determine the lay of the land. We determine which applications are a good fit to migrate to OpenCloudCX and develop a Roadmap that provides USPTO an artifact that can be used to understand where we are and

Pilot for United States Patent and Trademarks Office (USPTO) 30-60-90 Day Plan Technical Proposal where we are going with the migration. A Roadmap has diminished value if you don’t know your initial state. We use a layered maturity model that measures three simple objectives – do you deliver quality, stability, and do it in a timely manner? Our maturity model uses industry standard metrics and all SLAs for each contract for development (e.g. code coverage) and operations (e.g. mean time to recovery) but puts them in the context of governance (do you have control over the discipline?) and impact (how important is improving this area to your mission?). These measures reveal obstacles to adoption and provide measurable outcomes. They suggest areas within disciplines to prioritize. The assessment of maturity also explores constraints. We assess maturity through a lens of experience - understanding that isolated Teams, diverge from each other in practice. All these factors define a baseline maturity from which we will develop, with USPTO’s leadership, a full Roadmap. At the end of the assessment phase we will have the following deliverables.

Deliverable Benefits to USPTO Detailed Roadmap that outlines each agile team and USPTO will have a clear vision of the “end state” of application in priority order to migrate to the the migration and path to success for a multi-cloud OpenCloudCX platform. deployment. Documented EPICs in JIRA that detail the activities USPTO has full transparency into the progress being required to migrate to the OpenCloudCX platform. made with the migration. 2.2 Setup OpenCloudCX (Sprint 1) Our next two-week Sprint, Sprint 1, is focused on installing and configuring the base architecture shown in Figure 2. RIVA works with USPTO to determine which Cloud Providers are part of the Pilot. We have SME’s ready for this project that are experts in Microsoft Azure, Google Cloud and Amazon Web Services (AWS) cloud offerings. Our cloud experts will setup accounts for USPTO in cases where none exist and gain access to cloud environments where they do exist today.

Figure 2 OpenCloudCX Target Architecture and integration with USPTO’s Cloud and DevSecOps Pipelines

Pilot for United States Patent and Trademarks Office (USPTO) 30-60-90 Day Plan Technical Proposal We understand that USPTO is utilizing AWS and we will make sure that we setup our environments, so they do not clash with existing Agile Teams progress. There are a number of ways we can accomplish this. For example, we can setup a separate AWS Organization for the Pilot, so we are completely firewalled from viewing or having access to the AWS resources that the Agile Teams are using today that are not a part of the Pilot. Based on USPTO’s preference we can install OpenCloudCX on-premise, in AWS, Azure, Google Cloud or other preferred cloud offering. Once OpenCloudCX is installed then we need to integrate OpenCloudCX with the selected Agile Teams existing CI/CD pipeline through webhooks which are shown in Figure 2. In the event a CI/CD tool does not contain webhook integrations then RIVA can develop a web collector, if needed, or we can exclude it’s input for the Pilot. Typically, a web collector that interfaces with a REST API only takes a few days to implement and test. At the end of Sprint 2 we will have developed the following deliverables.

Deliverable Benefits to USPTO OpenCloudCX installed on USPTO preferred cloud or The feasibility study of OpenCloudCX can start to on-premise. begin. USPTO now can have hands on experience working with OpenCloudCX. Completed User Stories in JIRA with acceptance criteria Complete transparency into the progress and the signed off by USPTO. knowledge transfer of how OpenCloudCX has begun. Architect Documentation, Architecture Data Flow, Data RIVA adheres to USPTO standards and policies to Management that provides all required architecture views streamline the Pilot into a Production System to scale for success. .tar – A backup of the installed USPTO will have an easy way to reproduce/redeploy OpenCloudCX OpenCloudCX configured install

2.3 Integrate Pipeline with OpenCloudCX (Sprint 2) Now we have a repeatable installation for USPTO’s environment we need to fully integrate the pipeline with OpenCloudCX. One of the first steps we will do is configure the image bakery. Since the concept of Immutable Infrastructure is core to OpenCloudCX, we provide an image bakery to help USPTO produce machine images. We work with USPTO to start customizing the bake process or use artifacts not supported by the default configuration. OpenCloudCX has multiple options for both authentication and authorization. Instead of reinventing yet-another- login system, OpenCloudCX hooks into a login system USPTO already has, such as OAuth 2.0, SAML, or LDAP. For authorization, OpenCloudCX similarly leverages a role-provider that USPTO has already have set up, including Google Groups, GitHub Teams, SAML Roles, or LDAP groups. OpenCloudCX can be configured to listen to changes to a repository in GitHub. We configure webhook push events to send to OpenCloudCX from a single GitHub repository. We setup and define the infrastructure stages that operate on the underlying cloud infrastructure by creating, updating, or deleting resources. We define these stages for every cloud provider that we have select for the pilot. These steps are critical to leveraging multiple clouds, as it allows us to deploy to each cloud provider in a consistent way, reducing cognitive load for engineers. Examples of stages of this category include: . Bake (create an AMI or Docker image)

Pilot for United States Patent and Trademarks Office (USPTO) 30-60-90 Day Plan Technical Proposal . Tag Image . Find Image/Container from a Cluster/Tag . Deploy . Disable/Enable/Resize/Shrink/Clone/Rollback a Cluster/Server Group . Run Job (run a container in Kubernetes)

The rest of the infrastructure stages operate on USPTO’s clusters/server groups in some way. These stages do the bulk of the work in the deployment pipelines. OpenCloudCX provides integrations with custom systems to chain together logic performed on systems other than OpenCloudCX. OpenCloudCX can interact with Continuous Integration (CI) systems such as Jenkins, Google Cloud Build, Travis CI or Wercker. These CI tools are used for running custom scripts and tests. By providing webhooks at this stage allows existing functionality that is already built into Jenkins or other CI tools to be reused when migrating to OpenCloudCX. One possible integrated pipeline would like look Figure 3.

Figure 3 Example OpenCloudCX Integrated Pipeline

The final step in Sprint 2 is configuring the Dashboards. Dashboard enables senior leadership at USPTO to understand the DevSecOps state of maturity and risk across a wide range of product portfolios. This macro view of product portfolios allows USPTO to leverage a set precise goals for improvements on DevSecOps maturity, risk, and maintaining consistency across the portfolio. The dashboard, example Figure 4, provides insights into areas such as:

. Key Performance Indicators (KPIs) for quality aggregated across team instances . Enterprise-wide quality standards to measure SDLC maturity risk and compliance . Incidence visibility for service at various severity levels in production . Alerts for product standards across products in a portfolio . Drives usage of standard DevSecOps toolsets

Pilot for United States Patent and Trademarks Office (USPTO) 30-60-90 Day Plan Technical Proposal

Figure 4 Example Dashboard show complete visibility into the pipeline The deliverables that will be provided at the end of Sprint 2 are as follows: Deliverable Benefits to USPTO Fully Integrated OpenCloudCX installed on USPTO The feasibility study of OpenCloudCX can start to preferred cloud or on-premise. begin. USPTO now can have hands on experience working with OpenCloudCX. Completed User Stories in JIRA with acceptance criteria Complete transparency into the progress and the signed off by USPTO. knowledge transfer of how OpenCloudCX has begun. Updated Architect Documentation, Architecture Data RIVA adheres to USPTO standards and policies to Flow, Data Management that provides all required streamline the Pilot into a Production System to scale architecture views for success. Updated .tar – A backup of the installed USPTO will have an easy way to reproduce/redeploy OpenCloudCX with complete integration OpenCloudCX configured install

2.4 Productionize OpenCloudCX (Sprint 3) Now that we are integrated with USPTO’s DevSecOps pipeline and Cloud Provider we can start to passively collect metrics on the DevSecOps pipeline and start to tune OpenCloudCX and turn on the Immune Deployment System (IDS) so our predictive AI/ML deep learning components can start learning what constitutes a deployment success and failure. OpenCloudCX is a large system, made of many microservices, each intended to be scaled, restarted, and configured independently. This provides operators a great degree of flexibility and allows OpenCloudCX to handle massive scale (1K+ deployments/day, 10K+ managed machines). However, there is no one-size-fits-all approach for configuring OpenCloudCX; USPTO’s usage patterns will need to inform how to prepare the OpenCloudCX deployment to be used in production.

Pilot for United States Patent and Trademarks Office (USPTO) 30-60-90 Day Plan Technical Proposal We have many configuration options, whether it is to make OpenCloudCX more responsive, or reduce the load on a downstream dependency. If we notice slowdowns in some of OpenCloudCX’s operations, such as creating applications, running pipelines, submitting ad-hoc operations (clone, resize, rollback) then we tune the system either by adding more nodes to make OpenCloudCX more responsive. The deliverables that will be provided at the end of Sprint 3 are as follows: Deliverable Benefits to USPTO Fully Integrated OpenCloudCX installed on USPTO The feasibility study of OpenCloudCX can start to preferred cloud or on-premise. begin. USPTO now can have hands on experience working with OpenCloudCX. Completed User Stories in JIRA with acceptance criteria Complete transparency into the progress and the signed off by USPTO. knowledge transfer of how OpenCloudCX has begun. Updated Architect Documentation, Architecture Data RIVA adheres to USPTO standards and policies to Flow, Data Management that provides all required streamline the Pilot into a Production System to scale architecture views for success. Updated .tar – A backup of the installed USPTO will have an easy way to reproduce/redeploy OpenCloudCX with complete integration OpenCloudCX configured install IDS Configuration File – Tuned Config file for reference USPTO will have transparency into the tuning and backup. parameters for IDS and have a deep understanding of how we have weighted the Anomaly Ranking for the selected Agile Team.

2.5 Enable Immune Deployment System (IDS) (Sprint 4) Now that we have tuned IDS it’s time to turn it on and run in parallel with an Agile Team for a development Sprint to start seeing the Anomalies and filtering out the anomalies that are not issues. We perform this Sprint, so development teams are not bogged down with alerts and our models need time to develop. Employing multiple unsupervised, supervised, and deep learning techniques in a Bayesian framework, the Immune Deployment System can integrate a vast number of weak indicators of anomalous behavior to produce a single clear measure of threat probabilities. At its core, OpenCloudCX mathematically characterizes what constitutes ‘normal’ behavior, based on the analysis of many different measures of a continuous deployment platform, including: . Complexity (User Stories, Cognitive, Cyclomatic) . Duplications (Blocks, Files, Lines) . Timings of events . Issues (Opened, Confirmed, Reopened) . Maintainability (Code Smells, Maintainability Rating, Technical Debt, Technical Debt Ratio) . Quality Gates (Status, Details) . Reliability (Bugs, Rating, Remediation Effort) . Security (Vulnerabilities, Rating, Remediation Effort) . Size (Classes, Files, Directories, Lines of Code, Programming Languages) . Tests (Conditional Coverage, Line Coverage, Unit Tests)

The deliverables that will be provided at the end of Sprint 4 are as follows: Deliverable Benefits to USPTO

Pilot for United States Patent and Trademarks Office (USPTO) 30-60-90 Day Plan Technical Proposal

Fully Integrated OpenCloudCX installed on USPTO The feasibility study of OpenCloudCX can start to preferred cloud or on-premise. begin. USPTO now can have hands on experience working with OpenCloudCX. Completed User Stories in JIRA with acceptance criteria Complete transparency into the progress and the signed off by USPTO. knowledge transfer of how OpenCloudCX has begun. Updated Architect Documentation, Architecture Data RIVA adheres to USPTO standards and policies to Flow, Data Management that provides all required streamline the Pilot into a Production System to scale architecture views for success. Updated .tar – A backup of the installed USPTO will have an easy way to reproduce/redeploy OpenCloudCX with complete integration OpenCloudCX configured install

2.6 Agile Team Production use of OpenCloudCX (Sprint 5) Sprint 5 is our Pilot’s last sprint and our time to win! During this Sprint the Agile Development team will use OpenCloudCX for a production Sprint. This Agile team will be the first of many who will support the overall organization needs to support cloud agnostic CD. This means that the business, development and operations have started to align their practices to support continuous delivery. During this Sprint we introduce the concept of a canary release. A canary release is a technique to reduce the risk from deploying a new version of software into production. A new version of the software, referred to as the canary, is deployed to a small subset of users alongside the stable running version. Traffic is split between these two versions such that a portion of incoming requests is diverted to the canary. This approach can quickly uncover any problems with the new version without impacting the majority of users. The quality of the canary version is assessed by comparing key metrics that describe the behavior of the old and new versions. If there is a significant degradation in these metrics, the canary is aborted, and all of the traffic is routed to the stable version in an effort to minimize the impact of unexpected behavior. A canary release should not be used to replace testing methodologies such as unit or integration tests. The purpose of a canary is to minimize the risk of unexpected behavior that may occur under operational load. Using OpenCloudCX’s built-in capabilities, we will have set up a canary deployment which gradually routes requests to the new version of the application, and also includes a stage that performs smoke tests for good measure. Throughout these various pipelines, OpenCloudCX’s infrastructure view provides visibility regarding what did and did not happen along the pipelines, what tests passed or failed and what versions were deployed. Having a CD orchestration tool that provides an applications-on infrastructure view makes supporting the running application that much easier for operations, giving a level of insight and control that creates push-button deployments across environments. Although scripting pipelines is possible for more complex operations, the templates in the OpenCloudCX UI make stringing together pipeline stages fairly easy, with most tools and options needed for robust deployments. In our experience, Pipelines and its concept of pipeline templates are one of the areas where OpenCloudCX really shines. Creating custom pipeline

Pilot for United States Patent and Trademarks Office (USPTO) 30-60-90 Day Plan Technical Proposal templates is an easy way to create reusable pipeline modules that can be applied in varying situations. We have found this to be a very useful way to give each team control over their pipelines but ensure that individual components are only built once and built correctly. The templates make reuse natural, and not a chore requiring best practices or lengthy documentation, which custom code templates might require. Our Agile Project Manager, David Callner will summarize the Pilot in a comprehensive report that will include lessons learned, success and benefit summary and a recommendation for a path forward to scale to win. The deliverables that will be provided at the end of Sprint 5 are as follows: Deliverable Benefits to USPTO Fully Integrated OpenCloudCX installed on USPTO The feasibility study of OpenCloudCX can start to preferred cloud or on-premise. begin. USPTO now can have hands on experience working with OpenCloudCX. Completed User Stories in JIRA with acceptance criteria Complete transparency into the progress and the signed off by USPTO. knowledge transfer of how OpenCloudCX has begun. Updated Architect Documentation, Architecture Data RIVA adheres to USPTO standards and policies to Flow, Data Management that provides all required streamline the Pilot into a Production System to scale architecture views for success. Updated .tar – A backup of the installed USPTO will have an easy way to reproduce/redeploy OpenCloudCX with complete integration OpenCloudCX configured install A Comprehensive Report of the Pilot and a detailed USPTO will be in a position to scale OpenCloudCX Roadmap for how to scale to win with the other Agile to all the Agile Teams and start receiving Teams. predictability into their continuous delivery pipeline along with being cloud agnostic. 2.7 Beyond the Pilot (Sprint n+) Over the years, we have seen several common themes emerge that are key to a successful CI/CD implementation journey: Focus on Releasing Code and Not Building Code: It is important that everyone sees quality and speed of delivery as the most important aspect of any application. Fund as a Product and Not a Project: Too many agencies are still funding and building budgets around projects. Each project can have many features, and sometimes these budgets won’t allow features to be released independently. To ensure that releases are given priority, funds need to flow into a product as a whole. Establish a Culture of Automation: USPTO should support a culture of automation. This involves looking at new automation technologies, but also avoiding any infrastructure or application code that cannot be automated. This will kill the automation of your pipelines. You Build it You Own it: Creating ownership with the team that built the code is paramount in our experience. Every piece of code — microservice, pipeline and infrastructure — should have a clear owner, and we find it best not to create separations between the teams that manage and the teams that build the code. These types of separations typically end up creating handoffs that halt or impede the speed and automation CD pipelines are inherently designed for.

Pilot for United States Patent and Trademarks Office (USPTO) 30-60-90 Day Plan Technical Proposal 3 Management Approach We establish and build upon a true partnership between USPTO, and RIVA based on trust, transparency and direct and open communications. We engage with stakeholders at all levels regularly throughout the Pilot life cycle to understand and integrate OpenCloudCX. Through our Agile SCRUM process, we work with stakeholders to create value-driven stories that accurately reflect requirements and desired state. We engage with stakeholders to drive alignment between program and USPTO’s mission and objectives. Our Roadmap and JIRA Backlog helps in executing the program tasks, tracking, and reporting of the deliverables within the timeframe. We use a matured and robust framework of best practices for project management, from providing deliverables from initiation (Kick-off Meeting) to project related documentation and reporting (Closeout). RIVA’s practical, detailed, and high-quality work plans start with documentation of project objectives, milestones, constraints, and acceptance criteria documented in JIRA following the Sprint activities discussed in the Technical Approach. Using this information, we detail the activities, activity sequences, release timelines, and deliverables required to complete tasks. Activities are broken into multiple tasks that include internal/external dependencies, risks, issues, and deliverable dates. We detail Sprint tasks to include incremental deliverables, testing, and dependencies allowing visibility into sprint execution, and leverage existing JIRA tools to plan, build, and execute development work backlogs, giving transparency for integration tasks. 3.1 The Pilot Team Our team includes our Agile Project Manager (APM), David Callner, and two Cloud Architects, and a Data Architect from our OpenCloudCX delivery team whom all have extensive experience implementing OpenCloudCX for various customers. We include a deep bench of SMEs that understand have knowledge and experience of each open source product that we use in OpenCloudCX. In fact, we cross train our SMEs to work with OpenCloudCX in each of the various Cloud Providers so our team is well prepared for any issues that might arise.

Figure 5 RIVA's Pilot Team has RIVA's deep bench of SMEs

Pilot for United States Patent and Trademarks Office (USPTO) 30-60-90 Day Plan Technical Proposal 3.2 Metrics for Success Leveraging release metrics and lessons learned, feedback from stakeholders, performance metrics, and monthly processes maturity assessments, RIVA identifies, prioritizes, and implements process improvements. The table below provides performance metrics that are used to monitor and control quality of the Pilot delivery. Discrepant performance or negative trends are investigated, and process improvements are implemented. RIVA staff are incentivized to find areas to improve efficiencies; allowing process improvements to be identified by all staff members at any level. Below present performance measures to evaluate OpenCloudCX Pilot project execution.

Service Category Performance Minimum Target Measurement Frequency of Accountability / Service Measure Perf. Perf. Tool Reporting Standard Standard

Roadmap and % of time >=90% 100% JIRA, PPT, End of Sprint APM JIRA Product deliverables Word or Visio Backlog Creation complete, accurate, on time, and meet standards

Installation of % of defects found <= 3% <= 1% JIRA and Daily/Weekly OpenCloudCX OpenCloudCX in Confluence Architect USPTO environment Cloud Providers % schedule 100% 100% Visual End of Sprint Cloud Architect accounts created delivery of projects Inspection Program % of time reports, >=90% >=100% Email or Weekly/ APM Documenting and documents, and SharePoint Monthly Reporting deliverables Portal complete, accurate, on time, and meet standards

Program % schedule >=90% >=100% JIRA and Weekly APM Schedule delivery of projects Confluence

Program % of time Cost >=90% >=100% CostPoint Monthly APM Financial Performance Index is =< 1

Program % of timely and >=90% >=100% CostPoint Monthly Financial accurate invoicing

Adherence to % of time within >= 90% >= 100% JIRA and Daily/Weekly APM Sprint Schedule Sprint Schedule Confluence during TO

Post-production % of defects found <= 3% <= 1% JIRA and Daily/Weekly APM defects Confluence