Final Report – TVV8

E. Test Results: Billing Functional Usage Evaluation (TVV8)

1.0 Description

The Billing Functional Usage Evaluation examined Verizon Pennsylvania’s (Verizon PA) ability to capture customer telephone usage as data records. It also validated that the resulting data records were complete, accurate, and delivered in a timely manner to the KPMG Pseudo Competitive Local Exchange Carrier (CLEC). KPMG Consulting acted as a non-facilities based CLEC providing Resale and Unbundled Network Elements – Platform (UNE-P) services to business and residential customers. As part of its normal business process, Verizon PA captured information about each instance of network usage for the KPMG Consulting CLEC and delivered the resulting information to KPMG Consulting.

On each business day, Verizon PA collects call information from its switches. The switches generate events as calls are processed. Event information is recorded electronically. If any information is billable to a wholesale customer or if the information includes Inter-exchange Carrier (IXC) access records, then a file is created. This file, called the Daily Usage File (DUF), is delivered to CLECs using either cartridge tapes or an electronic transmission software package called the Network Data Mover (NDM).

The objectives of the Billing Functional Usage Evaluation test were to determine the completeness, accuracy, and timeliness of daily message processing by Verizon PA. A transaction-based test was conducted to complete this evaluation.

2.0 Methodology

This section summarizes the test methodology.

2.1 Business Process Description

Records appearing in the DUF come from two sources. First, when a CLEC customer completes a call on a line with a port leased from Verizon PA, the Verizon PA switch records call information on behalf of the CLEC. Second, if a CLEC leases an unbundled port, the switch will generate access records for InterLATA24 calls originating from and terminating to the unbundled port. Finally, when a CLEC customer completes a call on a resold line the Verizon PA switch records call information in the same manner as for Verizon PA retail customers. All of these call detail records are transmitted to Verizon PA’s data center for message processing.

24 InterLATA calls are calls where the originating and terminating exchanges reside in different Local Access Transport Areas (LATA). These are commonly known as “long distance calls.”

Final Report as of December 22, 2000 549 Published by KPMG Consulting – CONFIDENTIAL For Verizon Pennsylvania, Inc. and Commonwealth of Pennsylvania Public Utility Commission use only Final Report – TVV8 Verizon Pennsylvania

Non-billable records are discarded.25 Records determined to belong to a CLEC are translated into Exchange Message Interface (EMI) format and delivered to the CLEC on a daily basis over the transmission medium of the CLEC’s choice.

2.2 Scenarios

This transaction-based evaluation used scenarios representative of Resale and UNE products and services offered to business and residential customers in Pennsylvania. The scenarios represented available switch technologies ( DMS100 & TOPS, Lucent 5ESS), product and service types, and service change order types. Many of the scenarios involved a change, either in products or services, or in end user owning entity. Changes included conversions of account ownership, known as migrations, from one Local Exchange Carrier (LEC) to another, the addition of new lines, the addition of new accounts, feature changes, and/or class of service changes.

Once the scenarios were identified, the orders were scheduled and executed. The migration orders were submitted with a specific due date. Test calls were placed before and after the migration date to ensure accurate routing of DUF records.

KPMG Consulting also developed test cases emulating a variety of telephone calls typically made by business and residential customers. The test cases included local, intra-LATA toll and long distance calls, as well as operator assisted and network service call types.

Test scripts were created by combining test scenarios with test cases in a variety of permutations. In this manner, the test scripts applied real-world call types against representative customer accounts. Finally, testers executed the test scripts in the field.

2.3 Test Targets & Measures

The test targets and measures were carried forward from the Master Test Plan (MTP), Table VI-8 Scope of the Functional Usage Evaluation. The test target for the Billing Functional Usage Evaluation is the recording, assembly, and delivery of relevant usage data. Processes, sub- processes, evaluation measures, and associated test cross-reference numbers are summarized in Table 8-1 which follows. The last column, “Test Cross-Reference,” indicates where the particular measures are addressed in Section 3.1 “Results & Analysis.”

25 Switches will record both inbound and outbound calls. The Daily Usage File consists of outbound local usage, intra-LATA toll usage, Verizon operator-assisted calls, and IXC originating and terminating access records. Non-billable records generated by the switch include incoming local calls, non- measured outgoing local usage, and operator inquiries that may or may not be charged at the operator’s discretion. This list is non-exhaustive.

Final Report as of December 22, 2000 550 Published by KPMG Consulting – CONFIDENTIAL For Verizon Pennsylvania, Inc. and Commonwealth of Pennsylvania Public Utility Commission use only Final Report – TVV8 Verizon Pennsylvania

Table 8-1: Test Target Cross-Reference

Process Sub-Process Evaluation Measure Test Cross-Reference

Usage and Delivery Track valid usage Completeness and accuracy TVV-8-1-1 through of data TVV-8-1-5

Usage and Delivery Track valid usage Timeliness of DUF TVV-8-2-1 and TVV-8-2-2 Usage and Delivery Account for all usage Completeness of data TVV-8-3-1

2.4 Data Sources

The sources of data for this test included test calls placed by testers, test call logs, and DUFs received from Verizon PA, the Exchange Message Interface (EMI) standards manual V.17 released in April 2000, and reviews of documentation supplied by Verizon PA at KPMG Consulting’s request (see items listed below in Table 8-2).

The data collected for the test are summarized in Table 8-2 below.

Table 8-2: Data Sources for Billing Functional Usage Evaluation

Location in Document File Name Source Workpapers Verizon PA TIS www.bellatlantic.com/ Verizon Documentation Verizon PA Website for DUF wholesale/index.htm Tab Information CLEC and Resale www.bellatlantic.com/ Verizon Documentation Verizon PA Handbooks wholesale/index.htm Tab Yellow and White Hard Copy Verizon Documentation Verizon PA Pages for Each Calling Tab Region Exchange Message EMI17r1.pdf Verizon Documentation Alliance for Interface (EMI) version 17r1, April 2000 Tab Telecommunications Standards Manual Industry Solutions (ATIS) Terminating Numbers terminating_numbers_sorted Verizon Documentation Verizon PA List for Usage Testing .xls Tab Unbundled Call BA call scenarios rev 08-07- Verizon Documentation Verizon PA Scenarios for CLEC 2000.doc Tab Billing DUFs Transmitted to retestDUF062100.xls Verizon Documentation Verizon PA KPMG Consulting Tab CLEC

Final Report as of December 22, 2000 551 Published by KPMG Consulting – CONFIDENTIAL For Verizon Pennsylvania, Inc. and Commonwealth of Pennsylvania Public Utility Commission use only Final Report – TVV8 Verizon Pennsylvania

2.4.1 Data Generation/Volumes

KPMG Consulting developed test scripts that required testers to place calls in the Verizon PA calling region throughout the workday. Test calls included a variety of call types that were placed from Central Offices within the Verizon PA calling region. Table 8-3 identifies the locations used.

Table 8-3: TVV8 Test Calling Locations

Central Office Address

Bethlehem 525 North New St. Bethlehem City, Pennsylvania 18018 Homestead 400 Amity Street Homestead, Pennsylvania 15120 1631 Arch Street Philadelphia, Pennsylvania 19103 Pittsburgh 650 Smithfield Street Pittsburgh, Pennsylvania 15219

416 7th Avenue Pittsburgh, Pennsylvania 15219

Three additional test locations were used to place and accept long distance calls, third-party billing calls, and collect calls. One tester was located in Philadelphia, Pennsylvania, within the Verizon PA calling region, a second tester was located in Dallas, Texas, within the SBC Communications calling region, and a third tester was located in Denver, Colorado, within the Qwest calling region.

2.5 Evaluation Methods

Execution of the Billing Functional Usage Evaluation required Verizon PA to establish a test bed of accounts against which test calls were placed. The test calls consisted of commonly placed incoming and outgoing call types that were generated over multiple switch types. The test included validation of expected usage results based on test calls placed by KPMG Consulting against DUF records received by KPMG Consulting CLEC.

Evaluation of the accuracy and completeness of the DUFs was based on a comparison of the call details logged by KPMG Consulting when the test calls were placed, and the DUF records delivered to KPMG Consulting by Verizon PA.

Final Report as of December 22, 2000 552 Published by KPMG Consulting – CONFIDENTIAL For Verizon Pennsylvania, Inc. and Commonwealth of Pennsylvania Public Utility Commission use only Final Report – TVV8 Verizon Pennsylvania

2.6 Analysis Methods

The Billing Functional Usage Evaluation included a checklist of evaluation criteria developed by the Test Manager during the initial phase of the Verizon Pennsylvania OSS Evaluation. These evaluation criteria provided the framework of norms, standards, and guidelines for the Billing Functional Usage Evaluation.

Tester call logs were examined to determine whether that specific call should appear on the DUF. Calls not expected to appear on the DUF were evaluated to ensure that no DUF record was received. This included calls made from Retail accounts, Resale long distance calls, Resale toll free calls, incoming calls to Resale accounts, and local calls made from flat-rated Resale lines and other non-billable calls. For all test calls that should have appeared on the DUF, KPMG Consulting examined the DUF data to locate a valid record meeting the specifications of the call as it was recorded in the test call log.

DUF records were further examined to ensure DUF accuracy, as defined in the Carrier-to-Carrier Billing Performance Metric BI-4 DUF Accuracy. This measure captured the accuracy of the usage records transmitted from Verizon PA to KPMG Consulting on the DUF. The measure is derived by dividing the number of usage records delivered in the reporting period containing complete information and proper formatting by the total number of usage records delivered in the reporting period. DUF records were also verified to ensure that the records were received by the appropriate CLEC and to ensure that the records adhered to EMI guidelines. Results in Sections 3.1.1 and 3.1.2 of this report present findings for the completeness and accuracy of matched DUF records.

The DUFs received were examined to ensure that the pack header and trailer had an accurate count of DUF records transmitted in that DUF file pack.

DUF Timeliness, as defined in the Carrier-to-Carrier Billing Performance Metric BI-1 Timeliness of Daily Usage File, was measured by counting the number of business days between the creation of the message to the date the usage information is made available (transmission date) to the CLEC on the DUF. The day the call was made is excluded from this count. The records included in the tables below include all files that were received electronically in June 2000.

The data collected were analyzed employing the evaluation criteria referenced above.

Final Report as of December 22, 2000 553 Published by KPMG Consulting – CONFIDENTIAL For Verizon Pennsylvania, Inc. and Commonwealth of Pennsylvania Public Utility Commission use only Final Report – TVV8 Verizon Pennsylvania

3.0 Results Summary

This section identifies the evaluation criteria and test results.

3.1 Results & Analysis

The results of this test are presented in Table 8-4 below.

Table 8-4: TVV8 Evaluation Criteria and Results

Test Cross- Evaluation Criteria Result Comments Reference Usage Completeness and Accuracy: TVV-8-1-1 Test calls expected to Satisfied KPMG Consulting received 97% of create a DUF record did usage for telephone calls expected to result in a DUF record. generate usage. Of 625 test calls expected to generate usage, a usage record was generated for 606. The remaining 3% of telephone calls expected to generate usage included local, toll, and operator-handled calls. Detailed information appears in the “Test Call Log and DUF Record Accuracy Analysis” (Section 3.1.1) below. TVV-8-1-2 Test calls not expected to Satisfied KPMG Consulting did not receive usage create a DUF record did for 100% of the telephone calls not not result in a DUF expected to generate usage. There were record. 377 test calls not expected to generate usage. Detailed information appears in the “Test Call Log and DUF Record Accuracy Analysis” (Section 3.1.1) below. TVV-8-1-3 DUF records are received Satisfied For all DUFs received, KPMG by the correct CLEC. Consulting CLEC did not receive any unexpected messages. TVV-8-1-4 DUF record field values Satisfied Verizon PA produced DUF records in are in accordance with accordance with the EMI guidelines. EMI standard. TVV-8-1-5 Pack Header and Trailer Satisfied All Header and Trailer records contained records have accurate a correct count of the number of records counts of DUF records found within the pact. contained in the pack. Detailed information appears in the DUF “Header/Trailer Accuracy and Completeness Analysis” (Section 3.1.2) below.

Final Report as of December 22, 2000 554 Published by KPMG Consulting – CONFIDENTIAL For Verizon Pennsylvania, Inc. and Commonwealth of Pennsylvania Public Utility Commission use only Final Report – TVV8 Verizon Pennsylvania

Test Cross- Evaluation Criteria Result Comments Reference Usage Timeliness: TVV-8-2-1 Records appear on the Satisfied KPMG Consulting received 99.40% of its DUF in a timely manner. DUF records within four business days. The Carrier-to-Carrier Billing BI-1 DUF Timeliness metric states this metric will be satisfied if at least 95% of DUF records are received by the CLEC within four business days. Detailed information appears in the “DUF Timeliness Analysis” (Section 3.1.3) below. TVV-8-2-2 DUFs are received Satisfied KPMG Consulting received all expected according to the defined DUFs from Verizon PA according to the schedule. defined schedule.

File Completeness: TVV-8-3-1 DUF packs received Satisfied KPMG Consulting received 100% of all contain billable records. DUF packs received contained billable messages.

3.1.1 Test Call Log and DUF Record Accuracy Analysis

Table 8-5 reports expected results for test scripts from the June 2000 retest only.

Table 8-5: TVV8 Tester Log Entry Breakdown

Category Count

Total number of test scripts not expected to produce a DUF record 377 Total number of test scripts expected to produce a DUF record 625 Total number of test scripts 1002

Final Report as of December 22, 2000 555 Published by KPMG Consulting – CONFIDENTIAL For Verizon Pennsylvania, Inc. and Commonwealth of Pennsylvania Public Utility Commission use only Final Report – TVV8 Verizon Pennsylvania

Table 8-6: TVV8 Expected DUF Accuracy Analysis

Percentage Category Count of Total Total number of test scripts expected to produce a DUF record that 606 97% resulted in a matching DUF record Total number of test scripts expected to produce a DUF record that did not 19 3% result in a matching DUF record Total number of test scripts expected to produce a DUF record 625 100%

3.1.2 DUF Header/Trailer Accuracy and Completeness Analysis

Table 8-7: TVV8 Header/Trailer Accuracy and Completeness Summary

Total Number of DUFs Received Number of DUFs with Percentage with at least one billable record accurate record count 27 27 100%

The actual counts matched the count in the header and trailer for all DUFs received.

3.1.3 DUF Timeliness Analysis

Table 8-8 documents DUF record timeliness for all DUF records received.

Table 8-8: TVV8 DUF Timeliness Analysis

Cumulative Timeliness Count Percentage Total DUF records received 2827 100% DUF records received in 0 or 1 business day 1812 64.10% DUF records received within 2 business days 1859 65.76% DUF records received within 3 business days 2210 78.17% DUF records received within 4 business days 2810 99.40% DUF records received within 5 business days 2816 99.61% DUF records received within 6 business days 2816 99.61% DUF records received within 7 business days 2816 99.61% DUF records received within 8 business days 2825 99.93% DUF records received in more than 8 business days 2827 100%

Final Report as of December 22, 2000 556 Published by KPMG Consulting – CONFIDENTIAL For Verizon Pennsylvania, Inc. and Commonwealth of Pennsylvania Public Utility Commission use only Final Report – TVV8 Verizon Pennsylvania

3.1.4 Missing DUFs Analysis

There were no days during the testing that KPMG Consulting did not receive all expected DUFs.

Final Report as of December 22, 2000 557 Published by KPMG Consulting – CONFIDENTIAL For Verizon Pennsylvania, Inc. and Commonwealth of Pennsylvania Public Utility Commission use only Final Report – TVV8 Verizon Pennsylvania

This page is intentionally blank.

Final Report as of December 22, 2000 558 Published by KPMG Consulting – CONFIDENTIAL For Verizon Pennsylvania, Inc. and Commonwealth of Pennsylvania Public Utility Commission use only