Remote Debugging with Networked Bus Analyzers

Special ANALYZERS Remote debugging with networked bus analyzers

By Leif Erik Laerum

Remote debugging of VMEbus systems has in the past been unreliable and difficult. Now, new developments such as the inclusion of a network interface within VME bus analyzer technology have made remote analysis far easier and more reliable. This makes deploying test instruments into remote and inaccessible systems more achievable.

Many innovations have been made in the VMEbus market dur- Some network security departments will view connection to the ing the last two years. The so-called VMEbus “renaissance” has analyzer over the Internet as making the whole system available developed new standards with new protocols and infrastructure over the Internet. This is a misconception if configured correctly; such as VITA 41 VXS. Some are completely new and others offer however, it may be tough to win this battle, especially since mak- backwards compatibility with existing systems. In addition, the ing a direct connection to a bus analyzer over the Internet requires time to market and development cycle have continued to shrink, the IT group to configure routers to allow IP forwarding and allo- demanding the test and development tool vendors to innovate. cate a public IP address.

New developments within VME bus analyzer technology have Another easier method to connect over the Internet is that of uti- made remote analysis much easier and more reliable, which lizing a VPN tunnel. An example of this can be seen in Figure 1. makes deploying test instruments into remote and inaccessible The VPN tunnel provides a secure, encrypted connection between systems more achievable. Improvements in software, commu- two networks. This uses well-established means of communica- nication, and triggering make it easier for users to hone in on tions without jeopardizing security. There are numerous VPN trouble spots without being in the lab right next to the analyzer. protocols used today, and even small, inexpensive routers have support for one or more protocols. Establishing a remote connection To use a powerful bus analyzer tool while not actually being onsite opens up many possibilities for a system integrator or vendor. Placing a bus analyzer in systems delivered to the end customer in certain cases has proven to be a wise investment. This is especially true for field testing and trials of a new design or if the system is located in an area that is difficult to access. Shipping the bus analyzer to the location and installing it in a des- ignated analyzer slot provides the customer with faster response time. The service call may also be far less expensive. Often the end customer does not have the expertise on-site for debugging a problem. For a system integrator and hardware vendor, remote analysis means the engineering team will be able to troubleshoot the issue from any location.

The key element making remote debugging possible is the pres- ence of the Ethernet interface on the bus analyzer. The analyzer Figure 1 can be connected to the same network as the other boards in the system as well as PCs, workstations, and switches. The analyzer has a full TCP/IP stack, making it capable of handling DHCP, Terminal server technology is also a way to remotely connect to fixed IP, and other common TCP/IP settings. In a situation where a system. The analyzer GUI application will then be executed on the analyzer is to be used in a local area network, it is easiest to the remote computer. The remote users will log on to the remote use the IP address assigned by the DHCP server. computer from their office using the company’s existing security protocols. To access the analyzer with a WAN approach from the other side of a router, a fixed IP address is better since the address of the Another way to connect to the analyzer is via USB. USB lim- analyzer will not change. If using a fixed IP on a network with a its the connection length to five meters. However, USB provides DHCP server, the analyzer IP address should be reserved to avoid an extremely easy way of connecting a laptop to a bus analyzer conflicts. Remote debugging also includes situations where the already installed in the system. In the lab where network connec- lab is located in another building or floor where there are differ- tivity to the analyzer has a security impact, USB provides a fast ent subnets and networks. and efficient way of communicating with the analyzer.

Although our discussion focuses on the use of bus analyzers, a The analyzer’s power supply voltage monitor continuously reports remote connection to a system also allows remote software debug- the status of the voltages in the system. Alarm thresholds can be ging and upgrading as part of the debugging and repair process. set to report if the voltage is outside a specified range. Figure 2 demonstrates an example of the 5 V supply dropping below the Sharing the bus analyzer user-defined level of 4.80 V. Similarly, temperature measurements There are Web-based services available that allow remote users to can be obtained using a probe attached to the analyzer via a cable. see the screen of an analyzer at the local site. These services are Systems never have uniform temperature throughout so the tem- extremely easy to use and secure and will work on any network perature sensor (“Ext Temp” in Figure 2) can be placed anywhere with http Internet access. Users connected to the analyzer can use in the system for monitoring a particular critical area. The “Brd either USB or Ethernet. They then share the analyzer GUI and Temp” indicator is the temperature of the VMEbus analyzer itself. any other application windows to a website that the engineering team can see as well. Effectively, this will bring the entire engineering team to the remote site as well as any third party vendor. Engineers familiar with the system will recognize the source of the problem far quicker when they are able to see the screens, error messages, and live data.

Users can configure the analyzer setup with their settings. The Figure 2 GUI implements a workspace allowing each user to save settings and link all analyzer setups and saved data files. When the workspace is saved, it will open the same set of files next time. The user To control a system remotely, the most obvious method is to does not have to reconfigure the analyzer hardware from the previ- build and utilize software and network applications. However, ous user’s settings, thus enabling easy analyzer tool sharing. the VME analyzer can provide a redundant control unit that does not rely on a functional system. One example is the ability to Analysis can also be done offline once a trace file or performance reboot a malfunctioning system by asserting SYSRESET from log is saved. In situations where users have a specific time slot to an analyzer GUI. Using the bus analyzer to assert SYSRESET is debug their part of the system, offline trace viewing makes for identical to pushing the front panel reset button on the slot one more efficient use of the analyzer tool. Users collect trace, log, controller. The analyzer does not have to be in a system controller and performance data during their time slot and view the results slot to perform this function. back in the office. In a remote debugging situation, users can collect data from the remote site and bring the files back to the engi- The VMEbus analyzer also has I/O pins that can be toggled from neering team for analysis. The results can also be e-mailed from the GUI. These can be connected to other signals elsewhere in the remote site to the engineering headquarters or third party ven- the system. If a particular device in the system needs to be reset, dors for consultation. all that’s required is to run a wire from the analyzer header to the unit’s reset circuitry. These pins are controlled by the Exerciser Downloading data from the analyzer User Interface and can also be part of Exerciser scripts. Using Ethernet facilitates faster trace buffer download time from the analyzer to the computer than was previously available. Performance monitoring Downloading trace files and other data from the analyzer is not A bus analyzer also offers performance monitoring in a remote something that the user has to wait for like in the past. The ana- setting. This can be used as an early warning system of problems lyzer downloads data around the trigger point first. Thus, the user to come and hopefully reduce or prevent the cost of such a fail- can start viewing the result immediately, while the remainder of ure. The system architect can define real-time monitoring of cer- the trace data is downloaded in the background. Once the user has tain indicators. Statistics gathered and saved prior to deployment determined that the trace results are worth keeping, the tempo- can be used to compare to the deployed situation by replaying rary file can be saved with no download time. The networked bus saved results. analyzers offer user-scalable buffer sizes, reducing the download time even further. In most situations, the activity of interest is The most important VMEbus metrics are Bus Transfer Rate and found among the 1,000 samples before or after the trigger. Bus Utilization. Figure 3 shows a measurement taken in an example system. The utilization is measured to be 9.2 percent with System monitoring and control a max of 22.6 percent. Bus Utilization tells the user how much One of the great benefits to deploying a bus analyzer in the sys- bandwidth is used on the bus and how much more is available. tem is using it for system health and status monitoring purposes. During any given period of 100 samples, 9.2 samples were taken The analyzer’s external input ports can be connected to watchdog while there is activity on the bus, while 90.8 samples are taken timers and other sensor apparatus in the system. The analyzer can when the bus is idle. The bus is used only 9.2 percent of the time. be set up to trigger on these events. In a situation where there are error conditions that system software is trying to combat, an increase in bus utilization would be seen. Bus analyzers can be powered from an external PSU, enabling additional debugging such as during the boot process. The ana- Event-counting statistics are customized by the user to trace the lyzer is ready to sample before the system has been powered up. frequency of events such as writes to a certain address or within Using external power also eases the debugging in systems with an address range. Another example of event counting use is to marginal power supplies or where power is a suspected cause of track data values for a particular address range. the problems.

Reprinted from VMEbus Systems / October 2005 Copyright 2005 Figure 4 registers, check if a board is still alive or configured correctly, and Figure 3 move data. The Exerciser can also be used for remote debugging. Debugging techniques If a problem with a system is detected and the user is not sure The master function can be used to read and write registers, to what happened, the user should first capture a state trace while check status after a system has halted. The test function can be triggering on all “don’t cares.” If the analyzer does not trigger, used to run memory tests on a suspect region. The source or the system is locked up and there is no backplane traffic. In this destination of a DMA can be compared to determine if there is case, the user should switch to timing mode and again trigger on a pattern in why data is not being transferred completely. The all “don’t cares.” The timing display may reveal that there are Exerciser can also be used to write to a mailbox interrupt regis- backplane signals stuck in the asserted position (such as inter- ter, for example, to kick-start a system recovery process that may rupts). The trace display may then serve as input for setting up a otherwise not be available because of the system’s condition. trigger condition that can capture when this condition occurs. For Figure 6 shows an example of where a memory region is tested instance, the setup in Figure 4 will trigger if the analyzer detects twice using a walking 1 pattern. Then the Exerciser changes the that IRQ1 is asserted for 7.5 µs or longer. data contents of memory and, finally, a compare function is used.

The VMEbus analyzer also has VMEbus Protocol Violation monitoring. When enabled, it monitors for protocol violations such as signal glitches, timing errors, and voltage drops. The proto- Bus analysis for VITA 41 and VITA 46 col checker can be used as a trigger to the bus analyzer, where During the past five or six years, as innovations in the VMEbus market have the effect of the error can be observed in the form of incorrect mostly involved technology internally on VME boards containing a PCI bus, data and execution. In systems that have intermittent errors only FPGAs, and sideband I/O technology, the VMEbus analyzer has changed observed in the field, the deployed analyzer can be activated as a very little. The requirements did not change as much as they did for PCI and monitoring function. CompactPCI. With the VME renaissance, new protocols such as 2eSST, 2eVME, The voltage drop violation monitors the 5 V VCC supply just and others require innovation for bus analyzers. By using FPGA technology, like the previous voltage monitoring functions, with the differ- the bus analyzer can easily adapt to new protocols and standards. ence that it can trigger the analyzer if the threshold drops below 4.75 V. If power issues are suspected, this trigger can be used to However, new challenges lie ahead when it comes to test tools and analysis capture the bus activity around the voltage drop. for VXS (VITA 41). VME, VME64, and VXS are using the same P1 and P2 connector. The VMEbus analyzer technology offered today will be compatible Comparing to a good system Using the trace compare tool, bus activity collected in a good sys- with the VMEbus. However, the P0 connector in between P1 and P2 will host tem can be compared to the data collected after deployment. This a switch fabric interface such as PCI Express or RapidIO. Switched fabrics is particularly helpful in systems that have data integrity prob- consist of only endpoints and switches. This is point-to-point communication. lems or where software timing is in question. Figure 5 displays The traditional VMEbus analyzer scheme would result in the analyzer acting an example of where a data error is detected. The top screen is the as another endpoint on the fabric. It will not be able to see all communica- reference trace. The bottom screen shows a trace collected later. tion between other endpoints. Instead it must be placed between the switch The highlighted data pattern indicates a difference. and the DUT endpoint in the chassis. It will still not be able to see communi- Exerciser functionality cation between two other endpoints not in its path. In addition to a nonintrusive bus analysis tool, Exercisers offer additional debug and test functionality. The VME bus Exerciser VITA 46 utilizes a different connector and will, therefore, not be backward is a separate functional unit but operates from the same GUI. compatible with the VME, VME64, and VXS specifications. VITA 46 also con- It has four major components: tains the logical and electrical layer of the ANSI/VITA 1 – VME64 spec, but this is now optional. VITA 46 will make available a multitude of serial fabric ■ Master ■ Slave with memory switched protocols just like VXS. This makes reuse of existing analyzer tech- ■ Interrupt generation and acknowledgment nology challenging. Adapters and extenders may not provide technically ■ System controller functions sound solutions. Adapters will not solve the challenges caused by switched fabrics. The main purpose of an Exerciser is to provide functions not readily available from the SBC or I/O board. The user can read

Figure 5 Figure 6 The compare function fails because of the previously written data. to add a PMC analyzer to the SBC in the system slot in remote The Exerciser also has an onboard local memory that can be located debugging situations. anywhere in the VME address map. One specific remote debugging capability this memory can offer is as a system software log. Remote analysis The software can perform real-time write messages or bus marks Connecting to a VMEbus analyzer over Ethernet opens up many to certain locations in the memory. This status can be viewed by the possibilities for the engineering team to debug systems from a user locally from the GUI. The contents of the Exerciser memory remote site or further away from the system than was possible can be saved to a file for analysis or archiving. with previous bus analyzer technology. It also allows debugging from one central location where the engineering team can work This same memory can be loaded with user-specific data pat- together for faster problem resolutions. terns over the network. This data can, for example, be FPGA code or test patterns. The Exerciser DMA engine can move the Leif Erik Laerum has almost 15 years of experience in the data from the Exerciser memory to an address location in the embedded computer industry in test and troubleshooting tech- VMEbus system. niques. He has held positions within VMETRO, Inc. as test engineer, hardware manager, and VP of hardware products. The Exerciser has different script engines that allow the user to He currently serves as the vice president of quality assurance create automated canned routines to test, reprogram, or diagnose and customer support. He has an electrical engineering a system. degree from Kongsberg College of Engineering and a BS degree in electrical engineering from Texas A&M University. Cross-platform GUI Although our discussion referred to VMEbus technology, these To learn more, contact Leif at: topics are also valid for PCI, PMC, and CompactPCI systems. Nearly all the remote debugging capabilities of the bus analyzer VMETRO, Inc. can be leveraged in a similar manner on these types of platforms. 1880 Dairy Ashford, Ste. 400 There are networked bus analyzers available for these architec- Houston, TX 77077 tures as well. To allocate one PMC slot on the SBC for a bus Tel: 281-584-0728 analyzer will probably not provide enough system-level support Fax: 281-584-9034 for the user since the analyzer can only provide local status for E-mail: [email protected] this board. However, for some applications, it might be useful Website: www.vmetro.com