KOFAX
Communication Server
Knowledgebase article
ID: 00005239
FoIP Trace Guidelines
Version 1.00.00
Knowledgebase article ID: 00005239
Question / Problem The customer has problems with receiving or sending faxes over a Fax over IP (FoIP) network and you need to gather information to narrow down the problem and pass on traces to another support member for further analysis.
Identifying the Problem The most important step to solve a problem is to identify where the problem may be located. FoIP problems are most likely located inside the environment, either because the call routing is not correctly set up, T.38 support is not activated or there are problems with the fax communication itself.
The difficulty with call routing problems is that inbound calls are not traceable on the KCS-FoIP machine. This is because the calls never reach said machine. Only outbound calls are traceable and even they may result in unclear error messages or no errors at all, as routing problems can lead to various outcomes, e.g. call being routed to an unreachable destination, call being routed to the wrong destination etc.
Problems with T.38 support are relatively easy to identify but not so easy to locate. Typically, the Wireshark or TWS traces give a clear error message which is caused by a rejected request to switch to T.38 mode.
Fax communication problems are unique problems, most of the time, and rarely have the same cause as similar problems. These problems are difficult to trace and locate and a good portion of these are caused by software bugs either in the FoIP software or in the environment.
Preparation You need a text editor with advanced search capabilities. It should be able to list all lines matching a specific search string (Windows Notepad can only navigate to the next search result). If possible, the text editor should accept also 'regular expression' search terms. Additionally the text editor should be able to handle multiple files and search also in multiple Files (if possible also using regular expressions). UltraEdit is a good commercial product with advanced search capabilities.(http://www.ultraedit.com) PSPad (http://www.pspad.com/en) and Notepad++ (http://notepad-plus-plus.org/) are good free alternatives.
Collecting the traces There are basically 2 traces you can collect on the KCS-FoIP machine: TWS traces and Wireshark traces.
TWS TWS traces are the traces from the KCS-FoIP software itself. The name (TWS) derives from the underlying platform on which the KCS-FoIP runs. TWS traces by default are located in the directory: “C:\TOPCALL\FoIP\00\trace\” and the files itself are named in this format: “tws_xxx.trc” (xxx is used for file numbering). In a new installation you should have up to 10 files at any given time and each has a default maximum size of about 2 MB.
1.Trace Levels The default trace settings generate a dump of T.38 messages if the call fails. This means that many problems can be analysed with the default settings. If you decide to increase the traces levels you should check the other
FoIP Trace Guidelines Version 1.00.00 © 2012 Kofax, Inc. Page 2 of 16 Knowledgebase article ID: 00005239 trace file settings as well to prevent that traces are overwritten. Extra note: high trace settings may cause additional problems due to real-time constraints of FoIP. Trace settings are configured as follows: Start the configuration tool "Configure FoIP" from the "Kofax Communication Server" start menu group. Internet Explorer opens showing different sections, expand the section "Advanced". Here you can configure the TraceLevel, the MessageTraceSize (see below), the "Size of Trace file" (2000 or higher is recommended) and the "Number of Trace files" (10 or more is recommended). After changing any configuration values, KCS Foip must be restarted using KCS Monitor.
Recommended Trace settings:
TraceLevel MessageTraceSize Used for
basic trace information (that is often sufficient) without negatively 10 1 impacting performance. This is the default configuration.
Provides lot of information. More suitable when duplicating error situations 40 10000 in controlled environment.
Attention: Please do not increase the TraceLevel on your own initiative. Only increase it if you are instructed by a Kofax Support Technician as unnecessary high trace settings can make the complete trace useless.
2.Get an overview about all calls Use a text editor and then filter the traces for "bound Call:" to get a list of all inbound and outbound calls. Here is an example output:
… "03:UFI"} Inbound Call: CallerId=@55205, TSI=+4318635321 EC= (), Peer=172.20.148.52:30006 … "02:UFI"} Inbound Call: CallerId=@55205, TSI= EC=XL(I:HUMAN;55;21121;4-305), Peer=172.20.148.52:30010 … "03:UFI"} Inbound Call: CallerId=@55205, TSI=+4318635321 EC=ID(I:PROT;70;21120;4-305), Peer=172.20.148.52:30014 … "02:UFI"} Outbound Call: Number=I142 -> 142, CallerId=, CSI=, EC=JX(I:UNKN;45;12603;9-103), Peer=:0 … "02:UFI"} Outbound Call: Number=142 -> 0142, CallerId=, CSI=+4318635321, EC= (), Peer=172.20.148.52:30020 … "02:UFI"} Outbound Call: Number=142 -> 0142, CallerId=, CSI=+4318631, EC=XO(I:PROT;65;21313;15-305), Peer=172.20.148.52:3024 … "02:UFI"} Outbound Call: Number=142 -> 0142, CallerId=, CSI=, EC=ID(I:PROT;60;21322;4-305), Peer=172.20.148.52:30028 … "02:UFI"} Outbound Call: Number=142 -> 0142, CallerId=, CSI=+4318635321, EC= (), Peer=172.20.148.52:30032
In FoIP versions since KCS 9.1.1, the value "EC=" contains the 2-digit TCOSS error code as well as the extended FoIP error information as described in the FoIP manual. In older versions, only the FoIP error is present. “EC= ()” means success. Only the first three values in the brackets are important for error analysis.
Example of an error: EC=XL(I:HUMAN;55;21121;4-305) or EC= I:HUMAN;55;21121;4-305
As described above, the value before the brackets defines the 2-digit TCOSS error code. Values in the bracket are separated with semicolons. The first value (in this example "HUMAN") defines the error category – the list can be found in the FoIP Technical Manual, chapter "7.2.1 - Error Categories". The second value (in this example: "55") defines the connection level, i.e. in what state the call was aborted. –
FoIP Trace Guidelines Version 1.00.00 © 2012 Kofax, Inc. Page 3 of 16 Knowledgebase article ID: 00005239 definitions for this can be found in chapter "7.2.3 - Connection Level". The third value (in this example: "21121") defines the error code – the list can be found in chapter "7.2.2 - Error Codes".
3.Debug info’s of failed calls All failed calls generate additional debug info as shown in the example below. This debug info can be found directly for every "bound call:". In fact, the "bound call:" statement is the last entry in this debug info so you can always find it directly above.
"03:UFI"}
FoIP Trace Guidelines Version 1.00.00 © 2012 Kofax, Inc. Page 4 of 16 Knowledgebase article ID: 00005239
The content of
Element Description
Additional technical analysis The received/transmitted fax events are similar to the 'Wireshark Graph Analyses'. E.g. the reception of a HDLC frame is always indicated as a single fax event. Additional data may be appended after Data[length]= as hexadecimal digits. Note that these values are coded by using the most significant bit first, like Wireshark, and not by using the least significant bit first, like LS1 traces.
In case you need to investigate a problematic call and do not have Wireshark at hand, you can get, for example, the negotiated speed and error correction status out of the FoIP trace as well.
FoIP Trace Guidelines Version 1.00.00 © 2012 Kofax, Inc. Page 5 of 16 Knowledgebase article ID: 00005239
An example is shown below: KCS FoIP trace event:
"03:UFI"}
Wireshark output of the same fax event:
The detailed definition of DIS bits can be found in Table 2/T.30 (http://www.itu.int/rec/T-REC-T.30-200509-I/en).
FoIP Trace Guidelines Version 1.00.00 © 2012 Kofax, Inc. Page 6 of 16 Knowledgebase article ID: 00005239
In the case of the above DIS signal, we need to first figure out which of the hexadecimal coded bytes are interesting and contain the needed information. The above example was "ff c8 01 00 77 1f 22". The first 3 bytes (ff c8 01) are basically always the same and describe the element as a DIS signal. (The third byte - 01 - specifically describes it as a DIS signal, whereas a DCS would have "c1" in there).
Important is the part that follows (00 77 1f 22). Here the speed is located in the second byte (77). Note that the first bit in the first byte has bit number 1! 77 translates to these bits (bit number 9 to bit number 16):
0111 0111
The underlined part shows the bits important for the speed (bit 11, 12, 13 and 14). The screenshot below shows the definition of the 2nd DIS byte (0x77 in our example). So in this specific case the relevant bits are "1101" meaning that 7200 bit/s is used as transmission speed.
4.Using the trace context Since KCS 9.1.1 (and KIC 2.0) a trace context has been added to the common part of each trace line after the 'process-id/thread-id'. The trace context is used to find trace lines that belong to the same high-level activity, even if they are generated by different treads and processes. E.g. each inbound call with KCS FOIP gets a new trace context. An example with trace context “000d” (filtered with “/000d) “) is shown below.
22/20:08:01.348 (1ed0/0d38/000d) Dump-Req: Message 'Setup' (353 byte) from 1(sip) ---> 7(fx7) [s2-t2] 22/20:08:01.800 (1ed0/0d38/000d) Dump-Rsp: Message 'Connect' (245 byte) from 7(fx7) ---> 1(sip) [s2-t2] 22/20:08:01.910 (1ed0/0d38/000d) Dump-Req: Message 'ConnectVoice' (304 byte) from 1(sip) ---> 7(fx7) [s2-t2] 22/20:08:02.019 (1ed0/1bb0/000d) Dump-Req: Message 'ConnectRxVoice' (306 byte) from 7(fx7) ---> 0(T38) [s2-t3] 22/20:08:02.019 (1ed0/1bb0/000d) Dump-Rsp: Message 'RequestT38' (175 byte) from 0(T38) ---> 7(fx7) [s2-t3] 22/20:08:02.128 (1ed0/0d38/000d) Dump-Rsp: Message 'RequestT38' (175 byte) from 7(fx7) ---> 1(sip) [s2-t2] 22/20:08:02.206 (1ed0/0d38/000d) Dump-Req: Message 'ConnectT38' (295 byte) from 1(sip) ---> 7(fx7) [s2-t2] 22/20:08:02.237 (1ed0/1bb0/000d) Dump-Req: Message 'ConnectRxReq' (372 byte) from 7(fx7) ---> 0(T38) [s2-t3] 22/20:08:30.551 (1ed0/1bb0/000d) Dump-Rsp: Message 'ConnectRxConf' (200 byte) from 0(T38) ---> 7(fx7) [s2-t3] 22/20:08:38.835 (1ed0/1bb0/000d) Dump-Rsp: Message 'PageInd' (>4000 byte) from 0(T38) ---> 7(fx7) [s2-t3] 22/20:08:41.206 (1ed0/1bb0/000d) Dump-Req: Message 'PageResp' (193 byte) from 7(fx7) ---> 0(T38) [s2-t3] 22/20:08:43.796 (1ed0/1bb0/000d) Dump-Rsp: Message 'DisconnectInd' (>4000 byte) from 0(T38) ---> 7(fx7) [s2-t3] 22/20:08:43.827 (1ed0/0d38/000d) Dump-Rsp: Message 'Disconnect' (214 byte) from 7(fx7) ---> 1(sip) [s2-t2] 22/20:08:43.921 (1ed0/0d38/000d) Dump-Req: Message 'Disconnect' (>4000 byte) from 1(sip) ---> 7(fx7) [s2-t2] 22/20:08:43.921 (1ed0/0d38/000d) Connection has been disconnected by originator 1(sip). (State=3/0) 22/20:08:43.936 (1e18/1f84/000d) {1 "03:UFI"} Inbound Call: CallerId=@55205, TSI=+4318635321 EC= (), Peer=172.20.148.52:30006 22/20:08:43.936 (1ed0/1bb0/000d) Connection has been disconnected by originator 7(fx7). (State=3/0)
FoIP Trace Guidelines Version 1.00.00 © 2012 Kofax, Inc. Page 7 of 16 Knowledgebase article ID: 00005239
Wireshark Wireshark is a free-to-use network sniffer that can trace all network traffic from and to the system. It can be downloaded from http://www.wireshark.org/.
After installation and start-up, the screen should looks like this:
When you click on “Interface/Options”, you will see the following window:
FoIP Trace Guidelines Version 1.00.00 © 2012 Kofax, Inc. Page 8 of 16 Knowledgebase article ID: 00005239
Here you can select different things. Most importantly, you have to select the Network Interface Card (NIC) you want to capture the traffic off. Normally 'Wireshark capture files' capture every Ethernet frame going over the selected NIC. Here you have the possibility to specify capture filters that make it easier to only get the needed information as the network traffic can grow exponentially in larger networks. Here you also have the possibility to specify files where the trace should immediately be written to. In normal operation mode Wireshark only generates the trace in the memory of the machine not on the hard disk. This can lead to a crash of Wireshark when it can not address any more free memory. You can also adjust different display options to determine how the capture window should look during the capturing process. If you plan on doing a longer capture, probably over a few hours or even days, you should deselect every flag under “Display Options”.
Capture Filter To limit the amount of captured frames and to keep the file size of capture files low, it makes sense to specify capture filters. Before starting a Wireshark trace, you have to do a little research on the involved IP addresses during any given VoIP/FoIP call in the environment. This includes the FoIP machine itself of course, as well as every involved (or possibly involved) Gateway, Call Manager and IPPBX. You have to know the IP address of basically every participant in this call. Once you have this list, you can start composing a capture filter. Beware that the capture filter syntax is entirely different from the display filter syntax in the Wireshark main window. The most useful capture filter is a filter for IP addresses possibly involved in the call. You can white-list every participant of the call and only get the necessary information.
FoIP Trace Guidelines Version 1.00.00 © 2012 Kofax, Inc. Page 9 of 16 Knowledgebase article ID: 00005239
To do this, enter “host a.b.c.d [OR host e.f.g.h….]” in the capture filter, whereas every IP should be one from the list – excluding the own IP address of the server as this is part of every communicated frame.
Example: If your list reads
10.18.115.100 KCS-FoIP
172.20.240.100 Call Manager
172.20.240.8 Gateway
Then your filter would have to look like “host 172.20.240.100 OR host 172.20.240.8” to capture every communication between KCS-FoIP and Gateway AND KCS-FoIP and Call Manager.
Capture Files As mentioned, Wireshark captures network traffic until it runs out of memory. To prevent this you can specify a file (or multiple files) where Wireshark writes its trace output to. First, click on the “Browse” button and search for a location to store the file(s). Name them and do not forget to include an extension, normally “.pcap”, as Wireshark does not automatically add an extension if you forget it here. If you have done this, check the “Use multiple files” check-box. You then have additional options available. The most useful setting is to create a ring buffer which opens a new file after a specific amount of traces has been written. In our tests, an 11 page fax, containing 10 CCITT test pages (complex image content), leads to wireshark traces of about 1 MB in size.
Capturing process Click on “Start” to begin capturing. Now you can perform your tests/reproduction of the problem. Be sure to include the problem in the capture. If possible, it’s best to also capture a working call/scenario to see the difference here. Capture both incoming and outgoing calls and keep track of your performed tests and their results.
When you’ve finished your tests you need to stop capturing the network traffic. Click on “Stop the running live capture” to stop capturing the network traffic and then save the resulting trace with “File/Save As…” somewhere for future reference or to pass on to Kofax Technical Support.
FoIP Trace Guidelines Version 1.00.00 © 2012 Kofax, Inc. Page 10 of 16 Knowledgebase article ID: 00005239
Interpreting the tracing results When you’re finished with capturing the trace output, the Wireshark window should look like this:
Call Graph To get a clear overview of all the FoIP communication that took place during the capture, Click on “Telephony” - > “VOIP Calls”. This window shows you all the FoIP/VoIP calls inside the traced time frame:
Here you can see some general details about the call like the ‘Start and Stop Time’, Initial Speaker IP address, ‘FROM’ and ‘TO’ of the call, the used Protocol, amount of packets etc. Select the call where you suspect the problem or want to have a closer look at and click on the “Graph” button.
You will then get a window with the detailed overview of the communication between all participating parties.
FoIP Trace Guidelines Version 1.00.00 © 2012 Kofax, Inc. Page 11 of 16 Knowledgebase article ID: 00005239
The coloured columns represent all participating machines, each row represents one command. In this case, 10.18.115.101 is the FoIPv3 machine, 172.20.240.100 is a Cisco Call Manager 8 and 172.20.240.8 is a Cisco Gateway. As this is an outbound call, the 'setup command' is transmitted from our FoIP to the CCM, which then handles the call establishment. After the call has been established, the T.38 communication proceeds between the FoIP and the Gateway.
Here you can try to locate the cause of the problem.
RTP Player If you see a lot of RTP packets in the Graph coming from the other side without any switch to fax communication, it might be that the call got routed to a non-fax destination, for example a voice mail, or the provider plays their automated recording to inform you that the dialed number is not correct. If you would try such a call with a LineServer, you would have the possibility to create a Binary trace which allows you to hear what is happening on the line. It is not that simple with FoIP but possible nevertheless. First, locate the call in question via the VoIP Call list and the Graph. In nearly every call, you will find an RTP stream coming from the remote side to the KCS FoIP machine, select this RTP stream:
FoIP Trace Guidelines Version 1.00.00 © 2012 Kofax, Inc. Page 12 of 16 Knowledgebase article ID: 00005239
Now, in the main window of Wireshark, this specific RTP stream has been marked, the first packet of this stream to be exact. Here you can see the source and destination port of this stream:
FoIP Trace Guidelines Version 1.00.00 © 2012 Kofax, Inc. Page 13 of 16 Knowledgebase article ID: 00005239
With this information you can now go into “Telephony/RTP/Show all streams”. The following window will appear, which may contain some or many RTP streams.
Here you can locate the stream from before. Mark it and click “Analyse”, which leads to another window:
Here you can do 2 things. You can either save the audio data into a file and play it with an audio program of your choice or directly listen to the RTP stream inside Wireshark. The latter may not be possible because of missing sound cards in servers where FoIP is running.
To save the audio data to a file, click on “Save payload”, select a location for the file, be sure to select “.au” as format and save the file with the extension “.wav”. Leave the "channels" selection on "forward".
FoIP Trace Guidelines Version 1.00.00 © 2012 Kofax, Inc. Page 14 of 16 Knowledgebase article ID: 00005239
To hear to the stream directly, click on “Player” instead. The view here may vary on a per stream basis, but basically you click on “Decode”, select the channel you want to listen to and click on “Play”.
FoIP Trace Guidelines Version 1.00.00 © 2012 Kofax, Inc. Page 15 of 16 Knowledgebase article ID: 00005239
Special traces for CISCO devices
On a Cisco GW (connected directly or via a call manager) you also have the possibility to enable traces.
To connect to a CISCO Gateway you have to use a terminal program, and you connect either via a serial port, or via a telnet/ssh session over Ethernet.
When you connect to the GW, you have to log in.
To show the currently running configuration, type “sh run” (short for “show running-config”).
If you connected via telnet/ssh, you need to enable the terminal monitor with the command “term mon”. You are now able to see trace output in the current session. When connected via a serial cable, the output is visible without this command. Now you only have to enable the special traces you want.
As the trace or config output can be pretty extensive, it is suggested to configure your terminal program in such a way that the complete communication is written to a log file. The terminal program “Putty” has this ability. In this way you can analyse the traces more thoroughly afterwards.
These are commonly used debug commands.
Signaling problems debug isdn q931 (moderate amount of trace) debug voip dialpeer inout (moderate amount of trace) debug h225 asn1 (high amount of trace) debug h245 asn1 (high amount of trace) debug ccsip [messages|calls] (moderate to high amount of trace)
T38 communication problems debug fax relay t30 all-level-1 (small amount of trace)
MGCP problems debug mgcp packets all (moderate amount of trace)
Applies to All FoIP versions (Article based on FoIP 3.12.06) CISCO IOS version 12.4 or newer Wireshark 1.6.7 or newer
FoIP Trace Guidelines Version 1.00.00 © 2012 Kofax, Inc. Page 16 of 16