Traces are the most powerful tools for debugging telephony problems. They provide a definitive answer to the question "What happened during this call?"

When taking traces, try to find the simplest scenario that reproduces the problem. For example, if your problem is unrelated to call recording, turn off call recording during the trace. This will make traces much easier to follow. In particular, hunt groups and queues can add hundreds of extraneous lines to a trace, so please avoid using these unless the problem is related to them.

Enswitch web      Asterisk console      Asterisk full log      ngrep      Wireshark

Enswitch web

The easiest and most useful type of trace is using the Reports » traces menu on the Enswitch web interface. This shows call routing decisions. This method works by automatically running an Asterisk console trace for you with commonly used options, then storing the trace taken in Enswitch's file store for later display on the web or for download. Enswitch 3.7 or later is required to use this method.

1. Log in to the Enswitch web interface.
2. Go to Reports » Traces.
3. Click the "New" button.
4. Select which Asterisk machines to run the trace on. Typically, this will be all machines in the cluster.
5. Click the "Start" button.
6. Make a failing call. If at all possible, avoid having other calls active at the same time as they make the trace larger.
7. Click the "Stop" button, then return to the list of traces.
8. Click on the trace you've just taken.
9. Click on the failing call, or download the whole trace.
10. If collecting the trace for a ticket, inform Enswitch support on the ticket where they can find the call, or upload the trace as an attachment to the ticket.

Back to top

Asterisk console

Sometimes more detailed call flow information is required than produced by the Enswitch web. In this case, an Asterisk console trace can be taken manually. This is usually only necessary if Enswitch support asks you to do this.

1. SSH into the Asterisk machine or machines. For clusters with multiple Asterisk machines, it's usually best to trace all at the same time as it's uncertain which machine will handle the call.
2. Load the Asterisk console using the "asterisk -r" command.
3. Set Asterisk to report events to the console by typing:
   • core set verbose 9
   • core set debug 9
   • agi set debug on
   • sip set debug on
   or as advised by Enswitch support.
4. Make a failing call. If at all possible, avoid having other calls active at the same time as they will confuse the trace.
5. Copy and paste the console output into a file.
6. Exit the Asterisk console, optionally first turning off verbose and the debugs.
7. If collecting the trace for a ticket, upload the file as an attachment to the ticket.

Back to top

Asterisk full log

On busy systems, it's sometimes not possible to take a console trace as there are many other calls. In this case, a backup plan is to record the data to /var/log/asterisk/full instead.

1. SSH into the Asterisk machine or machines.
2. Edit /etc/asterisk/logger.conf, and uncomment the "full" line.
3. Load the Asterisk console using the "asterisk -r" command.
4. Run "logger reload" to make Asterisk re-read logger.conf.
5. Set Asterisk to report events as in Asterisk console traces above. In addition, "core set debug 9" is useful for logging detailed information to /var/log/asterisk/full.
6. Run "tail -f /var/log/asterisk/full" for a few seconds and check that new lines are being added. Hit ctrl-C to stop.
7. Make a failing call. Note the exact times of important events such as the call connecting. If at all possible, avoid having other calls active at the same time as they will confuse the trace.
8. Find the call in /var/log/asterisk/full based on exact call time and calling/called numbers.
9. Either copy and paste the appropriate lines into a file to upload to a ticket, or let support know exactly where to find the call in /var/log/asterisk/full.
10. Optionally, edit /etc/asterisk/logger.conf, and re-comment the "full" line.
11. If you did the above line, run "asterisk -rx 'logger reload'" to make Asterisk re-read logger.conf.

Back to top

ngrep

For SIP problems with a specific handset, an ngrep trace can be useful. It shows all SIP traffic to and from the handset.

1. SSH into the OpenSIPS/OpenSER machine. If using a pair of machines with heartbeat, it's only necessary to do this on the currently active machine with the service IP address.
2. Run ngrep. A typical command line would be "ngrep -q -t -W byline -d any -w 1234567 port 5060 or port 5070 | tee /tmp/trace.txt", replacing 1234567 with the account to be traced and trace.txt with the file to write to. On some systems, adding -R may be necessary to prevent "attempt to drop privileges failed" errors.
3. Make a failing call, or if it's a registration problem wait for at least a full registration interval.
4. Stop ngrep with ctrl-C.
5. Examine /tmp/trace.txt (or whatever file you created) to make sure it includes the call.
6. If collecting the trace for a ticket, upload the file as an attachment to the ticket.

Occasionally, support will ask you for simultaneous Asterisk console and ngrep traces. If they do, it's important that these are for the same call so that support can cross-reference the call information in each.

Wireshark

For RTP problems, Wireshark can be useful. It works in a similar way to ngrep but can collect many protocols and present them in a graphical browser. Its use is outside the scope of this document, but should be reasonably self-explanatory. It's useful to run at the Enswitch cluster and between a handset and the NAT device simultaneously. This can show whether the NAT device is blocking inbound SIP or RTP packets. It can save traces in .pcap format which can be uploaded to tickets.

Back to top