Augmenting Alert Evidence with Runtime Stack Traces

The Deepfactor management portal collects stack trace (‘backtrace’) information for each event that triggers an alert. Typically, Deepfactor is able to show some contextual data from the stack trace information such as the address or module in the trace.

If you have access to symbolic debugging information, such as debuginfo packages, or build artifacts that include debugging information, you can upload this data to the management portal and benefit from additional information and insights.

If the portal does not have symbolic debugging information available for a given stack trace, the following information is shown for each individual stack trace frame:

• The address in the process of the stack frame
• The function name (if the executable is not stripped)
• The module containing the function
• The offset into the function

If the portal does have symbolic debugging information available for a given stack trace, the following information is shown for each individual stack trace frame:

• The address in the process of the stack frame
• The function name
• The module name and path containing the function
• The offset into the function
• The source filename
• The source line number

Below is an example stack trace for which the management portal did not have symbolic debugging information:

0x7ea5 (/usr/lib64/libpthread-2.17.so)

0xfe9fd (/usr/lib64/libc-2.17.so)

Following is an example stack trace for which the management portal did have symbolic debugging information:

0x40068b(/home/user/app.c:16:0)[main + 0x64]

0x400627(/home/user/app.c:7:0)[main + 0x0]

Follow the steps below to upload symbolic debugging information to the management portal: #

1. First, locate any component by navigating to the “Components” view of any application in the management portal (the component selected does not have to be the component for which symbols are being uploaded).
2. Next, in the upper right of the portal, click “Upload binary with symbols.”
3. Follow the instructions to upload one or more binary executables/libraries which contain symbolic debugging information.
4. Finally, re-run the component to produce augmented stack trace information.

Note: Uploading symbolic debugging information will not augment any existing stack traces in alerts previously gathered.

Installing debug symbols for OS/distribution-provided binaries #

Note: Installing debug symbols for OS/distribution-provided binaries requires the ability to log into the Deepfactor management portal as the ‘dfadmin’ user (created during initial management portal setup). If you do not have this access, contact your Deepfactor management portal administrator for help.

Installing symbolic debugging information for OS-provided binaries (such as libc and other base system libraries and executables) can provide useful insight into application behavior by providing symbolic stack trace information when an application enters a system binary/component. To install symbolic debugging information for OS-provided binaries, follow the steps below:

1. Obtain and install (locally, in any convenient VM or machine) the debuginfo package for the library/executable you wish to provide symbolic stack information for. The process to install debuginfo packages varies based on which distribution you are using (for example, on CentOS/RHEL distributions, debuginfo-install is used, while on Debian/Ubuntu distributions, you will need to configure a ‘ddebs’ repository and install a package whose name typically ends with the characters ‘-dbg‘).
2. Obtain and install (locally, in any convenient VM or machine) the binary package matching the debuginfo package installed in step 1.
3. scp (secure copy) the debuginfo binary and the non-debuginfo binary to the Deepfactor management portal using the ‘dfadmin’ user. The ‘binary’ is the .so file or executable(s) installed in steps 1-2.
4. Log in to the Deepfactor management portal as ‘dfadmin’ (using ssh)
5. Use the following command to compute the hash of the non-debuginfo binary copied (where ‘binary_file’ is the name of the binary copied in step 3)

6.  /opt/deepfactor/bin/symbolsvc -gethash binary_file

7. Using the the from Step 6 (a SHA256 hash), copy the debug binary to the following location with ‘HASH’ as the output from Step 6 (where ‘debug_binary’ is the debuginfo binary copied in step 3).

    sudo cp debug_binary  /opt/deepfactor/symbols/customer/hv1/HASH

   Note: if /opt/deepfactor/symbols/customer/hv1 does not exist, use the following command to create it:

   sudo mkdir -p /opt/deepfactor/symbols/customer/hv1

8. (optional) remove the non-debuginfo binary copied in step 3

You may now advise your developers that any stack traces that enter the binary configured above will have symbolic stack trace information displayed.

Note: OS-provided binaries such as libc frequently change/are updated. You will need to perform the preceding steps each time a new library version is installed.