In certain circumstances, it may not be possible for Deepfactor to instrument (observe) a process. This results in a lack of event telemetry being delivered to the Deepfactor management portal, and consequently, lack of alerts or insights being shown.
In many cases, the Deepfactor runtime library can determine ahead of time if it will not be able to observe a new process being launched and communicate this fact to the management portal or via messages printed to the console. This gives the Deepfactor user the knowledge that some processes may not have been observed.
This document lists the various warning messages that are shown in the management portal or console when these situations occur, and possible remediations.
“Setuid bit set. You must run dfctl as root in this scenario.”
“Setgid bit set. You must run dfctl as root in this scenario.”
These messages are displayed when the command or container entrypoint being run by dfctl has either the setuid or setgid bit set. As Deepfactor uses LD_PRELOAD to insert its instrumentation library, this approach is incompatible with executables that have either the setuid or setgid bit set. In these scenarios, you can run dfctl as root to work around the operating system check that rejects LD_PRELOAD for setuid/setgid executables, or you can remove the setuid/setgid bit if not needed.
“OS distro version not supported”
This message is displayed when an unsupported distribution/version is detected. Deepfactor supports the Linux distributions listed in the following KB article:
Deepfactor Support Matrix
“Entrypoint does not match libc environment”
This message is displayed when Deepfactor detects that the entrypoint executable defined in a container manifest does not match the libc style used by the container (glibc or musl). This check is performed against the “standard” libc shipped with a particular distribution (musl for Alpine containers and glibc otherwise). Deepfactor can operate in hybrid libc environments (for example, when a glibc based executable is installed in an Alpine container along with glibc itself); this warning message is displayed to ensure that the developer understands that such a configuration is being used.
Note: dfctl run (in non-container/Kubernetes deployments) does not support hybrid libc.
“The libc is unknown for the provided entrypoint/command”
This message is displayed when Deepfactor does not recognize the type of libc library used by the command or container entrypoint. Deepfactor supports musl and glibc varieties of libc. The user can remedy this by verifying that the entrypoint is linked using musl or glibc. To do this, the user can use the system “ldd” command, or use /opt/deepfactor/bin/df-get-libc.sh. The Deepfactor df-get-libc.sh script uses ldd internally and will report the libc of an executable script/interpreter if applicable.
“DF JVMTI Agent disabled in presence of another agent”
These messages are displayed when the Deepfactor Java LSA (language specific agent) is not loaded when a Java application is being observed in the presence of another conflicting Java agent injected by potentially another tool. In such a case, stack traces and usage information for java applications will not be available
“Please remove libdf.so from your pod container env and allow for mutating webhook to inject.”
“libdf.so detected in pod container env LD_PRELOAD along with additional preload libraries.”
These messages are displayed when Deepfactor detected an attempt to instrument/observe an application whose container image was already built with Deepfactor. To remedy this, the user can use a container image that does not already have Deepfactor loaded. An alternate remedy is to ensure that the container manifest does not specify an LD_PRELOAD value with Deepfactor’s libdf.so.
“Unable to fetch image manifest”
This message is displayed when the mutating webhook cannot examine a container image manifest to determine entrypoint and libc characteristics or ImageID. This may be due to permission or communication issues when communicating with the container registry. To remedy this, the user can review the error description provided with the message. Common error messages and their remedies can be searched for in Deepfactor’s KB articles.
“package manager not found or supported”
This message is displayed during execution of a containerized application in situations when Deepfactor cannot determine which OS package manager (dpkg, apk, rpm, etc) should be used to query OS package information. OS package information is used to provide dependency analysis and CVE information for packages used by the application. This message may indicate that a distroless container is being used, or that the user has removed the package manager from their container image. To remedy this, ensure that the standard package manager binary is functional within the container (dpkg for Ubuntu and Debian based containers, rpm for CentOS and RedHat based containers, and apk for Alpine based containers).
“sh test failed”
This message is displayed during execution of a containerized application in situations when Deepfactor detects that a suitable “sh” shell is not available, and that a suitable package manager was not detected (see previous message). Deepfactor tests OS distribution libc version compatibility with “sh” in scenarios where a package manager cannot be located or used. To remedy this, either:
- Ensure that the standard package manager binary is functional within the container (dpkg for Ubuntu and Debian based containers, rpm for CentOS and RedHat based containers, and apk for Alpine based containers).
- Or, if a package manager cannot be used, ensure that “/bin/sh” exists in the shell and is a standard Linux shell executable (bash, etc).
“Deprecated Deepfactor CLI/Webhook version detected”
This message is displayed when you run your applications with an older deprecated version of Deepfactor CLI (dfctl) or K8s admission webhook.
If you are using dfctl, please upgrade it using the following command.
curl https://repo.deepfactor.io/install-dfctl.sh | sh -
If you are using Deepfactor’s helm charts on your K8s cluster, then follow the instructions in the following document to upgrade the webhook