Skip to main content

9.5 Practical ftrace Filter Options: From Firing Blind to Precision Guided

In the previous section, we solved the "how to see" problem—using function_graph tracer with various formatting options to turn kernel behavior into readable logs.

But the moment you try running function_graph on a server for just a few minutes, you'll hit a new pain point: the sheer volume of output is despair-inducing.

The kernel is a beast capable of executing millions of function calls per second. If you leave full tracing on to catch a single network packet, you'll find yourself drowning in a sea of scheduler_tick, rcu_.*, and timer_.* calls. That one netif_rx call you're looking for becomes a single grain of sand in a sandstorm.

Learning how to filter out irrelevant noise is the core skill of ftrace. It's less like reading a book and more like listening to an incredibly noisy party—you need to mute the background chatter just to hear one person's voice.

Traceable Function List: The Starting Point for Finding a Needle in a Haystack​

Ftrace lists every function it can trace in a single file: available_filter_functions.

On a 5.10 kernel for x86_64, this list is astonishingly long. We can use wc -l to get a sense of its scale:

$ wc -l /sys/kernel/tracing/available_filter_functions
48327 /sys/kernel/tracing/available_filter_functions

Over forty-eight thousand functions.

If you run function_graph without any filters, every single call to these forty-eight thousand functions will spit data at you. This isn't just "information overload"—it's an "information flood".

To keep you from drowning, tracefs provides a series of powerful filter pseudo-files. To keep you from getting lost in the manual, I've organized the most critical ones here (I recommend skimming this table first to get a general impression):

(Insert a simplified version or image of Table 9.4 here, covering key items like set_ftrace_filter, set_ftrace_notrace, set_ftrace_pid, set_event_pid, set_graph_function, etc.)

Next, we'll turn these options into weapons in our arsenal.

Mastering set_ftrace_filter: The Art of Glob Matching​

The most common filtering method is set_ftrace_filter. Manually typing function names is too tedious. For example, if you want to trace vfs_read and vfs_write, you could do it like this:

# Manually specifying functions (tedious)
echo vfs_read vfs_write > /sys/kernel/tracing/set_ftrace_filter

But the real power lies in Glob matching. If you've used shell wildcards before, you're already familiar with this. You can use this syntax to select functions in bulk:

  • tcp*: All functions starting with tcp.
  • *recv: All functions ending with recv.
  • *sock*: All functions with sock in their name.
  • ext4_*: All functions starting with ext4_.

This is incredibly useful for tracing a specific subsystem. For instance, if you want to see all netif related functions, simply using netif* will sweep up a whole bunch of them at once.

However, set_ftrace_filter is capable of much more. Rather than parroting the manual here, let's take a direct look at a fascinating file included with the kernel itself: set_ftrace_filter.

This isn't just a README; it's practically a mini-HOWTO document for ftrace. Try catting it:

$ cat /sys/kernel/tracing/set_ftrace_filter

(Insert screenshot of Figure 9.10's README here)

There's a very interesting feature here—dynamic control switches.

Look at the section about mod: in Figure 9.10. You can set a rule in the filter so that when the program executes a certain function, it automatically disables tracing.

What's the use of this? Imagine you only care about the system boot phase or the flow right before a specific error occurs. Once execution passes that point, the subsequent data is just garbage. With this feature, you can tell ftrace to automatically "hit the brakes" when it encounters a certain function, saving both disk space and brainpower.

Index-based Filtering: When Performance is Sensitive​

While Glob matching is handy, it relies on string processing under the hood. If you are extremely sensitive to performance, or if your filtering conditions are very complex, the string processing itself could become a bottleneck (though it usually isn't, as a kernel engineer, you need to have this kind of meticulousness).

This is where index-based filtering comes into play.

Every line in the available_filter_functions file has an implicit line number (index). Ftrace allows you to write these line numbers directly into set_ftrace_filter. This way, the kernel doesn't need to perform string matching—it just looks up the table, which is highly efficient.

How do we do this? Let's walk through a practical example.

Suppose we want to trace all functions with tcp in their name. First, we use grep -n to find their line numbers in the file:

# Find line numbers of all functions containing "tcp"
grep -n tcp /sys/kernel/tracing/available_filter_functions | \
  cut -d: -f1 | \
  tr '\n' ' '

Here's what this command chain does:

  1. grep -n tcp: Finds lines containing "tcp" and displays their line numbers.
  2. cut -d: -f1: Extracts the line number portion before the colon.
  3. tr '\n' ' ': Replaces newlines with spaces, because the filter file accepts a column of space-separated numbers.

The output looks something like this:

11234 11235 11236 11237 11238 11239 11240 11241 11242 11243 11244 11245 ...

That's a long string of numbers. How many are there? On the 5.10.60 kernel, there are 584:

# Count how many tcp functions there are
grep -c tcp /sys/kernel/tracing/available_filter_functions
584

Now, pipe this string of numbers directly into the filter file, and ftrace will only trace those 584 functions:

# Write index numbers directly to the filter
grep -n tcp /sys/kernel/tracing/available_filter_functions | \
  cut -d: -f1 | \
  tr '\n' ' ' > /sys/kernel/tracing/set_ftrace_filter

We'll use this trick later in our practical script trace_network.sh. To give you a sneak peek, here is a generic Bash function that encapsulates the logic above:

# A helper function to add functions by keyword using index-based filtering
trace_add_by_keyword() {
    local keyword="$1"
    local idx_list
    idx_list=$(grep -n "$keyword" /sys/kernel/tracing/available_filter_functions | \
               cut -d: -f1 | tr '\n' ' ')
    if [ -n "$idx_list" ]; then
        echo "$idx_list" > /sys/kernel/tracing/set_ftrace_filter
    fi
}

You can call it like this:

# Trace all network-related functions
trace_add_by_keyword "tcp"
trace_add_by_keyword "udp"
trace_add_by_keyword "netif"
trace_add_by_keyword "ip_"

This is essentially telling ftrace: "Keep an eye on all functions containing tcp, udp, netif, ip_, and so on." Instantly, your logs will be stripped down to nothing but network-related activity.

Reverse Filtering and Blacklists​

Sometimes, you don't want to specify "what I want to see"—instead, you want to specify "what I don't want to see".

This is blacklist mode. We wrote a script to implement it:

#!/bin/bash
# blacklist_trace.sh: Set up a blacklist to filter out noisy functions

TRACE_DIR="/sys/kernel/tracing"

# Functions we absolutely don't want to see
NOISY_FUNCS="cpu_idle_poll default_idle native_safe_halt"

# Write to notrace (no exclamation mark needed here)
for func in $NOISY_FUNCS; do
    echo "$func" >> "$TRACE_DIR/set_ftrace_notrace"
done

# Alternatively, use the ! modifier in the main filter
echo "!scheduler_tick" >> "$TRACE_DIR/set_ftrace_filter"

Notice a detail here: in set_ftrace_filter, we write !scheduler_tick. That exclamation mark ! is the key—it tells ftrace: "Do not trace this!"

Writing to set_ftrace_notrace, on the other hand, doesn't require an exclamation mark, as it is a dedicated "do-not-trace" list.

This is extremely handy for eliminating noise. For example, if you find the CPU idle loops too noisy:

# Silence the idle loops
echo "!cpu_idle_poll" >> /sys/kernel/tracing/set_ftrace_filter
echo "!default_idle" >> /sys/kernel/tracing/set_ftrace_filter

With just these lines, the world goes quiet.

Module-level Filtering and Filter Commands​

If you only want to trace a specific kernel module (like the ext4 filesystem driver), ftrace provides a very convenient syntax:

# Only trace functions belonging to the ext4 module
echo 'mod:ext4' > /sys/kernel/tracing/set_ftrace_filter

This mod: prefix is actually one form of what are called Filter Commands.

The syntax for Filter commands is extremely powerful, generally looking like this:

<function>:<command>[:<parameter>]

Here, the command can be one of the following:

  • mod:<module>: Specifies a module name.
  • traceon / traceoff: Automatically enables/disables tracing when this function is hit.
  • snapshot: Triggers a snapshot.
  • enable_event / disable_event: Enables/disables a specific event tracer.
  • dump: Dumps the stack once when the function is called.

⚠️ Warning There's a pitfall here that's easy to stumble into: Filter Commands do not affect the filter itself.

What does this mean? It means that if you set a traceoff command, it doesn't change the set of "which functions are traced." It only dynamically controls the on/off state at runtime. Don't expect to use filter commands to filter function names—that's what set_ftrace_filter is for.

For more advanced tricks with Filter Commands, I recommend diving into the official documentation: https://www.kernel.org/doc/html/v5.10/trace/ftrace.html#filter-commands

Preview of the Next Stop: Ping in Action​

Alright, the toolbox is open. We now have the ability to configure the kernel, simple tracing methods at our disposal, and a whole bunch of advanced filtering tricks—Glob, index, blacklist, and commands.

Theorizing on paper is no fun anymore.

In the next section, we're going to throw this knowledge into a real battlefield. We're going to fire a Ping packet at the kernel:

# The battlefield: a simple ping
ping -c 1 192.168.1.1

Then, using the ftrace mastery we just learned, we'll fully reproduce the entire story of what happened inside the kernel in that instant—from the construction of the ICMP packet, to the layer-by-layer encapsulation in the network stack, all the way down to the driver transmitting it.

It's going to be a fascinating process. Get ready, we're heading to the front lines.