LTTng kernel tracing support

[Trass3r]

@trgibeau @ivberg-zz you mentioned WSL2 in https://tracingsummit.org/ts/2019/WPA/

I successfully built https://github.com/lttng/lttng-modules#building using https://github.com/microsoft/WSL2-Linux-Kernel and ran the lttng daemon: https://lttng.org/docs/#doc-tracing-the-linux-kernel
but I always get 'Kernel tracer not available'
(a) the modules don't get loaded by the daemon as described in the docs, (b) when loading them manually it doesn't make a difference, (c) neither does building them right into the kernel

Have you tried this internally? Should it work?

[WSLUser]

Ref #5823

[Trass3r]

I don't see any connection other than they also compiled a custom kernel.

[WSLUser]

You need kexec/kdump.

[Trass3r]

How does lttng require those?

[WSLUser]

Should be a dependency. Also possible you need bcc kernel options enabled instead or in addition to depending on what you're trying to get out of it.

[Trass3r]

Well whole system tracing (kernel+user) for performance investigations just like ETW.

[WSLUser]

Well some tracing requires kexec/kdump, others just need the bcc kernel options. Most performance related tracing should be good with bcc but kdump can yield more data to churn through. Try enabling the options described here: https://github.com/iovisor/bcc/blob/master/INSTALL.md. Otherwise the other thread should help you to a certain extent.

[ivberg]

Sorry for the late reply @Trass3r. I didn't get a notification for some reason that I know of to see this.

P.S. We OSSed the LTTng tooling to read LTTng/perf traces which is now at https://github.com/microsoft/Microsoft-Performance-Tools-Linux

Yes, LTTng support can be compiled manually for WSL2 using these or similar instructions:
// Set distro to WSL2. Using Ubuntu user-space for this example

$ sudo apt-get update
$ sudo apt install bison flex build-essential libelf-dev libssl-dev
$ mkdir ~/WSL2/
$ cd ~/WSL2/
$ git clone https://github.com/microsoft/WSL2-Linux-Kernel.git
$ cd WSL2-Linux-Kernel

$uname -r
4.19.67-microsoft-standard

$ git tag
$ git checkout tags/$(uname -r) -b user/ivberg/$(uname -r)

$ cd ~/WSL2/WSL2-Linux-Kernel
$ touch .scmversion
$ make KCONFIG_CONFIG=Microsoft/config-wsl -j4 oldconfig
$ make KCONFIG_CONFIG=Microsoft/config-wsl -j4
$ make KCONFIG_CONFIG=Microsoft/config-wsl modules
$ sudo make modules_install -j4

$ git clone https://github.com/lttng/lttng-modules.git
$ cd ~/WSL2/WSL2-Linux-Kernel/lttng-modules/

// Checkout latest stable
$ git checkout v2.10.11
$ make
$ sudo make modules_install

$ sudo depmod -a

$ sudo modprobe lttng-tracer
$ modinfo lttng-tracer

// Install LTTng Tools

• Install lttng-tools from the stable-2.10 PPA, use no-install-recommends so we don't pull the modules package.

$ sudo apt-add-repository ppa:lttng/stable-2.10
$ sudo apt install lttng-tools babeltrace --no-install-recommends
$ sudo adduser $USER tracing

• Close the WSL shell for the group membership to work

$ take a trace with LTTng per https://lttng.org/docs/v2.10/#doc-tracing-the-linux-kernel

• Copy to Desktop if you want to use WPA and Microsoft-Performance-Tools-Linux to view the trace in Windows
  $ cp -R /tmp/my-kernel-trace /mnt/c/Users/ivberg/Desktop/

[Trass3r]

By now I've switched to https://github.com/nathanchance/WSL2-Linux-Kernel/commits/next but still getting Kernel tracer not available, at least when lttng is builtin: https://github.com/lttng/lttng-modules#kernel-built-in-support.
Edit: interesting, when building it as a module it seems to work.

I didn't get a notification for some reason that I know of to see this.

@ivberg maybe I picked the wrong user initially?

P.S. We OSSed the LTTng tooling to read LTTng/perf traces which is now at https://github.com/microsoft/Microsoft-Performance-Tools-Linux

Nice! Though perf is crippled in WSL due to #4678 and cpu frequency readings from /proc/cpuinfo or lscpu are not correct either (#5995).

[ivberg]

That's good to know about perf being crippled in WSL2 @Trass3r !

Glad to see you got it work when building as a module.

P.S. Yea apparently I have had 2 usernames on GitHub now associated with Microsoft. Probably due to issues with 2FA and GitHub needing to delete/recreate the account. I am guessing ivberg-zz is the old username for history purposes?

[Trass3r]

Recently I also had to disable CONFIG_SECURITY_LOCKDOWN_LSM to get the debug tracing folder at all.
https://gist.github.com/Trass3r/1aed0b9d30de084856ea2862f65e8cf4

It still only works when built as a module. And it seems to be crucial to do make modules_install on the plain kernel before compiling the lttng modules.