Tracing Node on Linux with LTTNG

Tracing

A specialised use of logging to record information about a programs execution

• Used by Sys Admins for fault diagnosis
• Used by Developers for debugging
• Vs logging/event tracing: high level analysis of less frequent events
• Tracing is low level, very high volume
• Tracing is highly robust and non-invasive
• Tracing can be turned on by demand

Tracing - what it isn't

• Not a debugger by itself, but can help with debugging..
• Not a profiler, but can help with performance..
• Not a logger, but can help with logging..
• Like an x-ray/MRI of a running system
• User apps and kernel can be traced together

Linux Tracing

• 2005 dtrace..
• kernel markers, kprobes, uprobes
• TRACE_EVENT macro
• debugfs - ram based fs for debugging
• sudo find /sys/kernel/debug/tracing/events -type d
• ~1100 tracepoints, ~600 syscalls, nfs, ext4, power, sched, etc
• perf, ftrace, systemtap, strace, ktap, eBPF, dtrace, sysdig, LTTNG
• See Brendan Gregg - www.brendangregg.com

Syscalls

• kernel -> userspace API
• GNU C library - glibc wrapper around syscalls
• Linux API stable for decades, new calls introduced conservatively
• open, connect, accept, read, select/poll
• sys_enter_open, sys_exit_open, sys_enter_connect, sys_exit_connect, etc
• LWN kernelnewbies.org Linux Kernel Development

LTTNG 2.x

• Mathieu Desnoyers - efficios.com
• Linux Trace Tool Next Generation
• Suite of tools to trace linux kernel and user applications
• Very low overhead, 10 years of active development, embedded background
• Easy to install, runs anywhere
• Tracer and viewers

LTTNG Components

• lttng-modules: kernel tracer
• lttng-ust: user space trace, in process
• lttng-tools:
• cli, lttng-ctl
• lttng-sessiond registry daemon
• lttng-consumerd extract trace data
• lttng-relayd: network streaming daemon

LTTNG Viewers

• BabelTrace: cli text viewer
• TraceCompass: Eclipse based front end
• Python Bindings
• github.com/lttng/lttng-analyses

Kernel Tracing Example..

• lttng create my-session
• lttng enable-event --kernel --all
• lttng add-context -k -t pid -t vpid
• lttng start
• do stuff!
• lttng stop
• lttng destroy

LTTNG - UST

• User Space Tracing
• Instrument your app with your own tracepoints
• c/c++ - same TRACE_EVENT macro
• liblttng-ust
• Same as kernel: session setup, run app, collect traces
• View/analyse traces, etc

Tracepoint Macro..

      TRACEPOINT_EVENT(
        hello_world,
        my_first_tracepoint,
      TP_ARGS(
        int, my_integer_arg,
        char*, my_string_arg
      ), TP_FIELDS(
        ctf_string(my_string_field, my_string_arg)
        ctf_integer(int, my_integer_field, my_integer_arg)
        )
      )
      ...
      tracepoint(hello_world, my_first_tracepoint, 23, "hi there!");
    

LTTNG Tracepoints in node.js/io.js

• Added LTTNG Tracepoints in node/io.js
• Landed in io.js v1.02 - #702
• Based on existing code for dtrace and etw
• http_server_request, http_server_response
• http_client_request, http_client_response
• net_server_connection, net_stream_end
• gc_start, gc_done
• Demo!

io.js FTW!

• Open Governance Model (vs corporate stewardship)
• Roadmap is company/industry driven
• Various working groups:
• Stability, NG, Streams, Release, Localisation
• Debugging and Tracing
• Node Foundation & reconciliation proposal

io.js Debugging and Tracing

• Tracing Working Group
• Trace Event
• Chromium Team - Tracing Interface for V8
• Part 1: Embedding API
• Part 2: Control API
• Async, doesn't call when probe is fired
• Next gen debug & profiling tools
• Async Wrap

Node.js App Tracepoints - nftrace

• github.com/nearform/bunyan-nftrace - Bunyan Tracepoints
• github.com/nearform/nftrace-scaffold - Native Tracepoint Generator
• github.com/nearform/grunt-nftrace-gen - Grunt Wrapper around Scaffold
• github.com/nearform/seneca-nftrace - Seneca Message Tracer
• github.com/nearform/nftrace-web-controller - Web Tracepoint Viewer
• github.com/nearform/lttng-vagrant - Vagrant box, set up all the things

LTTNG - quick review!

• High performance low impact tracer
• Kernel and userland
• Turn on/off without process restart
• Remote tracing
• Flight recorder & live tracing
• Common Trace Format
• Kernel tree?
• Trace the whole stack?

Monitoring & Metrics

• Network Monitoring Tools
• Magic Quadrant for Application Performance Monitoring
• Monitoring Categories: Functional, Time Series, Event
• Or: Who, How, What & Why
• Firehose! - Big Data approach
• Semantic/Business Monitoring
• prometheus.io
• Netflix - A Microscope on Microservices
• .. built on perf and Performance Co-Pilot

Finally..

• Interesting area and evolving rapidly.. also watch perf, eBPF, & unikernels!
• Node Tracing & Debugging Working Group
• DiaMon - Linux Foundation Diagnostic & Monitoring Group
• www.brendangregg.com
• nearForm are hiring!