Something is wrong with your web application. The time it’s taking to serve requests is growing. Your logs don’t contain enough. Your database appears bored. How do you know what’s going wrong?

In high-performance production servers it’s vital to know as much about the internals of your system as possible. Traditionally this is done by simple methods like logging anything of potential interest or sending error emails with unexpected exceptions. These methods are insufficient, both due to the level of noise inherent in such systems and because of the difficulty in anticipating what metrics are important during an incident.

Environments such as the JVM and .Net VM have advanced tools for communicating with the VM and for applications to expose internal state, but CPython has lacked similar tooling.

This talk will cover what options CPython application developers have for introspecting their programs; new tools for instrumenting, exposing, and compiling performance and behavior metrics; and techniques for diagnosing runtime issues without restarting the process.

Outline

• The problem
  • How do you know what your Python app is doing?
  • How do you determine its current state without interrupting its current activity?
  • What tools can you use to introspect, monitor, aggregate, and report that state?
• Common and Uncommon Solutions
  • Logging (stdlib, logbook, twiggy)
  • REPL Backdoors and gdb-heap
  • Graphite and Scales
• Other Platforms’ Solutions
  • JVM: Metrics, JMX, jconsole, jstack, jmap
  • .Net: WMI/System.Management, WinDbg, ManagedStackExplorer
  • Ruby: ruby-metrics (Metrics/Scales for Ruby)
• New Solutions for Python
  • socketconsole
  • mmstats
• Example instrumentation of a web application