pprof

golang applications can be profiled with pprof tool  which comes as part golang package. In order to use it is necessary to install golang ( in case it is not already installed ) on targeted system
# yum install golang
In this example we will take a look how to profile openshift master
pprof supports three profile types

  • cpu
  • mem
  • web

As you can conclude, cpu will profile cpu, mem and web will give option to profile mem/cpu via web browser.
In order to get openshift master to profile master process, it is necessary to start it with proper profile, and for case we want to edit /etc/sysconfig/atomic-openshift-master and add there

 OPENSHIFT_PROFILE=cpu

for mem case, replace cpu with mem. It is possible to have one profile at time, so if you specify multiple of them then last one in configuration file will be used, eg

OPENSHIFT_PROFILE=cpu
OPENSHIFT_PROFILE=mem

then only mem profile will be in started and used.
After restarting atomic-openshift-master, in /var/lib/origin there will be created cpu.pprof file

systemclt restart atomic-openshift-master
#ls -l /var/lib/origin/
total 19968
-rw-r--r--. 1 root root 14151744 Dec 17 03:48 cpu.pprof
drwx------. 3 root root       19 Dec 17 03:25 openshift.local.etcd
drwxr-x---. 4 root root       31 Nov 19 07:54 openshift.local.volumes

Immediately after start cpu.pprof will be zero bytes in size and after some time it will be populated, so give it time to start collecting data
Openshift has also openshift node daemon, and in order to profile openshift-node,follow same process, but only pay attention that above changes in configuration file will in /etc/sysconfig/atomic-openshift-node

If you run below command against cpu.pprof while still zero it will fail with below error message what is known as golang bug where it fails to hanle zero bytes cpu.pprof files

go tool pprof /usr/bin/openshift  /var/lib/origin/mem.pprof 
Entering interactive mode (type "help" for commands)
(pprof) top20
panic: runtime error: index out of range

goroutine 1 [running]:
cmd/pprof/internal/driver.sampleFormat(0xc20808e1c0, 0xc20809a180, 0xc2080c2770, 0x0, 0x0, 0x0, 0x0)
	/usr/lib/golang/src/cmd/pprof/internal/driver/driver.go:975 +0x21c
cmd/pprof/internal/driver.generate(0xc20808e101, 0xc20808e1c0, 0x7f5797229e28, 0xc20802afe0, 0x7f5797229de0, 0xc208038060, 0xc20809a180, 0x0, 0x0)
	/usr/lib/golang/src/cmd/pprof/internal/driver/driver.go:1012 +0x11f
cmd/pprof/internal/driver.generateReport(0xc20808e0e0, 0xc20801efc0, 0x2, 0x2, 0x7f5797229e28, 0xc20802afe0, 0x7f5797229de0, 0xc208038060, 0xc20809a000, 0x0, ...)
	/usr/lib/golang/src/cmd/pprof/internal/driver/interactive.go:133 +0x11e
cmd/pprof/internal/driver.interactive(0xc20808e0e0, 0x7f5797229e28, 0xc20802afe0, 0x7f5797229de0, 0xc208038060, 0xc20809a000, 0x0, 0x0)
	/usr/lib/golang/src/cmd/pprof/internal/driver/interactive.go:99 +0x586
cmd/pprof/internal/driver.PProf(0x7f5797229e68, 0xae1e60, 0x96b7a8, 0x96bd70, 0x7f5797229e28, 0xc20802afe0, 0x7f5797229de0, 0xc208038060, 0x0, 0x0, ...)
	/usr/lib/golang/src/cmd/pprof/internal/driver/driver.go:112 +0xc24
main.main()
	/usr/lib/golang/src/cmd/pprof/pprof.go:28 +0x129

pprof collected data can be checked with

# go tool pprof -text -output=pprofdata.txt /usr/bin/openshift  /var/lib/origin/cpu.pprof

This will get output in -text format, and write it in -output=pprofdata.txt file. pprof has many options for output files, run

 # go tool pprof

for full list of supported options. Using pprof can help to nail down issues related to golang and help debug applications based on it.

While working on this, I found materials at below links very useful, I recommend checking them

go-profiling

debugging-performance-issues-in-go-programs

 

Advertisements

#debugging-golang-applciations, #docker, #golang, #openshift-master, #openshift-origin, #pprof