golang applications can be profiled with pprof tool which comes as part golang package. In order to use it is necessary to install golang ( in case it is not already installed ) on targeted system
# yum install golang
In this example we will take a look how to profile openshift master
pprof supports three profile types
As you can conclude, cpu will profile cpu, mem and web will give option to profile mem/cpu via web browser.
In order to get openshift master to profile master process, it is necessary to start it with proper profile, and for case we want to edit /etc/sysconfig/atomic-openshift-master and add there
for mem case, replace cpu with mem. It is possible to have one profile at time, so if you specify multiple of them then last one in configuration file will be used, eg
then only mem profile will be in started and used.
After restarting atomic-openshift-master, in /var/lib/origin there will be created cpu.pprof file
systemclt restart atomic-openshift-master
#ls -l /var/lib/origin/ total 19968 -rw-r--r--. 1 root root 14151744 Dec 17 03:48 cpu.pprof drwx------. 3 root root 19 Dec 17 03:25 openshift.local.etcd drwxr-x---. 4 root root 31 Nov 19 07:54 openshift.local.volumes
Immediately after start cpu.pprof will be zero bytes in size and after some time it will be populated, so give it time to start collecting data
Openshift has also openshift node daemon, and in order to profile openshift-node,follow same process, but only pay attention that above changes in configuration file will in /etc/sysconfig/atomic-openshift-node
If you run below command against cpu.pprof while still zero it will fail with below error message what is known as golang bug where it fails to hanle zero bytes cpu.pprof files
go tool pprof /usr/bin/openshift /var/lib/origin/mem.pprof Entering interactive mode (type "help" for commands) (pprof) top20 panic: runtime error: index out of range goroutine 1 [running]: cmd/pprof/internal/driver.sampleFormat(0xc20808e1c0, 0xc20809a180, 0xc2080c2770, 0x0, 0x0, 0x0, 0x0) /usr/lib/golang/src/cmd/pprof/internal/driver/driver.go:975 +0x21c cmd/pprof/internal/driver.generate(0xc20808e101, 0xc20808e1c0, 0x7f5797229e28, 0xc20802afe0, 0x7f5797229de0, 0xc208038060, 0xc20809a180, 0x0, 0x0) /usr/lib/golang/src/cmd/pprof/internal/driver/driver.go:1012 +0x11f cmd/pprof/internal/driver.generateReport(0xc20808e0e0, 0xc20801efc0, 0x2, 0x2, 0x7f5797229e28, 0xc20802afe0, 0x7f5797229de0, 0xc208038060, 0xc20809a000, 0x0, ...) /usr/lib/golang/src/cmd/pprof/internal/driver/interactive.go:133 +0x11e cmd/pprof/internal/driver.interactive(0xc20808e0e0, 0x7f5797229e28, 0xc20802afe0, 0x7f5797229de0, 0xc208038060, 0xc20809a000, 0x0, 0x0) /usr/lib/golang/src/cmd/pprof/internal/driver/interactive.go:99 +0x586 cmd/pprof/internal/driver.PProf(0x7f5797229e68, 0xae1e60, 0x96b7a8, 0x96bd70, 0x7f5797229e28, 0xc20802afe0, 0x7f5797229de0, 0xc208038060, 0x0, 0x0, ...) /usr/lib/golang/src/cmd/pprof/internal/driver/driver.go:112 +0xc24 main.main() /usr/lib/golang/src/cmd/pprof/pprof.go:28 +0x129
pprof collected data can be checked with
# go tool pprof -text -output=pprofdata.txt /usr/bin/openshift /var/lib/origin/cpu.pprof
This will get output in -text format, and write it in -output=pprofdata.txt file. pprof has many options for output files, run
# go tool pprof
for full list of supported options. Using pprof can help to nail down issues related to golang and help debug applications based on it.
While working on this, I found materials at below links very useful, I recommend checking them