进程中线程查看及线程栈查看

进程线程查看

可以通过一下方法查看:

命令:

  • ps
  • top
  • pidstat
  • pstree

文件:

  • proc filesystem

使用 ps

ps与线程相关的参数

THREAD DISPLAY
       H      Show threads as if they were processes.
       -L     Show threads, possibly with LWP and NLWP columns.
       m      Show threads after processes.
       -m     Show threads after processes.
       -T     Show threads, possibly with SPID column.

使用-L ps -eLo pid,tid,tgid,pgrp,args

[root@centosgpt ~]# ps -eLo pid,tid,tgid,pgrp,args|grep python
  1173   1173   1173   1173 /usr/bin/python -Es /usr/sbin/tuned -l -P
  1173   1594   1173   1173 /usr/bin/python -Es /usr/sbin/tuned -l -P
  1173   1595   1173   1173 /usr/bin/python -Es /usr/sbin/tuned -l -P
  1173   1596   1173   1173 /usr/bin/python -Es /usr/sbin/tuned -l -P
  1173   1613   1173   1173 /usr/bin/python -Es /usr/sbin/tuned -l -P
 77189  77189  77189  77188 grep --color=auto python

使用-m显示 ps -mLe

[root@centosgpt ~]# ps -mLe
   PID    LWP TTY          TIME CMD
  1173      - ?        00:01:03 tuned
     -   1173 -        00:00:00 -
     -   1594 -        00:00:00 -
     -   1595 -        00:01:02 -
     -   1596 -        00:00:00 -
     -   1613 -        00:00:00 -

使用-T显示 ps -T -p [pid]

[root@centosgpt ~]# pidof mysqld
1390
[root@centosgpt ~]# ps -T -p 1390
   PID   SPID TTY          TIME CMD
  1390   1390 ?        00:00:00 mysqld
  1390   1401 ?        00:00:00 mysqld
  1390   1402 ?        00:00:12 mysqld
  1390   1403 ?        00:00:12 mysqld
  1390   1404 ?        00:00:12 mysqld
  1390   1405 ?        00:00:13 mysqld
  1390   1406 ?        00:00:13 mysqld
  1390   1407 ?        00:00:13 mysqld
  1390   1408 ?        00:00:12 mysqld
  1390   1409 ?        00:00:12 mysqld
  1390   1410 ?        00:00:13 mysqld
  1390   1411 ?        00:00:12 mysqld
  1390   1422 ?        00:00:47 mysqld
  1390   1423 ?        00:00:56 mysqld
  1390   1424 ?        00:00:02 mysqld
  1390   1425 ?        00:00:02 mysqld
  1390   1426 ?        00:00:00 mysqld
  1390   1427 ?        00:00:00 mysqld
  1390   1522 ?        00:00:00 mysqld

使用-H显示 ps -H -p [pid]

[root@centosgpt ~]# pidof python
1173
[root@centosgpt ~]# ps -hH -p 1173
  1173 ?        Ssl    0:00 /usr/bin/python -Es /usr/sbin/tuned -l -P
  1173 ?        Ssl    0:00 /usr/bin/python -Es /usr/sbin/tuned -l -P
  1173 ?        Ssl    1:02 /usr/bin/python -Es /usr/sbin/tuned -l -P
  1173 ?        Ssl    0:00 /usr/bin/python -Es /usr/sbin/tuned -l -P
  1173 ?        Ssl    0:00 /usr/bin/python -Es /usr/sbin/tuned -l -P

通过这种方式可以统计下线程的个数

[root@centosgpt ~]# ps -hH -p 1173|wc -l
5

使用 top

使用-H 参数 top -Hbn1 -p [pid]

[root@centosgpt ~]# top -Hbn1 -p 1173
top - 10:26:14 up 4 days,  2:34,  2 users,  load average: 0.00, 0.00, 0.00
Threads:   5 total,   0 running,   5 sleeping,   0 stopped,   0 zombie
%Cpu(s):  0.0 us,  0.0 sy,  0.0 ni,100.0 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
KiB Mem :  2008744 total,  1020476 free,   259172 used,   729096 buff/cache
KiB Swap:  4194300 total,  4194300 free,        0 used.  1444256 avail Mem

   PID USER      PR  NI    VIRT    RES    SHR S %CPU %MEM     TIME+ COMMAND
  1173 root      20   0  573820  23792  10796 S  0.0  1.2   0:00.54 tuned
  1594 root      20   0  573820  23792  10796 S  0.0  1.2   0:00.00 gmain
  1595 root      20   0  573820  23792  10796 S  0.0  1.2   1:02.65 tuned
  1596 root      20   0  573820  23792  10796 S  0.0  1.2   0:00.00 tuned
  1613 root      20   0  573820  23792  10796 S  0.0  1.2   0:00.00 tuned

使用 proc filesystem

proc filesystem 可以列出所有进程的线程信息
在 /proc/[pid]/task 目录下.

[root@centosgpt ~]# ls -l /proc/1173/task
total 0
dr-xr-xr-x. 7 root root 0 Sep 19 22:45 1173
dr-xr-xr-x. 7 root root 0 Sep 19 22:45 1594
dr-xr-xr-x. 7 root root 0 Sep 19 22:45 1595
dr-xr-xr-x. 7 root root 0 Sep 19 22:45 1596
dr-xr-xr-x. 7 root root 0 Sep 19 22:45 1613

使用htop

可以使用htop,F2-Display option 选择相关线程选项
安装:yum install htop

使用pidstat

使用pidstat -t -p [pid]

[root@centosgpt ~]# pidstat -t -p 1173
Linux 5.2.0-rc4 (centosgpt)     10/09/2019      _x86_64_        (2 CPU)

01:27:15 PM   UID      TGID       TID    %usr %system  %guest    %CPU   CPU  Command
01:27:15 PM     0      1173         -    0.01    0.00    0.00    0.02     1  tuned
01:27:15 PM     0         -      1173    0.00    0.00    0.00    0.00     1  |__tuned
01:27:15 PM     0         -      1594    0.00    0.00    0.00    0.00     1  |__gmain
01:27:15 PM     0         -      1595    0.01    0.00    0.00    0.02     0  |__tuned
01:27:15 PM     0         -      1596    0.00    0.00    0.00    0.00     1  |__tuned
01:27:15 PM     0         -      1613    0.00    0.00    0.00    0.00     0  |__tuned

使用pstree

[root@centosgpt ~]# pstree -Aup -h 1173
tuned(1173)-+-{tuned}(1594)
            |-{tuned}(1595)
            |-{tuned}(1596)
            `-{tuned}(1613)

函数栈查看打印

命令:

  • pstack
  • jstack (java)
  • gdb (C/C++/go)
  • kill -SIGQUIT [pid] (go)

API:

  • C:

    glibc backtrace
    Boost stacktrace
    libunwind

  • Java:
    getStackTrace;

  • go:
    panic
    debug.PrintStack
    pprof.Lookup("goroutine").WriteTo
    runtime.Stack

  • python
    traceback objects
    StackSummary Objects
    (FrameSummary Objects)

使用pstack查看

打印运行进行的栈信息 pstack [pid]

[root@centosgpt ~]# pstack 1173
Thread 5 (Thread 0x7f1538db2700 (LWP 1594)):
#0  0x00007f15488bb38d in poll () at ../sysdeps/unix/syscall-template.S:81
#1  0x00007f153b272c4c in g_main_context_iterate.isra.22 () from /lib64/libglib-2.0.so.0
#2  0x00007f153b272d7c in g_main_context_iteration () from /lib64/libglib-2.0.so.0
#3  0x00007f153b272db9 in glib_worker_main () from /lib64/libglib-2.0.so.0
#4  0x00007f153b299970 in g_thread_proxy () from /lib64/libglib-2.0.so.0
#5  0x00007f15492a5dd5 in start_thread (arg=0x7f1538db2700) at pthread_create.c:307
#6  0x00007f15488c602d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111
Thread 4 (Thread 0x7f1533fff700 (LWP 1595)):
#0  0x00007f15488bd0f3 in select () at ../sysdeps/unix/syscall-template.S:81
#1  0x00007f1540dc7000 in time_sleep () from /usr/lib64/python2.7/lib-dynload/timemodule.so
#2  0x00007f154959ecf0 in PyEval_EvalFrameEx () from /lib64/libpython2.7.so.1.0
#3  0x00007f15495a103d in PyEval_EvalCodeEx () from /lib64/libpython2.7.so.1.0
#4  0x00007f154959e53c in PyEval_EvalFrameEx () from /lib64/libpython2.7.so.1.0
#5  0x00007f15495a103d in PyEval_EvalCodeEx () from /lib64/libpython2.7.so.1.0
...

使用jstack查看

打印java堆栈信息

jstack [options] pid
[root@centosgpt ~]# jstack 45864
2019-10-12 05:36:30
Full thread dump Java HotSpot(TM) 64-Bit Server VM (13+33 mixed mode, sharing):

Threads class SMR info:
_java_thread_list=0x00007fc2d8002740, length=10, elements={
0x00007fc320017000, 0x00007fc3200a6800, 0x00007fc3200a8800, 0x00007fc3200ad800,
0x00007fc3200b0000, 0x00007fc3200b2000, 0x00007fc3200b4000, 0x00007fc3200eb800,
0x00007fc3200f0800, 0x00007fc2d8001000
}

"main" #1 prio=5 os_prio=0 cpu=73.15ms elapsed=15.44s \
tid=0x00007fc320017000 nid=0xb329 runnable  [0x00007fc328aa8000]
   java.lang.Thread.State: RUNNABLE
        at sun.nio.ch.Net.accept(java.base@13/Native Method)
        at sun.nio.ch.NioSocketImpl.accept(java.base@13/NioSocketImpl.java:755)
        at java.net.ServerSocket.implAccept(java.base@13/ServerSocket.java:649)
        at java.net.ServerSocket.platformImplAccept(java.base@13/ServerSocket.java:615)
        at java.net.ServerSocket.implAccept(java.base@13/ServerSocket.java:591)
        at java.net.ServerSocket.implAccept(java.base@13/ServerSocket.java:548)
        at java.net.ServerSocket.accept(java.base@13/ServerSocket.java:505)
        at WebSocket.main(WebSocket.java:24)

gdb查看

使用bt指令即可

kill -SIGQUIT [pid] (go)

发送SIGQUIT信号程序退出时打印响应堆栈信息。

相关API

C/C++

glibc backtrace
Boost stacktrace
libunwind

glibc backtrace
  • 相关函数:
//打印函数栈
int backtrace (void **buffer, int size)
//解析backtrace信息,转化为字符串数组
char ** backtrace_symbols (void *const *buffer, int size)
void backtrace_symbols_fd (void *const *buffer, int size, int fd)
  • 实例;
#include <execinfo.h>
#include <stdio.h>
#include <stdlib.h>

void
print_trace (void)
{
  void *array[10];
  size_t size;
  char **strings;
  size_t i;

  size = backtrace (array, 10);
  strings = backtrace_symbols (array, size);

  printf ("Obtained %zd stack frames.\n", size);

  for (i = 0; i < size; i++)
    printf ("%s\n", strings[i]);

  free (strings);
}

void
my_func_1 (void)
{
  print_trace ();
}

int
main (void)
{
  my_func_1 ();
  return 0;
}

  • 运行结果

# gcc -fno-pie -ggdb3 -O3 -o backtrace \ -rdynamic -std=c99 -Wall -Wextra -pedantic-errors \ backtrace.c # ./backtrace Obtained 4 stack frames. ./backtrace(print_trace+0x17) [0x400897] ./backtrace(main+0x9) [0x400789] /lib64/libc.so.6(__libc_start_main+0xf5) [0x7f9a9f376495] ./backtrace() [0x4007b9]

demo from GNU Backtraces

Boost stacktrace
  • 准备: 注意版本是boost169及以上
# yum install boost169.x86_64 boost169-devel.x86_64
  • 相关函数:

boost::stacktrace::stacktrace()

<pre><code><br />- 实例;

```C++
#include <iostream>

#define BOOST_STACKTRACE_USE_ADDR2LINE
#include <boost/stacktrace.hpp>

void
my_func_2 (void)
{
  std::cout << boost::stacktrace::stacktrace () << std::endl;
}

void
my_func_1 (double f)
{
  my_func_2 ();
}

void
my_func_1 (int i)
{
  my_func_2 ();
}

int
main ()
{
  my_func_1 (1);                /* line 19 */
  my_func_1 (2.0);              /* line 20 */
}

  • 运行结果
# g++ -fno-pie -ggdb3 -O0  -o main.out -std=c++11  \
  -Wall -Wextra -pedantic-errors main.cpp -ldl -I/usr/include/boost169

# ./main.out
 0# my_func_2() at /root/boost/main.cpp:9
 1# my_func_1(int) at /root/boost/main.cpp:22
 2# main at /root/boost/main.cpp:28
 3# __libc_start_main in /lib64/libc.so.6
 4# _start in ./main.out

 0# my_func_2() at /root/boost/main.cpp:9
 1# my_func_1(double) at /root/boost/main.cpp:16
 2# main at /root/boost/main.cpp:29
 3# __libc_start_main in /lib64/libc.so.6
 4# _start in ./main.out

libunwind
  • 准备:
# yum install libunwind.x86_64
# yum install libunwind-devel.x86_64
  • 相关函数:
libunwind(3) - overview of the libunwind API
libunwind-ia64(3) - IA-64-specifics
libunwind-ptrace(3) - ptrace(2) support for libunwind
libunwind-setjmp(3) - turbo-setjmp()
unw_apply_reg_state(3) - apply a unwind state
unw_backtrace(3) - fast backtrace() (around 10ns/frame)
unw_create_addr_space(3) - create address space
unw_destroy_addr_space(3) - destroy address space
unw_flush_cache(3) - flush cached info
unw_get_accessors(3) - get pointer to accessor call-backs
unw_get_fpreg(3) - read floating-point register
unw_get_proc_info(3) - get info on current procedure
unw_get_proc_name(3) - get name of current procedure
unw_get_reg(3) - read register
unw_getcontext(3) - get current machine-state
unw_init_local(3) - initialize cursor for local unwinding
unw_init_local2(3) - initialize cursor for local unwinding (extended arguments)
unw_init_remote(3) - initialize cursor for remote unwinding
unw_is_fpreg(3) - check whether floating-point register
unw_is_signal_frame(3) - check whether signal frame
unw_regname(3) - get register name
unw_reg_states_iterate(3) - iterate over unwind states
unw_resume(3) - resume execution in a stack frame
unw_set_cache_size(3) - set cache size
unw_set_caching_policy(3) - set caching policy
unw_set_fpreg(3) - write floating-point register
unw_set_reg(3) - write register
unw_step(3) - step to next (older) frame

from libuwind docs

  • 实例;
/* This must be on top. */
#define _XOPEN_SOURCE 700

#include <stdio.h>
#include <stdlib.h>

/* Paste this on the file you want to debug. */
#define UNW_LOCAL_ONLY
#include <libunwind.h>
#include <stdio.h>
void
print_trace ()
{
  char sym[256];
  unw_context_t context;
  unw_cursor_t cursor;
  unw_getcontext (&context);
  unw_init_local (&cursor, &context);
  while (unw_step (&cursor) > 0)
    {
      unw_word_t offset, pc;
      unw_get_reg (&cursor, UNW_REG_IP, &pc);
      if (pc == 0)
        {
          break;
        }
      printf ("0x%lx:", pc);
      if (unw_get_proc_name (&cursor, sym, sizeof (sym), &offset) == 0)
        {
          printf (" (%s+0x%lx)\n", sym, offset);
        }
      else
        {
          printf (" -- error: unable to obtain symbol name for this frame\n");
        }
    }
  puts ("");
}

void
my_func_3 (void)
{
  print_trace ();
}

void
my_func_2 (void)
{
  my_func_3 ();
}

void
my_func_1 (void)
{
  my_func_3 ();
}

int
main (void)
{
  my_func_1 ();                 /* line 46 */
  my_func_2 ();                 /* line 47 */
  return 0;
}
  • 运行结果
# gcc -fno-pie -ggdb3 -O0  -o main.out -std=c99\
   -Wall -Wextra -pedantic-errors main.c -lunwind
# ./main.out
0x40088f: (my_func_3+0xe)
0x4008a5: (my_func_1+0x9)
0x4008b0: (main+0x9)
0x7f5ff9272495: (__libc_start_main+0xf5)
0x4006c9: (_start+0x29)

0x40088f: (my_func_3+0xe)
0x40089a: (my_func_2+0x9)
0x4008b5: (main+0xe)
0x7f5ff9272495: (__libc_start_main+0xf5)
0x4006c9: (_start+0x29)

Java
  • 相关函数:
Thread.currentThread ().getStackTrace ();
  • 实例;

import java.io.*; class stack { public static void print_trace () { System.out.println ("Printing stack trace:"); StackTraceElement[] elements = Thread.currentThread ().getStackTrace (); for (int i = 1; i < elements.length; i++) { StackTraceElement s = elements[i]; System.out.println ("\tat " + s.getClassName () + "." + s.getMethodName () + "(" + s.getFileName () + ":" + s.getLineNumber () + ")"); } } public static void my_func_1 () { print_trace (); } public static void my_func_2 () { print_trace (); } public static void main (String[]args) throws Exception { my_func_1 (); my_func_2 (); } }
  • 运行结果
# javac stack.java
# java stack
Printing stack trace:
        at stack.print_trace(stack.java:9)
        at stack.my_func_1(stack.java:22)
        at stack.main(stack.java:30)
Printing stack trace:
        at stack.print_trace(stack.java:9)
        at stack.my_func_2(stack.java:26)
        at stack.main(stack.java:31)
Go

panic
debug.PrintStack
pprof.Lookup("goroutine").WriteTo
runtime.Stack

  • 准备
# yum install golang-bin.x86_64
panic
  • 相关函数
panic
  • 实例
package main

func main(){
   my_func_1()
}
func my_func_1(){
   panic("Wantstacktrace")
}

  • 运行结果
# go run stack.go
panic: Wantstacktrace

goroutine 1 [running]:
main.my_func_1(...)
        /root/go/stack.go:7
main.main()
        /root/go/stack.go:4 +0x39
exit status 2

debug.PrintStack
  • 相关函数
debug.PrintStack
  • 实例
package main

import (
     "runtime/debug"
)

func main(){
   my_func_1()
}
func my_func_1(){
   debug.PrintStack()
}

  • 运行结果
[root@centosgpt go]# go run stack1.go
goroutine 1 [running]:
runtime/debug.Stack(0xc000030750, 0xc00006af50, 0x404f7f)
        /usr/lib/golang/src/runtime/debug/stack.go:24 +0x9d
runtime/debug.PrintStack()
        /usr/lib/golang/src/runtime/debug/stack.go:16 +0x22
main.my_func_1(...)
        /root/go/stack1.go:11
main.main()
        /root/go/stack1.go:8 +0x21

pprof
  • 相关函数
pprof.Lookup("goroutine").WriteTo
  • 实例
package main
import (
    "os"
    "runtime/pprof"
)
func main(){
   my_func_1()
}
func my_func_1(){
   pprof.Lookup("goroutine").WriteTo(os.Stdout, 1)
}
  • 运行结果
# go run stack2.go
goroutine profile: total 1
1 @ 0x4b2095 0x4b1eb0 0x4aeada 0x4b8b95 0x4b8b20 0x42d0fe 0x455891
#       0x4b2094        runtime/pprof.writeRuntimeProfile+0x94  /usr/lib/golang/src/runtime/pprof/pprof.go:708
#       0x4b1eaf        runtime/pprof.writeGoroutine+0x9f       /usr/lib/golang/src/runtime/pprof/pprof.go:670
#       0x4aead9        runtime/pprof.(*Profile).WriteTo+0x3d9  /usr/lib/golang/src/runtime/pprof/pprof.go:329
#       0x4b8b94        main.my_func_1+0x64                     /root/go/stack2.go:12
#       0x4b8b1f        main.main+0x1f                          /root/go/stack2.go:9
#       0x42d0fd        runtime.main+0x21d                      /usr/lib/golang/src/runtime/proc.go:203

runtime.Stack
  • 相关函数
runtime.Stack
  • 实例

package main import ( "fmt" "runtime" ) func main(){ my_func_1() } func my_func_1(){ buf := make([]byte,32000); buf = buf[:runtime.Stack(buf, true)] fmt.Printf("%s\n", buf); }
  • 运行结果
# go run stack3.go
goroutine 1 [running]:
main.my_func_1()
        /root/go/stack3.go:13 +0x6d
main.main()
        /root/go/stack3.go:9 +0x20

这有一篇比较深入的文章
聊一聊goroutine stack

Python

2.7 版本:
traceback objects
3.5及以后:
StackSummary Objects
FrameSummary Objects

traceback objects
  • 相关函数
print_stack
  • 实例
import sys, traceback
def my_func_1():
    return print_trace()

def print_trace():
    traceback.print_stack()

if __name__ == '__main__':
    my_func_1()

  • 运行结果
  File "stack.py", line 9, in <module>
    my_func_1()
  File "stack.py", line 3, in my_func_1
    return print_trace()
  File "stack.py", line 6, in print_trace
    traceback.print_stack()

StackSummary Objects
  • 相关函数
classmethod extract(frame_gen, *, limit=None, lookup_lines=True,\
 capture_locals=False)¶
  • 实例
import sys, traceback
def my_func_1():
    return print_trace()

def print_trace():
    summary = traceback.StackSummary.extract(
        traceback.walk_stack(None)
    )
    print(''.join(summary.format()))

if __name__ == '__main__':
    my_func_1()
  • 运行结果
# python3 stack1.py
  File "stack1.py", line 7, in print_trace
    traceback.walk_stack(None)
  File "stack1.py", line 3, in my_func_1
    return print_trace()
  File "stack1.py", line 13, in <module>
    my_func_1()

FrameSummary Objects

FrameSummary Objects 是 StackSummary Objects的组成部分,可以设置打印格式

  • 实例
import sys, traceback

template = (
    '{fs.filename:<26}:{fs.lineno}:{fs.name}:\n'
        '    {fs.line}'
)

def my_func_1():
    return print_trace()

def print_trace():
    summary = traceback.StackSummary.extract(
        traceback.walk_stack(None)
    )
    for fs in summary:
        print(template.format(fs=fs))

if __name__ == '__main__':
    my_func_1()

  • 运行结果
[root@centosgpt python]# python3 stack2.py
stack2.py                 :13:print_trace:
    traceback.walk_stack(None)
stack2.py                 :9:my_func_1:
    return print_trace()
stack2.py                 :20:<module>:
    my_func_1()

参考及引用

Linux: How to View Threads of a Process
Count the Number of Threads in a Process on Linux
How to view threads of a process on Linux
pstree命令的用法(查看进程树)
print call stack in C or C++
GNU 34.1 Backtraces
How to print current call stack
Printing a Stack Trace anywhere in Java
调试利器:dump goroutine 的 stacktrace
traceback — Print or retrieve a stack traceback
17.6. traceback — 异常和调用堆栈跟踪

Be First to Comment

发表评论

电子邮件地址不会被公开。 必填项已用*标注