FAQ

Page Discussion History

Difference between revisions of "Debugging"

(Debugging log)
m (Debugging log)
(8 intermediate revisions by 2 users not shown)
Line 1: Line 1:
 
= Introduction =
 
= Introduction =
  
nginx has wide range of debugging features, including detailed debug log. Note that most most debugging nits are only activated when nginx compiled with --with-debug configure argument.
+
nginx has wide range of debugging features, including detailed debug log. Note that most most debugging nits are only activated when nginx compiled with ''--with-debug'' configure argument.
  
 
= Debugging log =
 
= Debugging log =
Line 7: Line 7:
 
See [http://nginx.org/en/docs/debugging_log.html A debugging log] in documentation for details.
 
See [http://nginx.org/en/docs/debugging_log.html A debugging log] in documentation for details.
  
To activate debugging log you have to compile nginx with --with-debug configure option and set debug level in error_log directive.
+
To activate debugging log you have to compile nginx with ''--with-debug'' configure option and set debug level in [[CoreModule#error_log|error_log]] directive.
  
It's possible to debug only connections from specified addresses via debug_connection directive.
+
It's possible to debug only connections from specified addresses via [[EventsModule#debug_connection|debug_connection]] directive.
  
Note that in hard cases (e.g. debugging event method related problems) it's good idea to obtain full debugging log by setting debug level in global error_log.
+
Note that in hard cases (e.g. debugging event method related problems) it's good idea to obtain full debugging log by setting debug level in global ''error_log''.
  
 
= Core dump =
 
= Core dump =
Line 17: Line 17:
 
To obtain core dump you usually have to tune your OS. Though nginx simplifies some typical cases and usually adding
 
To obtain core dump you usually have to tune your OS. Though nginx simplifies some typical cases and usually adding
  
 +
<geshi lang=nginx>
 
worker_rlimit_core  500M;
 
worker_rlimit_core  500M;
 
working_directory  /path/to/cores/;
 
working_directory  /path/to/cores/;
 +
</geshi>
 +
 
to nginx.conf is enough. Then run gdb to obtain backtrace as usual, e.g.
 
to nginx.conf is enough. Then run gdb to obtain backtrace as usual, e.g.
  
 +
<geshi lang=bash>
 
gdb /path/to/nginx /path/to/cores/nginx.core
 
gdb /path/to/nginx /path/to/cores/nginx.core
 
bt
 
bt
 
backtrace full
 
backtrace full
 +
</geshi>
 +
 
If your gdb backtrace warns that No symbol table info available. then you will need to recompile Nginx with the appropriate compiler flags for debugging symbols.
 
If your gdb backtrace warns that No symbol table info available. then you will need to recompile Nginx with the appropriate compiler flags for debugging symbols.
  
 
The exact flags required depend on the compiler used. If you use GCC, the flag -g enables the inclusion of debugging symbols. Additionally disabling compiler optimization using -O0 will make the debugger output easier to understand.
 
The exact flags required depend on the compiler used. If you use GCC, the flag -g enables the inclusion of debugging symbols. Additionally disabling compiler optimization using -O0 will make the debugger output easier to understand.
  
 +
<geshi lang=bash>
 
CFLAGS="-g -O0" ./configure ....
 
CFLAGS="-g -O0" ./configure ....
 +
</geshi>
  
 
= Socket leaks =
 
= Socket leaks =
Line 34: Line 42:
 
Sometimes socket leaks happen. This usually results in "[alert] 15248#0: open socket #123 left in connection 456" messages in error log on nginx reload/restart/shutdown. To debug add
 
Sometimes socket leaks happen. This usually results in "[alert] 15248#0: open socket #123 left in connection 456" messages in error log on nginx reload/restart/shutdown. To debug add
  
 +
<geshi lang=nginx>
 
debug_points abort;
 
debug_points abort;
 +
</geshi>
 +
 
to nginx.conf and configure core dumps (see above). This will result in abort() call once nginx detects leak and core dump.
 
to nginx.conf and configure core dumps (see above). This will result in abort() call once nginx detects leak and core dump.
  
 
Something like this in gdb should be usefull (assuming 456 is connection number from error message from the process which dumped core):
 
Something like this in gdb should be usefull (assuming 456 is connection number from error message from the process which dumped core):
  
 +
<geshi lang=bash>
 
set $c = &ngx_cycle->connections[456]
 
set $c = &ngx_cycle->connections[456]
 
p $c->log->connection
 
p $c->log->connection
Line 44: Line 56:
 
set $r = (ngx_http_request_t *) $c->data
 
set $r = (ngx_http_request_t *) $c->data
 
p *$r
 
p *$r
 +
</geshi>
 +
 
In particular, "p $c->log->connection" will print connection number as used in logs. It will be possible to grep debug log for relevant lines, e.g.
 
In particular, "p $c->log->connection" will print connection number as used in logs. It will be possible to grep debug log for relevant lines, e.g.
  
 +
<geshi lang=bash>
 
fgrep ' *12345678 ' /path/to/error_log;
 
fgrep ' *12345678 ' /path/to/error_log;
+
</geshi>
 +
 
 
= Asking for help =
 
= Asking for help =
  
 
When asking for help with debugging please provide:
 
When asking for help with debugging please provide:
  
nginx -V output
+
* nginx -V output
full config
+
* full config
debug log
+
* debug log
backtrace (if nginx exits on signal)
+
* backtrace (if nginx exits on signal)
 
+
  
 
= Links =
 
= Links =
  
A debugging log
+
[http://nginx.org/en/docs/debugging_log.html A debugging log]

Revision as of 13:11, 27 April 2012

Contents

Introduction

nginx has wide range of debugging features, including detailed debug log. Note that most most debugging nits are only activated when nginx compiled with --with-debug configure argument.

Debugging log

See A debugging log in documentation for details.

To activate debugging log you have to compile nginx with --with-debug configure option and set debug level in error_log directive.

It's possible to debug only connections from specified addresses via debug_connection directive.

Note that in hard cases (e.g. debugging event method related problems) it's good idea to obtain full debugging log by setting debug level in global error_log.

Core dump

To obtain core dump you usually have to tune your OS. Though nginx simplifies some typical cases and usually adding

worker_rlimit_core  500M;
working_directory   /path/to/cores/;

to nginx.conf is enough. Then run gdb to obtain backtrace as usual, e.g.

gdb /path/to/nginx /path/to/cores/nginx.core
bt
backtrace full

If your gdb backtrace warns that No symbol table info available. then you will need to recompile Nginx with the appropriate compiler flags for debugging symbols.

The exact flags required depend on the compiler used. If you use GCC, the flag -g enables the inclusion of debugging symbols. Additionally disabling compiler optimization using -O0 will make the debugger output easier to understand.

CFLAGS="-g -O0" ./configure ....

Socket leaks

Sometimes socket leaks happen. This usually results in "[alert] 15248#0: open socket #123 left in connection 456" messages in error log on nginx reload/restart/shutdown. To debug add

to nginx.conf and configure core dumps (see above). This will result in abort() call once nginx detects leak and core dump.

Something like this in gdb should be usefull (assuming 456 is connection number from error message from the process which dumped core):

set $c = &ngx_cycle->connections[456]
p $c->log->connection
p *$c
set $r = (ngx_http_request_t *) $c->data
p *$r

In particular, "p $c->log->connection" will print connection number as used in logs. It will be possible to grep debug log for relevant lines, e.g.

fgrep ' *12345678 ' /path/to/error_log;

Asking for help

When asking for help with debugging please provide:

  • nginx -V output
  • full config
  • debug log
  • backtrace (if nginx exits on signal)

Links

A debugging log