Investigate about server side errors for dmwmmon data service

[nataliaratnikova]

Look at the errors logged by cmsweb server, classify and investigate.

[nataliaratnikova]

As of last week. most frequent error is about size limit, e.g.:

Fri Jan 29 00:45:27 2016 Apache2::SizeLimit httpd process too big, exiting at SIZE=409964 KB SHARE=3292 KB UNSHARED=406672 REQUESTS=495 LIFETIME=85879 seconds

Other errors are these:

-sh-4.1$ cat /dmwmmon/error_log_ | sort -u | grep -v Apache2::Size
[Fri Jan 29 13:59:14 2016] [notice] Apache/2.2.31 (Unix) mod_ssl/2.2.31 OpenSSL/1.0.1p mod_perl/2.0.9 Perl/v5.10.1 configured -- resuming normal operations
[Fri Jan 29 13:59:40 2016] [notice] Apache/2.2.31 (Unix) mod_ssl/2.2.31 OpenSSL/1.0.1p mod_perl/2.0.9 Perl/v5.10.1 configured -- resuming normal operations
Issuing rollback() due to DESTROY without explicit disconnect() of DBD::Oracle::db handle cms_sitedb at /data/srv/beHG1602d-comp/sw.coimbra/slc6_amd64_gcc493/external/p5-apache-dbi/1.08/lib/perl5/Apache/DBI.pm line 189.
Issuing rollback() due to DESTROY without explicit disconnect() of DBD::Oracle::db handle cms_sitedb at /data/srv/beHG1602f/sw/slc6_amd64_gcc493/external/p5-apache-dbi/1.08/lib/perl5/Apache/DBI.pm line 189.
[Thu Jan 28 18:18:47 2016] [notice] Apache/2.2.31 (Unix) mod_ssl/2.2.31 OpenSSL/1.0.1p mod_perl/2.0.9 Perl/v5.10.1 configured -- resuming normal operations
[Thu Jan 28 18:18:48 2016] [notice] Apache/2.2.31 (Unix) mod_ssl/2.2.31 OpenSSL/1.0.1p mod_perl/2.0.9 Perl/v5.10.1 configured -- resuming normal operations
[Tue Feb 02 12:12:03 2016] [notice] Apache/2.2.31 (Unix) mod_ssl/2.2.31 OpenSSL/1.0.1p mod_perl/2.0.9 Perl/v5.10.1 configured -- resuming normal operations
[Tue Feb 02 14:58:42 2016] [notice] Apache/2.2.31 (Unix) mod_ssl/2.2.31 OpenSSL/1.0.1p mod_perl/2.0.9 Perl/v5.10.1 configured -- resuming normal operations
[Tue Feb 02 14:58:49 2016] [notice] Apache/2.2.31 (Unix) mod_ssl/2.2.31 OpenSSL/1.0.1p mod_perl/2.0.9 Perl/v5.10.1 configured -- resuming normal operations
[Tue Feb 02 14:58:59 2016] [notice] Apache/2.2.31 (Unix) mod_ssl/2.2.31 OpenSSL/1.0.1p mod_perl/2.0.9 Perl/v5.10.1 configured -- resuming normal operations
Use of uninitialized value in subroutine entry at /data/srv/beHG1602d-comp/sw.coimbra/slc6_amd64_gcc493/cms/DMWMMON-datasvc/1.0.5/perl_lib/PHEDEX/Web/FrontendAuth.pm line 39.
Use of uninitialized value in subroutine entry at /data/srv/beHG1602e/sw/slc6_amd64_gcc493/cms/DMWMMON-datasvc/1.1.0/perl_lib/PHEDEX/Web/FrontendAuth.pm line 39.
Use of uninitialized value in subroutine entry at /data/srv/beHG1602f/sw/slc6_amd64_gcc493/cms/DMWMMON-datasvc/1.1.0/perl_lib/PHEDEX/Web/FrontendAuth.pm line 39.
Using default DB access for 'dumpspacequery' API
Using default DB access for 'getlastrecord' API
Using default DB access for 'nodes' API
-sh-4.1$

#2
Updated by Natalia Ratnikova 2 days ago
Comment Edit

This error:

Use of uninitialized value in subroutine entry at /data/srv/beHG1602f/sw/slc6_amd64_gcc493/cms/DMWMMON-datasvc/1.1.0/perl_lib/PHEDEX/Web/FrontendAuth.pm line 39

happens regularly on two servers:

-sh-4.1$ grep -c "FrontendAuth.pm line 39" vocms_/dmwmmon/error_
vocms0131/dmwmmon/error_log_20160127.txt:8
vocms0131/dmwmmon/error_log_20160128.txt:10
vocms0131/dmwmmon/error_log_20160129.txt:4
vocms0131/dmwmmon/error_log_20160130.txt:2
vocms0131/dmwmmon/error_log_20160131.txt:10
vocms0131/dmwmmon/error_log_20160201.txt:10
vocms0131/dmwmmon/error_log_20160202.txt:6
vocms0132/dmwmmon/error_log_20160127.txt:2
vocms0132/dmwmmon/error_log_20160128.txt:10
vocms0132/dmwmmon/error_log_20160129.txt:12
vocms0132/dmwmmon/error_log_20160130.txt:6
vocms0132/dmwmmon/error_log_20160131.txt:2
vocms0132/dmwmmon/error_log_20160201.txt:6
vocms0132/dmwmmon/error_log_20160202.txt:8
vocms0136/dmwmmon/error_log_20160127.txt:0
vocms0136/dmwmmon/error_log_20160129.txt:0
vocms0136/dmwmmon/error_log_20160130.txt:0
vocms0136/dmwmmon/error_log_20160202.txt:0
vocms0161/dmwmmon/error_log_20160127.txt:0
vocms0161/dmwmmon/error_log_20160128.txt:0
vocms0161/dmwmmon/error_log_20160129.txt:0
vocms0161/dmwmmon/error_log_20160131.txt:0
vocms0161/dmwmmon/error_log_20160202.txt:0
vocms0163/dmwmmon/error_log_20160127.txt:0
vocms0163/dmwmmon/error_log_20160128.txt:0
vocms0163/dmwmmon/error_log_20160129.txt:0
vocms0163/dmwmmon/error_log_20160130.txt:0
vocms0163/dmwmmon/error_log_20160201.txt:0
vocms0163/dmwmmon/error_log_20160202.txt:0
vocms0165/dmwmmon/error_log_20160126.txt:0
vocms0165/dmwmmon/error_log_20160127.txt:0
vocms0165/dmwmmon/error_log_20160128.txt:0
vocms0165/dmwmmon/error_log_20160130.txt:0
vocms0165/dmwmmon/error_log_20160201.txt:0
vocms0165/dmwmmon/error_log_20160202.txt:0

The corresponding code line in perl_lib/PHEDEX/Web/FrontendAuth.pm is:

$self->{HEADER}{lc $} = unac_string('utf-8',$headers->{$});