Connection to mySQL database failed : (1040) Too many connections -- test on Staging

Project:RUcore Workflow Management System (WMS)
Version:8.1
Component:General
Category:bug report
Priority:normal
Assigned:ananthan
Status:closed
Description

Some people can go from descriptive to rights metadata tabs but not source or technical tabs there are too many logins in both chromes and Firefox.

Geoff found using chrome that after after server failure he lost access to a number of fields in the descriptive metadata tab, everything below subject (geographic-hierarchical.

Debbie went from descriptive tab to source tab and was locked out. She went into the technical tab and could not get in either. Could get in on the rights tab and then could go back ti source and technical and was able to get in.

Comments

#1

This happened again during testing this morning.

#2

Assigned to:Anonymous» yuyang

#3

I suspect file processing is causing this issue.

#4

mysql max connections was set to 150 which led to the
"Too many connections" error

On Monday 5/2/16 I increased max connections to 300
so we could see what happened with testing on Weds.

#5

Today (Wed 5/4/16) I decided to watch and study what is going on
with mysql when testing was taking place.

Looking at /tmp/php_errors showed

Between 10:26:55 and 10:58:59 there were 1984 instances of

[04-May-2016 10:49:14 America/New_York] PHP Warning: mysqli::mysqli(): (08004/
1040): Too many connections in /mellon/htdocs/openwms/test/dwms/common/db/DbUti
l.php on line 83

Using the mysql processlist command showed that a large number of mysql connections
to dwms were in the Sleep state. While this fluctuated from time to time it always was
the most listed connection over any other database. (Most other ones cleared out pretty
quickly).

One snapshot in time reveals that of 238 connections in the mysql processlist 234 of them
were for dwms.

| Id | User | Host | db | Command | Time | State | Info
| 610691 | dwms | localhost | dwms | Sleep | 1272 | |
| 610693 | dwms | localhost | dwms | Sleep | 1239 | |
....
| 674876 | dwms | localhost | dwms | Sleep | 83 | |
| 674877 | dwms | localhost | dwms | Sleep | 83 | |

While the other four were:

| Id | User | Host | db | Command | Time | State | Info
| 7558 | fedora | localhost:42009 | fedora32 | Sleep | 0 | |
| 607839 | fedora | localhost:33006 | fedora32 | Sleep | 1341 | |
| 687031 | portal | localhost | portal-test | Sleep | 0 | |
| 687039 | root | localhost | | Query | 0 | | show processlist

So it definitely seems dwms related, the question is whether or not WMS is making
too many connections and not closing them quick enough, or if it truly needs this many
connections to function.

My impression is that something is going wrong in how WMS creates and closes
mysql connections and that it will continue to eat up as many connections as I give it.

#6

Dave, great sleuth work.

#7

I also tried stepping through a rep-test WMS metadata/file creation process
while watching connections.

WMS rucore30022400001/10475
I uploaded a .tiff file used StillImage as the type and had the system generate
all presentation copies.

The mysql processlist showed 50 connections at its peak (once the work was
completed they started to clear). These were all for dwms and I believe were
isolated to my work.

I also had turned on mysql binary logging to capture all the mysql (write) commands
that were processed. There are alot of COMMITs and SET TIMESTAMP entries as well
as what appears to be single value updates followed by commits. I have attached the
binary log to be reviewed.

There are three attachments

mysql_file_process.txt - the WMS popup window output
mysql_process2.tx - the mysql processlist output
mysl_bin_log.txt - the actual mysql operations performed

#8

Assigned to:yuyang» ananthan
Status:active» test

Test for "too many connections". -YY

#9

Title:Connection to mySQL database failed : (1040) Too many connections» Connection to mySQL database failed : (1040) Too many connections -- test on Staging
Assigned to:ananthan» martyb

We didn't experience this problem during yesterday's testing session. I will keep this open to make sure this does not happen on the Staging server. Assigning it to Marty B.

#10

Assigned to:martyb» ananthan
Status:test» fixed

#11

Status:fixed» closed

Back to top