Directory upload errors handling a large number of files

Project:RUcore Workflow Management System (WMS)
Version:8.x
Component:File Upload Module
Category:bug report
Priority:normal
Assigned:yuyang
Status:Moved to JIRA
Description

I created a ZIP file for directory upload that has about 8,000 files spread across many directories and sub-directories. The files themselves are very small(~1KB). The test was to see how the system handling a large quantity of files through directory upload.

On test under WMS ID 10596 in the "Mills Chad Collection" was an attempt by me to upload a dataset using directory upload. Only 730 files look like they were processed before something broke.

Attached to this issue is the source ZIP file used to upload.

Comments

#1

Assigned to:yuyang» chadmills
Status:active» test

The error message is "MySQL server has gone away", usually indicating MySQL server has timed out. If in production we have the need to upload directories of this size, we need to ask Dave to increase the MySQL's "wait_timeout" value. -YY

#2

Thanks Yang.

Maybe when dev is up and running again we can play with that timeout value to get an idea of what we need it changed to. We should have that level of control on MySQL on the dev system.

#3

Yang,

If you have the time can we tinker with the dev system and see if changing the "wait_timout" will work? I can perform the test on the dev system. If successful, we can ask that rucore-test have the "wait_timeout" changed as well.

Thanks,
Chad

#4

It is currently set to 8.0 hours.

wait_timeout 28800 seconds
interactive_timeout 28800 seconds

#5

Assigned to:chadmills» yuyang
Status:test» active

Yang,

What would be the address of WMS on dev since I want to start the test there.

Currently I have this URL; but it is throwing an error when I visit it.

https://rep-dev.libraries.rutgers.edu/workflow/dwms/index.php

Notice: Undefined index: auth_source in /mellon/htdocs/workflow/dwms/index.php on line 55

#6

Assigned to:yuyang» chadmills
Status:active» test

Chad, the address in your#5 is correct. Maybe the cache issue? https://rep-dev.libraries.rutgers.edu/workflow/dwms/index.php -YY

#7

Assigned to:chadmills» yuyang
Status:test» active

Using another browser helped.

On dev when I tried to create a resource under a data project or my own faculty collection I am not able to perform a directory upload. The only options available are:

Presentation
- pdf
- txt
Thumbnail
- jpeg

I think the configuration on the dev system needs updating.

#8

Version:8.1» 8.x

I'm moving to 8-x after discussing with Yang and Chad.

#9

Status:active» Moved to JIRA

Back to top