Migration of RARCH objects to managed ARCH datastreams in RUcore

Project:RUcore Jobs & Reports
Component:Job - production
Category:task
Priority:normal
Assigned:dhoover
Status:closed
Description

This task is for the migration of RARCH objects to managed ARCH datastreams. References to scripts and jobs to run for this task will be mentioned as needed here.

Comments

#1

day#.sh scripts are being run on rep-prod

day1.sh run on March 22

I forgot to run the time command so no run time is available

dhoover@rep-prod:/home/RARCH> more day1.log
completed rutgers-lib:36336...
completed rutgers-lib:39092..

Note for day1.sh I had to cancel the run and restart it that led to
two ARCH being made instead of one:

/rep-prod/repository/datastreams/2016/0322/08/47/rutgers-lib_36336+ARCH1+ARCH1.0
/rep-prod/repository/datastreams/2016/0322/18/55/rutgers-lib_36336+ARCH1+ARCH1.1

So the one from 03/22/2106 at 8:47 am can be deleted.

#2

day2.sh run on 3/23/16

completed rutgers-lib:39097...
completed rutgers-lib:38536...
completed rutgers-lib:36333...
0.11user 0.63system 2:41:37elapsed 0%CPU (0avgtext+0avgdata 2780maxresident)k
15048inputs+24outputs (27major+5270minor)pagefaults 0swaps

#3

day3.sh run on 3/25/16

dhoover@rep-prod:/home/RARCH> more day3.log
completed rutgers-lib:36423...
completed rutgers-lib:36360...
completed rutgers-lib:36348...
0.12user 0.59system 2:19:49elapsed 0%CPU (0avgtext+0avgdata 2892maxresident)k
10088inputs+24outputs (12major+5205minor)pagefaults 0swaps

#4

day4.sh run on 3/26/2106

dhoover@rep-prod:/home/RARCH> more day4.log
completed rutgers-lib:36337...
completed rutgers-lib:38539...
completed rutgers-lib:34233...
completed rutgers-lib:36374...
0.10user 0.56system 2:24:04elapsed 0%CPU (0avgtext+0avgdata 2784maxresident)k
19520inputs+32outputs (17major+6745minor)pagefaults 0swaps

#5

day5.log run on 3/27/2016

dhoover@rep-prod:/home/RARCH> more day5.log
completed rutgers-lib:36334...
completed rutgers-lib:39090...
completed rutgers-lib:39091...
completed rutgers-lib:36331...
completed rutgers-lib:36354...
0.11user 0.54system 2:17:57elapsed 0%CPU (0avgtext+0avgdata 2892maxresident)k
17752inputs+40outputs (18major+8307minor)pagefaults 0swaps

#6

day6.log run on 4/5/16 00:28

dhoover@rep-prod:/home/RARCH> more day6.log
completed rutgers-lib:36352...
completed rutgers-lib:36375...
completed rutgers-lib:38525...
completed rutgers-lib:36270...
completed rutgers-lib:36489...
completed rutgers-lib:38490...
0.14user 0.65system 2:41:29elapsed 0%CPU (0avgtext+0avgdata 2780maxresident)k
18672inputs+48outputs (23major+9866minor)pagefaults 0swaps

#7

day7.log run on 4/5/16 17:53

dhoover@rep-prod:/home/RARCH> more day7.log
completed rutgers-lib:38498...
completed rutgers-lib:39093...
completed rutgers-lib:36371...
completed rutgers-lib:36366...
completed rutgers-lib:38538...
completed rutgers-lib:38498...
0.10user 0.48system 2:03:30elapsed 0%CPU (0avgtext+0avgdata 2780maxresident)k
18808inputs+48outputs (28major+9893minor)pagefaults 0swaps

#8

day8.log run on 4/5/16 23:15

rep-prod:/home/RARCH # more day8.log
completed rutgers-lib:39232...
completed rutgers-lib:36368...
completed rutgers-lib:34330...
completed rutgers-lib:36358...
completed rutgers-lib:36361...
completed rutgers-lib:35869...
completed rutgers-lib:36362...
0.10user 0.54system 2:12:53elapsed 0%CPU (0avgtext+0avgdata 2784maxresident)k
25712inputs+56outputs (28major+11445minor)pagefaults 0swaps

#9

day9.log run on 4/6/16 19:47

rep-prod:/home/RARCH # more day9.log
completed rutgers-lib:34308...
completed rutgers-lib:36335...
completed rutgers-lib:39095...
completed rutgers-lib:38523...
completed rutgers-lib:36338...
completed rutgers-lib:36350...
completed rutgers-lib:36364...
completed rutgers-lib:36365...
0.08user 0.58system 2:14:13elapsed 0%CPU (0avgtext+0avgdata 2780maxresident)k
26440inputs+64outputs (31major+13005minor)pagefaults 0swaps

#10

RARCH day#.sh scritps are temporarily halted

While the system resources have allowed for the pushing of multiple
batches per day it ended up having a negative impact on the
Repository_incremental backup. Until staging space clears and
some more files can make it to tape I will have to hold off on the
remaining scripts day10-day14 accounting for 103 more files and
approximately 1.00 TB of space. The first 9 scripts processed 44 files
accounting for approximately 2.0TB of space.

I will let you know when I can resume the scripts.

#11

In addition to the space that the new ARCHs take up for the incremental
backup, these files are now being backed twice in the full backup.

After checking that everything went ok with the moving of
the RARCH files, please make sure that the files under
/rep-prod/repository/rarch for the completed rutgers-lib ids
get deleted promptly.

#12

Lastly, this project has provided 147 rutgers-ids with RARCHs
but on the filesystem there are 473 rarch/rutgers-lib* directories
which leaves 327 unaccounted for. I expected that this project
would result in the total removal of all rarch/rutgers-lib directories
once all the RARCHs were moved to Managed.

attached is a sorted list of all the rarch/rutgers-lib directory names
currently on the filesystem. Please review and report on what is to
be done with the remaining rarch files.

Two examples of the ones not related to the list of ids in this project are

dhoover@rep-prod:/tmp> tar -tvf /rep-prod/repository/rarch/rutgers-lib_24146/rutgers-lib_24146.tar
-rw-r--r-- eris/staff 403409990 2008-10-10 16:53 Gombergat82.avi
drwxr-xr-x eris/staff 0 2008-10-10 16:16 Gombergat82/
drwxr-xr-x eris/staff 0 2004-12-29 13:37 Gombergat82/VIDEO_TS/
-rw-r--r-- eris/staff 12288 2004-12-29 13:37 Gombergat82/VIDEO_TS/VIDEO_TS.BUP
-rw-r--r-- eris/staff 12288 2004-12-29 13:37 Gombergat82/VIDEO_TS/VIDEO_TS.IFO
-rw-r--r-- eris/staff 219136 2004-12-29 13:37 Gombergat82/VIDEO_TS/VIDEO_TS.VOB
-rw-r--r-- eris/staff 34816 2004-12-29 13:37 Gombergat82/VIDEO_TS/VTS_01_0.BUP
-rw-r--r-- eris/staff 34816 2004-12-29 13:37 Gombergat82/VIDEO_TS/VTS_01_0.IFO
-rw-r--r-- eris/staff 1044379648 2004-12-29 13:37 Gombergat82/VIDEO_TS/VTS_01_1.VOB
-rw-r--r-- eris/staff 1044310016 2004-12-29 13:37 Gombergat82/VIDEO_TS/VTS_01_2.VOB
-rw-r--r-- eris/staff 102668288 2004-12-29 13:37 Gombergat82/VIDEO_TS/VTS_01_3.VOB

dhoover@rep-prod:/tmp> tar -tvf /rep-prod/repository/rarch/rutgers-lib_39264/rut
gers-lib_39264-RARCH1.tar
drwx------ scaredpoet/scaredpoet 0 2013-03-01 16:53 ./Masters/
-rwx------ scaredpoet/scaredpoet 82 2013-02-26 16:11 ./Masters/._eBethArke_Video_00001 copy.mp4
-rwx------ scaredpoet/scaredpoet 868118000 2013-02-26 16:11 ./Masters/eBethArke_Video_00001 copy.mp4
-rwx------ scaredpoet/scaredpoet 82 2013-02-26 16:11 ./Masters/._eBethArke_Video_00001.mp4
-rwx------ scaredpoet/scaredpoet 868118000 2013-02-26 16:11 ./Masters/eBethArke_Video_00001.mp4
-rwx------ scaredpoet/scaredpoet 314 2010-06-28 20:55 ./Masters/._eBethArke_Video_00001.wmv
-rwx------ scaredpoet/scaredpoet 241775085 2010-06-28 20:55 ./Masters/eBethArke_Video_00001.wmv

#13

I've checked all the objects from day 1 through day 9 (the last one run so far). I compared the checksums in Fedora with the outside list compiled by Isaiah. They all match except for one object: rutgers-lib:38490. We will need to compare the ARCH1 and RARCH1 for this video see what the problem might be. There was one other object (rutgers-lib:38498) where there were two different RARCHs for the same object. These are both in as ARCH1 versions and both on Isaiah's list. Their checksums match, but we need to check this object too to see which one is right. The others from days 1-9 are all matched and could have their RARCH datastreams removed to save space before the rest of the scripts are run and checked.

#14

Assigned to:dhoover» triggs

Isaiah should have permissions and should be the one to delete the RARCHs
that you have deemed can be removed.

Also in terms of 38498 check the rutgers-lib numbers you gave me in the day#.sh
scripts as I seem to remember there mat have been a duplicate in them. So a rutgers-lib
may have been processed twice.

#15

I will update when items are deleted and also report back as soon as I have determined disposition of failed object.

#16

Assigned to:triggs» dhoover

To do the whole set of RARCHs, I added a day15.sh script and updated scripts 10-14. I created a new readme and a new days.tar on /mellon/cvsroot reflecting this.
tar tvf days.tar
-rwxr-xr-x triggs/developers 588 2016-04-18 14:34 day10.sh
-rwxr-xr-x triggs/developers 624 2016-04-18 14:38 day11.sh
-rwxr-xr-x triggs/developers 732 2016-04-18 14:40 day12.sh
-rwxr-xr-x triggs/developers 1652 2016-04-18 14:44 day13.sh
-rwxr-xr-x triggs/developers 3128 2016-04-18 14:48 day14.sh
-rwxr-xr-x triggs/developers 4052 2016-04-18 14:48 day15.sh
-rwxr-xr-x triggs/developers 444 2016-03-22 14:09 day1.sh
-rwxr-xr-x triggs/developers 462 2016-03-22 14:10 day2.sh
-rwxr-xr-x triggs/developers 462 2016-03-22 14:10 day3.sh
-rwxr-xr-x triggs/developers 480 2016-03-22 14:11 day4.sh
-rwxr-xr-x triggs/developers 498 2016-03-22 14:12 day5.sh
-rwxr-xr-x triggs/developers 516 2016-03-22 14:13 day6.sh
-rwxr-xr-x triggs/developers 988 2016-03-22 14:19 day7.sh
-rwxr-xr-x triggs/developers 534 2016-03-22 14:20 day8.sh
-rwxr-xr-x triggs/developers 552 2016-03-22 14:21 day9.sh
For days 10 to 15, you should use the new scripts for in days.tar.

#17

I can delete the files in the filesystem, but as I'm verifying that objects have managed ARCH datastreams, I am seeing that the RARCH is still listed. My RUcore management account does not permit me to purge datastreams.

If someone else with that capability is willing to coordinate the purge of RARCHs in the object records with my removal of the files at the linux filesystem level, that's fine. Or if there is a willingness to permit my account to purge datastreams, I can purge them in dlr/EDIT and do the deletions.

Once we have a decision on this either way, I can proceed. I'm not comfortable with deleting the files while RUcore still thinks there should be RARCHs.

#18

We should note that purging RARCHs (control group "R") datastreams in Fedora will not purge the files, but simply remove the datastream links in the object.

#19

The following directories have been removed from the ./rarch/ directory, with the RARCH datastream also purged from the corresponding RUcore object:

rutgers-lib_34233 rutgers-lib_36338 rutgers-lib_36368 rutgers-lib_38538
rutgers-lib_34308 rutgers-lib_36348 rutgers-lib_36371 rutgers-lib_38539
rutgers-lib_34330 rutgers-lib_36350 rutgers-lib_36374 rutgers-lib_39090
rutgers-lib_35869 rutgers-lib_36352 rutgers-lib_36375 rutgers-lib_39091
rutgers-lib_36270 rutgers-lib_36354 rutgers-lib_36423 rutgers-lib_39092
rutgers-lib_36331 rutgers-lib_36358 rutgers-lib_36489 rutgers-lib_39093
rutgers-lib_36333 rutgers-lib_36360 rutgers-lib_38490 rutgers-lib_39095
rutgers-lib_36334 rutgers-lib_36361 rutgers-lib_38498 rutgers-lib_39097
rutgers-lib_36335 rutgers-lib_36362 rutgers-lib_38523 rutgers-lib_39232
rutgers-lib_36336 rutgers-lib_36364 rutgers-lib_38525
rutgers-lib_36337 rutgers-lib_36365 rutgers-lib_38536

1.9TB have been freed as a result.

#20

Assigned to:dhoover» ibeard

I checked the datastream sha256s for day10 and they all match the snapshot Isaiah took of the rarch files, so they are good to go as it were.

#21

Assigned to:ibeard» dhoover

Day 10 RARCH objects have been removed from the ./rarch/ directory:

41G ./rutgers-lib_36339
23G ./rutgers-lib_34327
21G ./rutgers-lib_36367
24G ./rutgers-lib_36899
23G ./rutgers-lib_34306
24G ./rutgers-lib_38534
24G ./rutgers-lib_36373
21G ./rutgers-lib_39094

196GB have been freed.

Ready for Day 11.

#22

Note; This is the previous report that did not get stored in this ticket.
No need for any further action.

day10.sh run on 2016-07-11 16:28:17:48

more /home/dhoover/day10.log
completed rutgers-lib:36373...
completed rutgers-lib:36899...
completed rutgers-lib:38534...
completed rutgers-lib:34327...
completed rutgers-lib:34306...
completed rutgers-lib:36339...
completed rutgers-lib:39094...
completed rutgers-lib:36367...
0.12user 0.55system 2:09:07elapsed 0%CPU (0avgtext+0avgdata 2884maxresident)k
11800inputs+64outputs (23major+13070minor)pagefaults 0swaps

#23

day11.sh run on 2016-07-13 18:02:17

more day11.log
completed rutgers-lib:38522...
completed rutgers-lib:36370...
completed rutgers-lib:38533...
completed rutgers-lib:34304...
completed rutgers-lib:39096...
completed rutgers-lib:34329...
completed rutgers-lib:36353...
completed rutgers-lib:37387...
completed rutgers-lib:39107...
completed rutgers-lib:34326...
completed rutgers-lib:36372...

#24

Assigned to:dhoover» ibeard

I checked day11 and all the checksums in Isaiah's list line up with those created upon ingest to Fedora, so it should be good to go. One object (37387) already had an ARCH1 datastream (a dummy tar placeholder). Thus the real tar datastream went in as ARCH1.1. After testing the new datastream I removed the ARCH1.0 dummy.

#25

Assigned to:ibeard» dhoover

Day 11 RARCH streams have been removed from the ./rarch/ directory:

16G ./rutgers-lib_36372
18G ./rutgers-lib_39096
18G ./rutgers-lib_37387
18G ./rutgers-lib_34329
16G ./rutgers-lib_34326
16G ./rutgers-lib_39107
20G ./rutgers-lib_38522
20G ./rutgers-lib_36370
20G ./rutgers-lib_38533
19G ./rutgers-lib_34304
18G ./rutgers-lib_36353

193GB have been freed.

Ready for Day 12.

#26

day12.sh run on 2016-07-18 17:07:46

Here is the log:

rep-prod:/home/RARCH # more day12.log
completed rutgers-lib:36359...
completed rutgers-lib:36349...
completed rutgers-lib:35756...
completed rutgers-lib:35744...
completed rutgers-lib:38521...
completed rutgers-lib:38535...
completed rutgers-lib:36340...
completed rutgers-lib:36351...
completed rutgers-lib:36363...
Object or selected datastream could not be found.
completed rutgers-lib:24938...
completed rutgers-lib:37386...
completed rutgers-lib:38524...
completed rutgers-lib:34305...
completed rutgers-lib:34303...
0.10user 0.52system 2:02:01elapsed 0%CPU (0avgtext+0avgdata 2892maxresident)k
7088inputs+104outputs (19major+22437minor)pagefaults 0swaps

#27

Assigned to:dhoover» ibeard

The day12 objects all check out and are good except for rutgers-lib:24938, which seems to have been purged from the repository. We might want to see what the RARCH was before deleting it, but if the object was one that should have been purged, that can probably be deleted.

#28

rutgers-lib:24938 is (or rather, was) an event recorded at Alexander Library on January 24, 2008. It was titled "There's More to New Jersey than the Sopranos," and was a lecture given by author Marc Mappen of the New Jersey Historical Society.

I am not certain why this object was purged, but we still have the original video in /rarch/. This will need to be investigated further.

This issue aside, the following rarch items were removed:

12G ./rutgers-lib_37386
14G ./rutgers-lib_35744
14G ./rutgers-lib_38535
27G ./rutgers-lib_36340
13G ./rutgers-lib_36363
9.4G ./rutgers-lib_34303
16G ./rutgers-lib_36359
11G ./rutgers-lib_38524
14G ./rutgers-lib_36351
29G ./rutgers-lib_36349
11G ./rutgers-lib_34305
14G ./rutgers-lib_38521
15G ./rutgers-lib_35756

193GB of space was freed. rutgers-lib:24938 remains in place pending resolution on whether it should be restored.

Ready for Day 13. I will inquire about rutgers-lib:24938.

#29

day13.sh run on 2016-07-29 17:37:49

Here is the log:

rep-prod:/home/RARCH # more day13.log
completed rutgers-lib:36369...
Object or selected datastream could not be found.
completed rutgers-lib:25995...
completed rutgers-lib:35747...
completed rutgers-lib:35749...
completed rutgers-lib:35751...
Object or selected datastream could not be found.
completed rutgers-lib:26019...
completed rutgers-lib:35753...
completed rutgers-lib:36332...
Object or selected datastream could not be found.
completed rutgers-lib:26042...
Object or selected datastream could not be found.
completed rutgers-lib:26046...
Object or selected datastream could not be found.
completed rutgers-lib:25996...
Object or selected datastream could not be found.
completed rutgers-lib:26004...
Object or selected datastream could not be found.
completed rutgers-lib:25985...
Object or selected datastream could not be found.
completed rutgers-lib:26001...
Object or selected datastream could not be found.
completed rutgers-lib:25997...
completed rutgers-lib:32846...
Object or selected datastream could not be found.
completed rutgers-lib:25990...
Object or selected datastream could not be found.
completed rutgers-lib:34504...
completed rutgers-lib:34514...
completed rutgers-lib:32850...
Object or selected datastream could not be found.
completed rutgers-lib:26017...
completed rutgers-lib:24290...
completed rutgers-lib:24291...
Object or selected datastream could not be found.
completed rutgers-lib:24836...
completed rutgers-lib:24156...
completed rutgers-lib:24147...
completed rutgers-lib:24151...
completed rutgers-lib:24181...
0.12user 0.32system 1:14:13elapsed 0%CPU (0avgtext+0avgdata 2892maxresident)k
7376inputs+120outputs (16major+44715minor)pagefaults 0swaps

#30

There were quite a few in this batch that have been purged from the repository. All the others had good checksums and can be considered migrated. The purged objects are listed below:
Object or selected datastream could not be found.
rutgers-lib:25995...
Object or selected datastream could not be found.
rutgers-lib:26019....
Object or selected datastream could not be found.
rutgers-lib:26042...
Object or selected datastream could not be found.
rutgers-lib:26046...
Object or selected datastream could not be found.
rutgers-lib:25996...
Object or selected datastream could not be found.
rutgers-lib:26004...
Object or selected datastream could not be found.
rutgers-lib:25985...
Object or selected datastream could not be found.
rutgers-lib:26001...
Object or selected datastream could not be found.
rutgers-lib:25997...
Object or selected datastream could not be found.
rutgers-lib:25990...
Object or selected datastream could not be found.
rutgers-lib:34504...
Object or selected datastream could not be found.
rutgers-lib:26017...
Object or selected datastream could not be found.
rutgers-lib:24836...

#31

Assigned to:ibeard» chadmills

The deleted objects in this batch all appear to be VMC-related objects. This is not unheard of however, as I know that some shorter clips were purged when the analytic capabilities came online in place for more comprehensive videos that could then be annotated into segments.

The exception is rutgers-lib:24836, which is a William Paterson University Video that was no doubt migrated to the NJVid platform.

I've consulted with Chad, and he will look at the purge logs and advise if this OK.

#32

Assigned to:chadmills» ibeard

----------------------
rutgers-lib:25995
----------------------
| Towers problem, grade 11, March 29, 1999, raw footage | 2009-12-15 | 10:32:15 | ingested | -UNKNOWN- |
| MovingImage | 2010-02-18 | 13:38:08 | edited | datacenter |
| MovingImage | 2010-02-22 | 01:01:40 | dsfail | !FEDORA! |
| MovingImage | 2010-03-01 | 00:52:15 | dsfail | !FEDORA! |
| Towers problem, grade 11, March 29, 1999, raw footage | 2010-08-16 | 09:53:41 | purged | sa |

----------------------
rutgers-lib:26019
----------------------
| Fractions problem, grade 4, September 27, 1993, raw footage | 2009-12-16 | 11:57:13 | ingested | Beard Isaiah |
| MovingImage | 2010-02-18 | 13:31:58 | edited | datacenter |
| MovingImage | 2010-02-22 | 01:03:06 | dsfail | !FEDORA! |
| MovingImage | 2010-03-01 | 00:53:42 | dsfail | !FEDORA! |
| Fractions problem, grade 4, September 27, 1993, raw footage | 2010-08-16 | 09:30:08 | purged | sa |

----------------------
rutgers-lib:26042
----------------------
| Fractions problem, grade 4, September 21, 1993, raw footage | 2009-12-16 | 12:47:37 | ingested | -UNKNOWN- |
| MovingImage | 2010-02-18 | 13:30:11 | edited | datacenter |
| MovingImage | 2010-02-22 | 01:03:29 | dsfail | !FEDORA! |
| MovingImage | 2010-03-01 | 00:54:05 | dsfail | !FEDORA! |
| Fractions problem, grade 4, September 21, 1993, raw footage | 2010-08-16 | 09:24:05 | purged | sa |

----------------------
rutgers-lib:26046
----------------------
| Fractions problem, grade 4, September 21, 1993, raw footage | 2009-12-16 | 14:02:59 | ingested | -UNKNOWN- |
| Fractions problem, grade 4, September 21, 1993, raw footage | 2009-12-16 | 15:46:02 | edited | sa |
| MovingImage | 2010-02-18 | 13:31:12 | edited | datacenter |
| MovingImage | 2010-02-22 | 01:03:37 | dsfail | !FEDORA! |
| MovingImage | 2010-03-01 | 00:54:13 | dsfail | !FEDORA! |
| Fractions problem, grade 4, September 21, 1993, raw footage | 2010-08-16 | 09:26:42 | purged | sa |

----------------------
rutgers-lib:25996
----------------------
| Towers problem, grade 11, March 29, 1999. raw footage | 2009-12-15 | 10:33:39 | ingested | Ananthan Kalaivani |
| MovingImage | 2010-02-18 | 13:38:51 | edited | datacenter |
| MovingImage | 2010-02-22 | 01:01:46 | dsfail | !FEDORA! |
| MovingImage | 2010-03-01 | 00:52:21 | dsfail | !FEDORA! |
| Towers problem, grade 11, March 29, 1999. raw footage | 2010-08-16 | 09:54:32 | purged | sa |

----------------------
rutgers-lib:26004
----------------------
| Taxicab problem, grade 12, March 5, 2000, raw footage | 2009-12-16 | 10:29:24 | ingested | Beard Isaiah |
| MovingImage | 2010-02-22 | 01:02:14 | dsfail | !FEDORA! |
| MovingImage | 2010-03-01 | 00:52:49 | dsfail | !FEDORA! |
| Taxicab problem, grade 12, March 5, 2000, raw footage | 2010-08-16 | 09:33:45 | purged | sa |

----------------------
rutgers-lib:25985
----------------------
| Taxicab problem, grade 12, March 5, 2000, raw footage | 2009-12-15 | 10:08:07 | ingested | -UNKNOWN- |
| MovingImage | 2010-02-18 | 13:36:04 | edited | datacenter |
| MovingImage | 2010-02-22 | 01:01:21 | dsfail | !FEDORA! |
| MovingImage | 2010-03-01 | 00:51:55 | dsfail | !FEDORA! |
| Taxicab problem, grade 12, March 5, 2000, raw footage | 2010-08-16 | 09:34:33 | purged | sa |

----------------------
rutgers-lib:26001
----------------------
| Towers problem, grade 10, January 9, 1998, raw footage | 2009-12-15 | 17:00:30 | ingested | -UNKNOWN- |
| MovingImage | 2010-02-22 | 01:01:58 | dsfail | !FEDORA! |
| MovingImage | 2010-03-01 | 00:52:33 | dsfail | !FEDORA! |
| Towers problem, grade 10, January 9, 1998, raw footage | 2010-08-16 | 09:38:38 | purged | sa |

----------------------
rutgers-lib:25997
----------------------
| Towers problem, grade 11, March 29, 1999, raw footage | 2009-12-15 | 10:34:55 | ingested | Ananthan Kalaivani |
| MovingImage | 2010-02-22 | 01:01:52 | dsfail | !FEDORA! |
| MovingImage | 2010-03-01 | 00:52:27 | dsfail | !FEDORA! |
| Towers problem, grade 11, March 29, 1999, raw footage | 2010-08-16 | 09:55:36 | purged | sa |

----------------------
rutgers-lib:25990
----------------------
| Towers problem, grade 11, March 29, 1999, raw footage | 2009-12-15 | 10:24:43 | ingested | -UNKNOWN- |
| MovingImage | 2010-02-22 | 01:01:28 | dsfail | !FEDORA! |
| MovingImage | 2010-03-01 | 00:52:01 | dsfail | !FEDORA! |
| Towers problem, grade 11, March 29, 1999, raw footage | 2010-08-16 | 09:59:20 | purged | sa |

----------------------
rutgers-lib:34504
----------------------
Nothing; not even an ingest is listed
mysql> select collection, date, time, action, user from notification_log where object = 'rutgers-lib:34504';
Empty set (0.02 sec)

----------------------
rutgers-lib:26017
----------------------
| Taxicab problem, grade 12, March 5, 2000, raw footage | 2009-12-16 | 11:46:52 | ingested | Beard Isaiah |
| MovingImage | 2010-02-22 | 01:02:53 | dsfail | !FEDORA! |
| MovingImage | 2010-03-01 | 00:53:29 | dsfail | !FEDORA! |
| Taxicab problem, grade 12, March 5, 2000, raw footage | 2010-08-16 | 09:36:58 | purged | sa |

#33

I remember Grace asking me to purge some of the VMC objects but can't remember why she wanted them purged. Since these objects were all purged on the same day, I am assuming they're from that batch she asked to purge.

#34

The following items were removed from the rarch directory:

6.0G ./rutgers-lib_32846
5.8G ./rutgers-lib_26001
6.6G ./rutgers-lib_24836
6.3G ./rutgers-lib_24156
8.7G ./rutgers-lib_36369
8.7G ./rutgers-lib_24290
5.6G ./rutgers-lib_24181
6.5G ./rutgers-lib_26046
7.5G ./rutgers-lib_26019
5.8G ./rutgers-lib_25985
6.2G ./rutgers-lib_25996
7.8G ./rutgers-lib_35747
5.8G ./rutgers-lib_34514
5.8G ./rutgers-lib_34504
7.6G ./rutgers-lib_35753
6.6G ./rutgers-lib_26042
6.0G ./rutgers-lib_24147
5.8G ./rutgers-lib_24151
5.8G ./rutgers-lib_25997
14G ./rutgers-lib_36332
5.9G ./rutgers-lib_26004
7.7G ./rutgers-lib_35751
7.7G ./rutgers-lib_35749
5.7G ./rutgers-lib_25990
5.6G ./rutgers-lib_32850
5.3G ./rutgers-lib_26017
7.9G ./rutgers-lib_24291
8.2G ./rutgers-lib_25995

192GB were freed.

Ready for the next run.

#35

day14.sh run on 2016-08-01 16:46:53

Object or selected datastream could not be found.
completed rutgers-lib:25991...
completed rutgers-lib:27065...
Object or selected datastream could not be found.
completed rutgers-lib:26005...
Object or selected datastream could not be found.
completed rutgers-lib:26002...
completed rutgers-lib:27067...
completed rutgers-lib:34501...
completed rutgers-lib:27071...
completed rutgers-lib:27068...
completed rutgers-lib:29527...
Object or selected datastream could not be found.
completed rutgers-lib:26007...
completed rutgers-lib:27069...
Object or selected datastream could not be found.
completed rutgers-lib:26003...
completed rutgers-lib:34500...
Object or selected datastream could not be found.
completed rutgers-lib:26018...
completed rutgers-lib:36949...
completed rutgers-lib:27066...
Object or selected datastream could not be found.
completed rutgers-lib:34510...
completed rutgers-lib:34512...
completed rutgers-lib:31241...
Object or selected datastream could not be found.
completed rutgers-lib:26008...
completed rutgers-lib:38537...
Object or selected datastream could not be found.
completed rutgers-lib:26020...
completed rutgers-lib:34502...
completed rutgers-lib:27063...
completed rutgers-lib:34513...
completed rutgers-lib:27062...
Object or selected datastream could not be found.
completed rutgers-lib:26047...
completed rutgers-lib:29522...
Object or selected datastream could not be found.
completed rutgers-lib:26006...
completed rutgers-lib:29516...
Object or selected datastream could not be found.
completed rutgers-lib:29515...
completed rutgers-lib:29514...
completed rutgers-lib:35049...
completed rutgers-lib:29525...
completed rutgers-lib:27070...
completed rutgers-lib:29293...
completed rutgers-lib:27064...
completed rutgers-lib:29295...
completed rutgers-lib:29294...
completed rutgers-lib:35757...
Object or selected datastream could not be found.
completed rutgers-lib:26010...
completed rutgers-lib:29296...
./day14.sh: line 10: syntax error near unexpected token `rutgers-lib:24153'
./day14.sh: line 10: `rutgers-lib:24153 rutgers-lib:24146 rutgers-lib:24155 rutgers-lib:24152 rutgers-lib:24154 rutgers-lib:24150 rutgers-lib:24180 rutgers-
lib:24157 rutgers-lib:24183 rutgers-lib:24158 '
Command exited with non-zero status 2
0.11user 0.40system 1:15:25elapsed 0%CPU (0avgtext+0avgdata 2892maxresident)k
7120inputs+240outputs (15major+66860minor)pagefaults 0swaps

#36

I think the above error is due to an errant line break in the second command line
in day14.sh

### special case for non standard filenames
for file in
rutgers-lib:24153 rutgers-lib:24146 rutgers-lib:24155 rutgers-lib:24152 rutgers- ....

There are also line break problems in the day15.sh script as well.

In addition when I was looking back through this issue (#16) I saw that not only
was a new day15.sh script given but that scripts 10-14 were also adjusted.
I missed this fact when I resumed running the scripts as the size of the tar
file did not change so I thought I had the newest version.

So at this point between day10-day15 scripts there are rutgers-lib numbers that have
not been processed at all as well as some that may have been in different scripts between
the first days.tar file and the second days.tar file that was put in /mellon/cvsroot on rep-devel

I would suggest that the current day14.log posted above be processed for the ones that
worked and that after all the old rarch/rutgers-lib* directories from day14 are removed that
a new list be generated for the ones that are still left.

#37

I'm not sure how all those line breaks got into day15.sh (looks like some copy-paste issue - certainly not the editor I always use, vi). In any event, I created a newdays.tar in /mellon/cvsroot with corrected day14 and day15. Isaiah and I have added some corrected anomalies from earlier runs to day15.sh.

#38

I downloaded the newdays.tar file so I could get the new day14 and day15 scripts.

-rw-r--r-- 1 triggs developers 30720 Aug 2 09:58 newdays.tar
drwxrwsr-x 154 root developers 29226 Aug 2 09:58 .
-rw-r--r-- 1 triggs developers 10240 Aug 2 09:57 days.tar

Note the days.tar file also has a 8/2/16 date but it is no longer a tar file?

dhoover@rep-devel:/mellon/cvsroot> file days.tar
days.tar: data

Since it is no longer what it should be it should probably be removed
from /mellon/cvsroot

#39

The new day14.sh script from newdays.tar was run 2016-08-09 22:32:40
on production.

I will wait for reports that the cleanup of the rarch directory has been done
for both day14 scripts that have been run and reported here before running
the last day15.sh script.

dhoover@rep-prod:/home/RARCH/3rd_tar> more day14_log.txt
completed rutgers-lib:34513...
completed rutgers-lib:27062...
Object or selected datastream could not be found.
completed rutgers-lib:26047...
completed rutgers-lib:29522...
Object or selected datastream could not be found.
completed rutgers-lib:26006...
completed rutgers-lib:29516...
Object or selected datastream could not be found.
completed rutgers-lib:29515...
completed rutgers-lib:29514...
completed rutgers-lib:35049...
completed rutgers-lib:29525...
completed rutgers-lib:27070...
completed rutgers-lib:29293...
completed rutgers-lib:27064...
completed rutgers-lib:29295...
completed rutgers-lib:29294...
completed rutgers-lib:35757...
Object or selected datastream could not be found.
completed rutgers-lib:26010...
completed rutgers-lib:29296...
completed rutgers-lib:31242...
completed rutgers-lib:34509...
completed rutgers-lib:39264...
completed rutgers-lib:32845...
completed rutgers-lib:29521...
completed rutgers-lib:35050...
completed rutgers-lib:29299...
completed rutgers-lib:32847...
completed rutgers-lib:34506...
completed rutgers-lib:29301...
completed rutgers-lib:29520...
completed rutgers-lib:29292...
completed rutgers-lib:29518...
completed rutgers-lib:39209...
completed rutgers-lib:38549...
completed rutgers-lib:32843...
completed rutgers-lib:31244...
completed rutgers-lib:31239...
Object or selected datastream could not be found.
completed rutgers-lib:26021...
completed rutgers-lib:39208...
completed rutgers-lib:29512...
Object or selected datastream could not be found.
completed rutgers-lib:26011...
completed rutgers-lib:29298...
completed rutgers-lib:29303...
completed rutgers-lib:32844...
completed rutgers-lib:29513...
completed rutgers-lib:38692...
completed rutgers-lib:39210...
completed rutgers-lib:29519...
completed rutgers-lib:37060...
completed rutgers-lib:37063...
completed rutgers-lib:37067...
completed rutgers-lib:34507...
completed rutgers-lib:38556...
completed rutgers-lib:38552...
completed rutgers-lib:37604...
completed rutgers-lib:38542...
completed rutgers-lib:39212...
completed rutgers-lib:38553...
completed rutgers-lib:35038...
completed rutgers-lib:38547...
completed rutgers-lib:27060...
completed rutgers-lib:35036...
completed rutgers-lib:37066...
completed rutgers-lib:31243...
completed rutgers-lib:37862...
completed rutgers-lib:39214...
completed rutgers-lib:38555...
completed rutgers-lib:35293...
completed rutgers-lib:38550...
completed rutgers-lib:38551...
completed rutgers-lib:29517...
completed rutgers-lib:37605...
completed rutgers-lib:39211...
completed rutgers-lib:37059...
completed rutgers-lib:37816...
completed rutgers-lib:37064...
completed rutgers-lib:38557...
completed rutgers-lib:39213...
completed rutgers-lib:38548...
completed rutgers-lib:29524...
completed rutgers-lib:29526...
completed rutgers-lib:29297...
completed rutgers-lib:37812...
completed rutgers-lib:37065...
completed rutgers-lib:38545...
completed rutgers-lib:37251...
completed rutgers-lib:29300...
completed rutgers-lib:38543...
completed rutgers-lib:37813...
completed rutgers-lib:38541...
completed rutgers-lib:38544...
completed rutgers-lib:37814...
completed rutgers-lib:37247...
completed rutgers-lib:34508...
completed rutgers-lib:37591...
completed rutgers-lib:37767...
completed rutgers-lib:37763...
completed rutgers-lib:37601...
completed rutgers-lib:37766...
completed rutgers-lib:37249...
completed rutgers-lib:37768...
completed rutgers-lib:37250...
completed rutgers-lib:29523...
completed rutgers-lib:38546...
completed rutgers-lib:36857...
completed rutgers-lib:37708...
completed rutgers-lib:37600...
completed rutgers-lib:38554...
completed rutgers-lib:36854...
completed rutgers-lib:37245...
Object or selected datastream could not be found.
completed rutgers-lib:37246...
completed rutgers-lib:37817...
completed rutgers-lib:31240...
completed rutgers-lib:37244...
completed rutgers-lib:37713...
completed rutgers-lib:37602...
completed rutgers-lib:37593...
completed rutgers-lib:37242...
completed rutgers-lib:24153...
completed rutgers-lib:24146...
completed rutgers-lib:24155...
completed rutgers-lib:24152...
completed rutgers-lib:24154...
completed rutgers-lib:24150...
completed rutgers-lib:24180...
completed rutgers-lib:24157...
completed rutgers-lib:24183...
completed rutgers-lib:24158...
0.31user 0.89system 2:34:02elapsed 0%CPU (0avgtext+0avgdata 2896maxresident)k
12136inputs+896outputs (26major+202259minor)pagefaults 0swaps
dhoover@rep-prod:/home/RARCH/3rd_tar>

#40

This seems to have been a good run all in all. I tested all the objects and found 120 where the rarch went in and the checksums were the same and 7 where the objects had been purged. A handful of duplicates at the beginning were dedup'ed. I'm attaching lists of the good objects and the purged objects for Isaiah.

#41

The following items were removed. 248GB of disk space were freed. Ready for Day 15.

1.5G ./rutgers-lib_38555
713M ./rutgers-lib_38554
670M ./rutgers-lib_37244
3.3G ./rutgers-lib_29522
3.2G ./rutgers-lib_29516
2.8G ./rutgers-lib_24158
2.8G ./rutgers-lib_27060
1.2G ./rutgers-lib_39213
3.0G ./rutgers-lib_24180
2.6G ./rutgers-lib_31242
1004M ./rutgers-lib_37814
842M ./rutgers-lib_37768
785M ./rutgers-lib_36854
2.0G ./rutgers-lib_31239
778M ./rutgers-lib_37245
1.1G ./rutgers-lib_37813
965M ./rutgers-lib_37247
2.5G ./rutgers-lib_26010
1.6G ./rutgers-lib_38547
2.9G ./rutgers-lib_29525
2.0G ./rutgers-lib_39208
3.3G ./rutgers-lib_37067
2.1G ./rutgers-lib_32843
1.3G ./rutgers-lib_29517
1.7G ./rutgers-lib_38553
3.1G ./rutgers-lib_24146
6.6G ./rutgers-lib_27062
859M ./rutgers-lib_34508
3.4G ./rutgers-lib_38692
5.0G ./rutgers-lib_37862
5.6G ./rutgers-lib_27070
844M ./rutgers-lib_37249
697M ./rutgers-lib_29523
3.0G ./rutgers-lib_29293
1.7G ./rutgers-lib_38542
1.2G ./rutgers-lib_37812
2.2G ./rutgers-lib_29518
1.1G ./rutgers-lib_29297
3.0G ./rutgers-lib_24157
1.1G ./rutgers-lib_29524
2.9G ./rutgers-lib_29295
3.1G ./rutgers-lib_35049
2.1G ./rutgers-lib_31244
1.9G ./rutgers-lib_32844
959M ./rutgers-lib_37763
764M ./rutgers-lib_38546
1.6G ./rutgers-lib_35038
1.4G ./rutgers-lib_38551
2.3G ./rutgers-lib_37064
1.4G ./rutgers-lib_38550
731M ./rutgers-lib_37817
2.3G ./rutgers-lib_32847
2.6G ./rutgers-lib_29296
2.2G ./rutgers-lib_29520
2.9G ./rutgers-lib_29294
3.5G ./rutgers-lib_34513
1.5G ./rutgers-lib_31243
808M ./rutgers-lib_36857
1.9G ./rutgers-lib_26011
2.4G ./rutgers-lib_29301
790M ./rutgers-lib_37708
3.2G ./rutgers-lib_29515
2.4G ./rutgers-lib_29521
614M ./rutgers-lib_37242
1.9G ./rutgers-lib_26021
1.7G ./rutgers-lib_38552
1.9G ./rutgers-lib_29298
643M ./rutgers-lib_31240
979M ./rutgers-lib_29300
1.1G ./rutgers-lib_38541
1.7G ./rutgers-lib_38556
906M ./rutgers-lib_37601
2.5G ./rutgers-lib_32845
812M ./rutgers-lib_37250
3.4G ./rutgers-lib_24153
864M ./rutgers-lib_37591
1.9G ./rutgers-lib_39210
1.8G ./rutgers-lib_29519
1.6G ./rutgers-lib_37604
2.4G ./rutgers-lib_29299
2.0G ./rutgers-lib_29303
1.7G ./rutgers-lib_37063
975M ./rutgers-lib_37767
2.4G ./rutgers-lib_37059
5.5G ./rutgers-lib_27064
2.4G ./rutgers-lib_35293
3.1G ./rutgers-lib_24155
3.3G ./rutgers-lib_26047
1.1G ./rutgers-lib_37251
1.4G ./rutgers-lib_37816
1.2G ./rutgers-lib_38548
1.9G ./rutgers-lib_29513
2.1G ./rutgers-lib_37065
584M ./rutgers-lib_37602
1.1G ./rutgers-lib_29526
2.3G ./rutgers-lib_34506
3.0G ./rutgers-lib_24152
1.7G ./rutgers-lib_34507
2.4G ./rutgers-lib_35050
688M ./rutgers-lib_37713
3.0G ./rutgers-lib_24150
2.6G ./rutgers-lib_34509
1.2G ./rutgers-lib_38557
2.2G ./rutgers-lib_38549
1.3G ./rutgers-lib_39211
1.3G ./rutgers-lib_37605
3.4G ./rutgers-lib_37060
2.1G ./rutgers-lib_29512
1023M ./rutgers-lib_38544
1.6G ./rutgers-lib_39212
2.1G ./rutgers-lib_39209
3.1G ./rutgers-lib_26006
1.5G ./rutgers-lib_35036
3.2G ./rutgers-lib_29514
2.2G ./rutgers-lib_29292
763M ./rutgers-lib_37600
2.8G ./rutgers-lib_37066
710M ./rutgers-lib_37246
874M ./rutgers-lib_37766
2.6G ./rutgers-lib_35757
3.0G ./rutgers-lib_24154
637M ./rutgers-lib_37593
1.4G ./rutgers-lib_39214
3.3G ./rutgers-lib_39264
2.9G ./rutgers-lib_24183
1.2G ./rutgers-lib_38545
1.1G ./rutgers-lib_38543

#42

day15.sh was run on 8/19/2016 at 21:54:50

The results are:

completed rutgers-lib:29302...
completed rutgers-lib:37762...
completed rutgers-lib:36855...
completed rutgers-lib:37248...
completed rutgers-lib:37764...
completed rutgers-lib:36853...
completed rutgers-lib:37594...
completed rutgers-lib:37599...
completed rutgers-lib:37596...
completed rutgers-lib:37590...
completed rutgers-lib:37815...
completed rutgers-lib:37710...
completed rutgers-lib:38540...
completed rutgers-lib:37595...
completed rutgers-lib:37811...
completed rutgers-lib:37243...
completed rutgers-lib:37709...
completed rutgers-lib:37761...
completed rutgers-lib:37701...
completed rutgers-lib:37818...
completed rutgers-lib:36851...
completed rutgers-lib:37597...
completed rutgers-lib:32848...
completed rutgers-lib:36858...
completed rutgers-lib:37589...
completed rutgers-lib:37598...
completed rutgers-lib:36850...
completed rutgers-lib:37588...
completed rutgers-lib:35037...
completed rutgers-lib:37707...
completed rutgers-lib:37711...
completed rutgers-lib:37650...
completed rutgers-lib:37810...
completed rutgers-lib:37717...
completed rutgers-lib:37718...
completed rutgers-lib:37204...
completed rutgers-lib:36856...
completed rutgers-lib:37189...
completed rutgers-lib:37714...
completed rutgers-lib:37716...
completed rutgers-lib:32849...
completed rutgers-lib:36852...
completed rutgers-lib:37726...
completed rutgers-lib:37705...
completed rutgers-lib:37592...
completed rutgers-lib:37209...
completed rutgers-lib:37190...
completed rutgers-lib:37347...
completed rutgers-lib:37187...
completed rutgers-lib:37191...
completed rutgers-lib:37747...
completed rutgers-lib:37192...
completed rutgers-lib:37185...
completed rutgers-lib:37704...
completed rutgers-lib:37745...
completed rutgers-lib:37184...
completed rutgers-lib:37668...
completed rutgers-lib:37746...
completed rutgers-lib:37673...
completed rutgers-lib:37743...
completed rutgers-lib:37188...
completed rutgers-lib:37728...
completed rutgers-lib:37212...
completed rutgers-lib:37648...
completed rutgers-lib:37186...
completed rutgers-lib:37183...
completed rutgers-lib:37729...
completed rutgers-lib:37727...
completed rutgers-lib:37207...
completed rutgers-lib:27061...
completed rutgers-lib:37211...
completed rutgers-lib:37725...
completed rutgers-lib:37715...
completed rutgers-lib:37744...
completed rutgers-lib:37214...
completed rutgers-lib:37603...
completed rutgers-lib:37213...
completed rutgers-lib:37205...
completed rutgers-lib:37193...
completed rutgers-lib:37210...
completed rutgers-lib:27534...
completed rutgers-lib:37696...
completed rutgers-lib:37670...
completed rutgers-lib:37208...
completed rutgers-lib:37748...
completed rutgers-lib:27530...
completed rutgers-lib:37721...
completed rutgers-lib:27537...
completed rutgers-lib:37206...
completed rutgers-lib:37388...
completed rutgers-lib:28131...
completed rutgers-lib:27525...
completed rutgers-lib:37720...
completed rutgers-lib:27533...
completed rutgers-lib:37346...
completed rutgers-lib:37742...
completed rutgers-lib:37723...
completed rutgers-lib:37722...
completed rutgers-lib:27454...
completed rutgers-lib:27520...
completed rutgers-lib:37698...
completed rutgers-lib:27527...
completed rutgers-lib:37675...
completed rutgers-lib:37202...
completed rutgers-lib:37700...
completed rutgers-lib:27518...
completed rutgers-lib:37712...
completed rutgers-lib:27540...
completed rutgers-lib:27541...
completed rutgers-lib:37663...
completed rutgers-lib:37678...
completed rutgers-lib:27517...
completed rutgers-lib:37703...
completed rutgers-lib:28128...
completed rutgers-lib:27538...
completed rutgers-lib:37664...
completed rutgers-lib:27519...
completed rutgers-lib:37724...
completed rutgers-lib:37349...
completed rutgers-lib:37677...
completed rutgers-lib:37740...
completed rutgers-lib:37649...
completed rutgers-lib:27532...
completed rutgers-lib:37732...
completed rutgers-lib:27528...
completed rutgers-lib:37706...
completed rutgers-lib:27524...
completed rutgers-lib:37737...
completed rutgers-lib:37672...
completed rutgers-lib:27531...
completed rutgers-lib:37665...
completed rutgers-lib:27536...
completed rutgers-lib:37702...
completed rutgers-lib:37651...
completed rutgers-lib:37699...
completed rutgers-lib:37734...
completed rutgers-lib:37361...
completed rutgers-lib:37198...
completed rutgers-lib:27535...
completed rutgers-lib:37730...
completed rutgers-lib:37733...
completed rutgers-lib:37731...
completed rutgers-lib:37676...
completed rutgers-lib:27557...
completed rutgers-lib:27555...
completed rutgers-lib:28125...
completed rutgers-lib:37735...
completed rutgers-lib:37357...
completed rutgers-lib:37736...
completed rutgers-lib:37200...
completed rutgers-lib:37697...
completed rutgers-lib:37345...
completed rutgers-lib:37695...
completed rutgers-lib:37671...
completed rutgers-lib:27549...
completed rutgers-lib:28126...
completed rutgers-lib:37669...
completed rutgers-lib:37194...
completed rutgers-lib:27539...
completed rutgers-lib:37739...
completed rutgers-lib:37647...
completed rutgers-lib:37199...
completed rutgers-lib:37652...
completed rutgers-lib:37738...
completed rutgers-lib:27529...
completed rutgers-lib:37362...
completed rutgers-lib:37196...
completed rutgers-lib:27556...
completed rutgers-lib:37203...
completed rutgers-lib:28129...
completed rutgers-lib:28130...
completed rutgers-lib:28127...
completed rutgers-lib:37197...
completed rutgers-lib:37390...
completed rutgers-lib:27526...
completed rutgers-lib:27547...
completed rutgers-lib:37646...
completed rutgers-lib:37674...
completed rutgers-lib:27553...
completed rutgers-lib:37348...
completed rutgers-lib:27552...
completed rutgers-lib:37741...
completed rutgers-lib:37667...
completed rutgers-lib:27522...
completed rutgers-lib:37391...
completed rutgers-lib:27546...
completed rutgers-lib:27550...
completed rutgers-lib:27551...
completed rutgers-lib:27523...
completed rutgers-lib:37666...
completed rutgers-lib:27542...
completed rutgers-lib:27545...
completed rutgers-lib:27521...
completed rutgers-lib:27548...
completed rutgers-lib:37201...
completed rutgers-lib:27543...
completed rutgers-lib:27544...
completed rutgers-lib:37389...
completed rutgers-lib:37195...
Object or selected datastream could not be found.
completed rutgers-lib:24917...
Object or selected datastream could not be found.
completed rutgers-lib:27361...
Object or selected datastream could not be found.
completed rutgers-lib:34498...
completed rutgers-lib:36366...
completed rutgers-lib:38490...
0.51user 0.65system 1:06:37elapsed 0%CPU (0avgtext+0avgdata 2892maxresident)k
5064inputs+256outputs (5major+325977minor)pagefaults 0swaps

#43

I think we're good to go Isaiah. I tested day15, and all the ARCH signatures check out except for the three objects which have been purged:
triggs@rep-devel:/mellon/htdocs/dlr/EDIT> diff day15sigs.txt day15rarchssigs.txt
200,202c200,202
<
<
<
---
> 4566dc8d9d79b90c8f15793c72ce9ad76fb761b55a5479f3d35587bbc6ad9663 rutgers-lib:24917
> 4566dc8d9d79b90c8f15793c72ce9ad76fb761b55a5479f3d35587bbc6ad9663 rutgers-lib:27361
> 4566dc8d9d79b90c8f15793c72ce9ad76fb761b55a5479f3d35587bbc6ad9663 rutgers-lib:34498

And yes, I did a double take, but these three purged objects all show the same sha256 checksum in Isaiah's spreadsheet.

#44

day15.sh now leaves us with the following objects in ./rarch/:

rutgers-lib_24149 rutgers-lib_24208 rutgers-lib_24937 rutgers-lib_26008 rutgers-lib_27068 rutgers-lib_34501
rutgers-lib_24201 rutgers-lib_24209 rutgers-lib_24938 rutgers-lib_26018 rutgers-lib_27069 rutgers-lib_34502
rutgers-lib_24202 rutgers-lib_24845 rutgers-lib_25991 rutgers-lib_26020 rutgers-lib_27071 rutgers-lib_34510
rutgers-lib_24203 rutgers-lib_24846 rutgers-lib_26002 rutgers-lib_27063 rutgers-lib_29527 rutgers-lib_34512
rutgers-lib_24204 rutgers-lib_24847 rutgers-lib_26003 rutgers-lib_27065 rutgers-lib_31241 rutgers-lib_36949
rutgers-lib_24206 rutgers-lib_24848 rutgers-lib_26005 rutgers-lib_27066 rutgers-lib_33945 rutgers-lib_38537
rutgers-lib_24207 rutgers-lib_24916 rutgers-lib_26007 rutgers-lib_27067 rutgers-lib_34500

The following RARCH directories were removed. 102GB were freed.

372M ./rutgers-lib_37184
37M ./rutgers-lib_28129
213M ./rutgers-lib_37729
261M ./rutgers-lib_37205
433M ./rutgers-lib_37191
145M ./rutgers-lib_37388
12M ./rutgers-lib_27548
206M ./rutgers-lib_37206
237M ./rutgers-lib_37746
674M ./rutgers-lib_37701
129M ./rutgers-lib_37720
24M ./rutgers-lib_37390
458M ./rutgers-lib_37190
134M ./rutgers-lib_37346
638M ./rutgers-lib_37204
79M ./rutgers-lib_27538
335M ./rutgers-lib_36850
21M ./rutgers-lib_37391
231M ./rutgers-lib_37728
66M ./rutgers-lib_37699
45M ./rutgers-lib_28126
90M ./rutgers-lib_27517
547M ./rutgers-lib_37590
446M ./rutgers-lib_37595
600M ./rutgers-lib_36855
32M ./rutgers-lib_37738
145M ./rutgers-lib_27530
80M ./rutgers-lib_37664
296M ./rutgers-lib_24917
160M ./rutgers-lib_37670
93M ./rutgers-lib_27541
155M ./rutgers-lib_37202
22M ./rutgers-lib_27522
71M ./rutgers-lib_37706
553M ./rutgers-lib_29302
569M ./rutgers-lib_37248
158M ./rutgers-lib_37715
19M ./rutgers-lib_27550
76M ./rutgers-lib_27532
14M ./rutgers-lib_27545
27M ./rutgers-lib_37646
547M ./rutgers-lib_37599
60M ./rutgers-lib_27535
67M ./rutgers-lib_37702
35M ./rutgers-lib_27539
91M ./rutgers-lib_37663
66M ./rutgers-lib_27536
295M ./rutgers-lib_35037
48M ./rutgers-lib_37357
456M ./rutgers-lib_32848
334M ./rutgers-lib_37705
54M ./rutgers-lib_37731
15M ./rutgers-lib_37201
23M ./rutgers-lib_37348
81M ./rutgers-lib_37703
418M ./rutgers-lib_37818
309M ./rutgers-lib_37711
76M ./rutgers-lib_27519
339M ./rutgers-lib_37212
121M ./rutgers-lib_27533
35M ./rutgers-lib_37697
17M ./rutgers-lib_27523
271M ./rutgers-lib_37592
338M ./rutgers-lib_37726
31M ./rutgers-lib_37647
31M ./rutgers-lib_27556
392M ./rutgers-lib_37192
134M ./rutgers-lib_27537
315M ./rutgers-lib_37207
129M ./rutgers-lib_37696
296M ./rutgers-lib_27361
57M ./rutgers-lib_37676
305M ./rutgers-lib_27061
73M ./rutgers-lib_27528
97M ./rutgers-lib_27540
72M ./rutgers-lib_37732
57M ./rutgers-lib_37733
5.7M ./rutgers-lib_37389
433M ./rutgers-lib_37187
326M ./rutgers-lib_37347
482M ./rutgers-lib_37650
33M ./rutgers-lib_28127
334M ./rutgers-lib_37186
260M ./rutgers-lib_37213
114M ./rutgers-lib_37723
18M ./rutgers-lib_27551
70M ./rutgers-lib_37740
192M ./rutgers-lib_37725
485M ./rutgers-lib_32849
46M ./rutgers-lib_37735
23M ./rutgers-lib_27553
514M ./rutgers-lib_37710
41M ./rutgers-lib_37345
21M ./rutgers-lib_37667
63M ./rutgers-lib_37734
314M ./rutgers-lib_37716
345M ./rutgers-lib_37588
588M ./rutgers-lib_37189
123M ./rutgers-lib_37742
352M ./rutgers-lib_37188
141M ./rutgers-lib_37748
52M ./rutgers-lib_37194
185M ./rutgers-lib_37744
253M ./rutgers-lib_37673
19M ./rutgers-lib_27546
64M ./rutgers-lib_37651
527M ./rutgers-lib_37596
37M ./rutgers-lib_37669
68M ./rutgers-lib_37672
95M ./rutgers-lib_27518
94M ./rutgers-lib_37712
524M ./rutgers-lib_37815
43M ./rutgers-lib_37736
348M ./rutgers-lib_37598
38M ./rutgers-lib_37197
271M ./rutgers-lib_37214
66M ./rutgers-lib_27531
597M ./rutgers-lib_36853
252M ./rutgers-lib_37193
34G ./rutgers-lib_38490
144M ./rutgers-lib_37721
37M ./rutgers-lib_28130
258M ./rutgers-lib_37668
106M ./rutgers-lib_27527
454M ./rutgers-lib_37709
283M ./rutgers-lib_37810
239M ./rutgers-lib_36852
7.5M ./rutgers-lib_37195
8.0M ./rutgers-lib_27543
66M ./rutgers-lib_37737
62M ./rutgers-lib_37200
76M ./rutgers-lib_37649
222M ./rutgers-lib_37208
49M ./rutgers-lib_37199
46M ./rutgers-lib_37196
448M ./rutgers-lib_37811
233M ./rutgers-lib_37704
31G ./rutgers-lib_36366
73M ./rutgers-lib_37724
22M ./rutgers-lib_27552
92M ./rutgers-lib_37198
452M ./rutgers-lib_37761
51M ./rutgers-lib_27555
130M ./rutgers-lib_27525
582M ./rutgers-lib_37764
209M ./rutgers-lib_37727
268M ./rutgers-lib_36856
66M ./rutgers-lib_37665
38M ./rutgers-lib_37671
77M ./rutgers-lib_37677
354M ./rutgers-lib_37718
29M ./rutgers-lib_37362
7.3M ./rutgers-lib_27544
110M ./rutgers-lib_27520
21M ./rutgers-lib_37741
443M ./rutgers-lib_37243
37M ./rutgers-lib_27549
33M ./rutgers-lib_37695
51M ./rutgers-lib_27557
387M ./rutgers-lib_37185
365M ./rutgers-lib_37589
556M ./rutgers-lib_37594
248M ./rutgers-lib_37210
157M ./rutgers-lib_27534
16M ./rutgers-lib_27542
223M ./rutgers-lib_37648
104M ./rutgers-lib_28128
213M ./rutgers-lib_37603
58M ./rutgers-lib_37730
324M ./rutgers-lib_37714
26M ./rutgers-lib_27547
420M ./rutgers-lib_37597
79M ./rutgers-lib_37349
301M ./rutgers-lib_37211
474M ./rutgers-lib_37209
32M ./rutgers-lib_37739
91M ./rutgers-lib_37678
609M ./rutgers-lib_37762
476M ./rutgers-lib_38540
111M ./rutgers-lib_27454
229M ./rutgers-lib_37743
44M ./rutgers-lib_37203
323M ./rutgers-lib_37707
236M ./rutgers-lib_37745
62M ./rutgers-lib_28125
27M ./rutgers-lib_27526
13M ./rutgers-lib_27521
64M ./rutgers-lib_37361
91M ./rutgers-lib_37698
325M ./rutgers-lib_37183
16M ./rutgers-lib_37666
113M ./rutgers-lib_37675
296M ./rutgers-lib_34498
402M ./rutgers-lib_36858
255M ./rutgers-lib_37747
169M ./rutgers-lib_28131
380M ./rutgers-lib_37717
24M ./rutgers-lib_37674
111M ./rutgers-lib_37722
72M ./rutgers-lib_27524
101M ./rutgers-lib_37700
32M ./rutgers-lib_37652
32M ./rutgers-lib_27529
447M ./rutgers-lib_36851

#45

OK. There is a standalone, clean up script, day16.sh, in /mellon/cvsroot to take care of these last bits.

#46

day16.sh run on rep-prod on 9/1/16 18:00

rep-prod:/home/RARCH/3rd_tar # more day16_log.txt
completed rutgers-lib:24149...
completed rutgers-lib:24208...
Object or selected datastream could not be found.
completed rutgers-lib:24937...
Object or selected datastream could not be found.
completed rutgers-lib:26008...
completed rutgers-lib:27068...
completed rutgers-lib:34501...
completed rutgers-lib:24201...
completed rutgers-lib:24209...
Object or selected datastream could not be found.
completed rutgers-lib:24938...
Object or selected datastream could not be found.
completed rutgers-lib:26018...
completed rutgers-lib:27069...
completed rutgers-lib:34502...
completed rutgers-lib:24202...
Object or selected datastream could not be found.
completed rutgers-lib:24845...
Object or selected datastream could not be found.
completed rutgers-lib:25991...
Object or selected datastream could not be found.
completed rutgers-lib:26020...
completed rutgers-lib:27071...
Object or selected datastream could not be found.
completed rutgers-lib:34510...
completed rutgers-lib:24203...
Object or selected datastream could not be found.
completed rutgers-lib:24846...
Object or selected datastream could not be found.
completed rutgers-lib:26002...
completed rutgers-lib:27063...
completed rutgers-lib:29527...
completed rutgers-lib:34512...
completed rutgers-lib:24204...
Object or selected datastream could not be found.
completed rutgers-lib:24847...
Object or selected datastream could not be found.
completed rutgers-lib:26003...
completed rutgers-lib:27065...
completed rutgers-lib:31241...
completed rutgers-lib:36949...
completed rutgers-lib:24206...
Object or selected datastream could not be found.
completed rutgers-lib:24848...
Object or selected datastream could not be found.
completed rutgers-lib:26005...
completed rutgers-lib:27066...
Object or selected datastream could not be found.
completed rutgers-lib:33945...
completed rutgers-lib:38537...
completed rutgers-lib:24207...
Object or selected datastream could not be found.
completed rutgers-lib:24916...
Object or selected datastream could not be found.
completed rutgers-lib:26007...
completed rutgers-lib:27067...
completed rutgers-lib:34500...
0.12user 0.24system 46:41.35elapsed 0%CPU (0avgtext+0avgdata 2896maxresident)k
2536inputs+128outputs (8major+65382minor)pagefaults 0swaps

#47

Assigned to:ibeard» dhoover

I checked the day16 objects. I'm attaching a screenshot of the diff showing the signatures of the new ARCH1 in Fedora in the left pane and the signatures from Isaiah's spreadsheet in the right pane. The empty lines in the left pane represent objects that no longer exist in the repository. The lines that are the same in each pane are objects where the RARCH was ingested properly. All were either good or purged with one anomaly, rutgers-lib:24201, which I was unable to redo by hand (it may be too large). I'm asking Dave to run one more command for this last one:
lynx -source "http://mss3.libraries.rutgers.edu/dlr/EDIT/INT/getfedorarest.php?pid=rutgers-lib:24201&altid=&label=rarch+conversion&mimetype=application/x-tar&dslocation=http://mss3.libraries.rutgers.edu/rarch/rutgers-lib_24201/rutgers-lib_24201.tar&logmessage=Triggs+rarch+conversion&dsid=ARCH1&controlgroup=M&checksumtype=SHA-256&view=addds"
Once it is in, Isaiah can go ahead and remove the RARCHs for the good objects and purged ones.

#48

Above command for rutgers-lib_24201 was run.
The was no output generated to screen.

This was created:

dhoover@rep-prod:/repository/datastreams/2016/0912/12/03> ls -al /repository/datastreams/2016/0912/12/03/
total 1575766
drwxr-xr-x 2 nobody nobody 49 Sep 12 12:03 .
drwxr-xr-x 3 nobody nobody 20 Sep 12 12:03 ..
-rw-r--r-- 1 nobody nobody 1286993920 Sep 12 12:04 rutgers-lib_24201+ARCH1+ARCH1.1

Dave

#49

That one was so big that it maxed out the exif, but the Fedora dsprofile with SHA-256 shows that it went in OK and matches the RARCH signature in Isaiah's spreadsheet. That should be it then. Thanks Dave!

#50

Status:active» closed

The following objects were removed from the ./rarch directory:

4.3G ./rutgers-lib_26007
12G ./rutgers-lib_24937
4.2G ./rutgers-lib_26003
4.2G ./rutgers-lib_34500
25G ./rutgers-lib_33945
34M ./rutgers-lib_24847
7.6G ./rutgers-lib_27066
4.0G ./rutgers-lib_34510
1.6G ./rutgers-lib_24201
4.4G ./rutgers-lib_29527
8.2G ./rutgers-lib_27069
5.2G ./rutgers-lib_25991
5.5M ./rutgers-lib_24204
4.0G ./rutgers-lib_36949
4.0G ./rutgers-lib_34512
15M ./rutgers-lib_24207
5.5M ./rutgers-lib_24209
8.0M ./rutgers-lib_24208
3.8G ./rutgers-lib_26008
7.0G ./rutgers-lib_27063
8.8M ./rutgers-lib_24202
9.4G ./rutgers-lib_27067
4.0G ./rutgers-lib_26018
4.9G ./rutgers-lib_34501
3.8G ./rutgers-lib_38537
9.0G ./rutgers-lib_27071
26M ./rutgers-lib_24203
73M ./rutgers-lib_24845
3.8G ./rutgers-lib_34502
30M ./rutgers-lib_24848
118M ./rutgers-lib_24846
5.0G ./rutgers-lib_26002
12G ./rutgers-lib_24938
3.7G ./rutgers-lib_26020
10G ./rutgers-lib_27065
296M ./rutgers-lib_24916
9.0G ./rutgers-lib_27068
1.8M ./rutgers-lib_24206
5.1G ./rutgers-lib_26005
4.0G ./rutgers-lib_31241
175M ./rutgers-lib_24149

181GB of space was freed.

There are no more RARCH objects remaining. We're done!

Back to top