Purge all MOV-* and FLV-* datastreams

Project:RUcore Jobs & Reports
Component:Job - development
Category:task
Priority:normal
Assigned:dhoover
Status:Moved to JIRA
Description

Migration to MP4 as the presentation datastream is complete. Please write a script to delete all FLV-* and MOV-* datastreams from dev, test, staging, and production.

Thanks!

Comments

#1

OK. This should be fun.

#2

Assigned to:triggs» dhoover

I have removed all the datastreams from dev and test and have a list ready for prod (but no list for staging yet). I used Chad's script to get the DS list for rep and test and the web page display to get the list for production. Is there one for staging too that I could use?

Here are the results for dev and test.
rep-dev:
triggs@rep-devel2:/mellon/htdocs/dlr/EDIT> cat /home/triggs/zaparchtars.sh
#!/bin/bash
source /mellon/includes/incs.sh
for line in `$CAT $1`
do
$LYNX -source "$line"
done
triggs@rep-devel2:/mellon/htdocs/dlr/EDIT> df
Filesystem 1K-blocks Used Available Use% Mounted on
/dev/sda2 50094804 35889380 13136636 74% /
udev 3030028 112 3029916 1% /dev
tmpfs 3030028 736 3029292 1% /dev/shm
/dev/mapper/lv1-lv1 206420664 75608564 120326548 39% /mellon
triggs@rep-devel2:/mellon/htdocs/dlr/EDIT> time /home/triggs/zaparchtars.sh repdev-vid-purge.txt
Object or selected datastream could not be found.
Object or selected datastream could not be found.

real 0m3.577s
user 0m0.028s
sys 0m0.020s
triggs@rep-devel2:/mellon/htdocs/dlr/EDIT> df
Filesystem 1K-blocks Used Available Use% Mounted on
/dev/sda2 50094804 35889424 13136592 74% /
udev 3030028 112 3029916 1% /dev
tmpfs 3030028 736 3029292 1% /dev/shm
/dev/mapper/lv1-lv1 206420664 73621364 122313748 38% /mellon
triggs@rep-devel2:/mellon/htdocs/dlr/EDIT> wc repdev-vid-purge.txt
24 24 2644 repdev-vid-purge.txt

rep-test:
triggs@rep-devel:/mellon/htdocs/dlr/EDIT> df
Filesystem 1K-blocks Used Available Use% Mounted on
/dev/cciss/c0d0p2 139010444 115781128 16167948 88% /
udev 16434672 196 16434476 1% /dev
tmpfs 16434672 1088 16433584 1% /dev/shm
/dev/cciss/c0d1p1 206422068 96423700 99512744 50% /home
/dev/cciss/c0d1p2 370402152 139316956 212269868 40% /mellon
isilon2:/ifs/data/rep-devel 58053531008 43091380032 12870876448 78% /rep-devel
isilon0:/ifs/data/SLES 58053531136 43091379712 12870876672 78% /SLES
triggs@rep-devel:/mellon/htdocs/dlr/EDIT> time /home/triggs/zaparchtars.sh reptest-vid-purge.txt
Object or selected datastream could not be found.

real 0m25.485s
user 0m0.640s
sys 0m0.732s
triggs@rep-devel:/mellon/htdocs/dlr/EDIT> df
Filesystem 1K-blocks Used Available Use% Mounted on
/dev/cciss/c0d0p2 139010444 115782612 16166464 88% /
udev 16434672 196 16434476 1% /dev
tmpfs 16434672 1088 16433584 1% /dev/shm
/dev/cciss/c0d1p1 206422068 96423700 99512744 50% /home
/dev/cciss/c0d1p2 370402152 139316956 212269868 40% /mellon
isilon2:/ifs/data/rep-devel 58053531008 43039271424 12922985056 77% /rep-devel
isilon0:/ifs/data/SLES 58053531136 43039271424 12922985472 77% /SLES
triggs@rep-devel:/mellon/htdocs/dlr/EDIT> wc reptest-vid-purge.txt
324 324 36779 reptest-vid-purge.txt

Number of lines in the production file:
triggs@rep-devel:/mellon/htdocs/dlr/EDIT> wc /mellon/cvsroot/prod-vid-purge.txt
1551 1551 169059 /mellon/cvsroot/prod-vid-purge.txt

Dave can run the /mellon/cvsroot/zaparchtars.sh (the same as for untar purge log) on the prod-vid-purge.txt list whenever ready.

#3

Hi Jeffery!

> Is there one for staging too that I could use?

No there isn't. I would suggest using the script that was used on dev/test on staging and production to get the PID's and DSIDs in real-time.

#4

I'll work on beefing up findvideos.php so that it handles the zapping. I suppose this might be useful for other tasks like this further on.

#5

I've created an all in one script that can be run on staging and production with simple dryrun and realrun options. The following files are ready in /mellon/cvsroot on rep-test:
-rw-r--r-- 1 triggs developers 403 Feb 10 16:18 findvideos-readme.txt
-rw-r--r-- 1 triggs developers 1514 Feb 10 16:12 findvideos.php

The readme is as follows:
This is a readme for findvideos.php, a script to remove obsolete FLV-* or MOV-* datastreams from the repository.

To run the dryrun version, cd to the directory where findvideos.php has been downloaded and type:
php findvideos.php dryrun
This will find any FLV or MOV datastreams and print a list of the commands needed to remove them.
To run the actual removal script, type:
php findvideos.php realrun

#6

Ran on rep-staging in dryrun mode output is attached.

I am not sure if the MP4 datastreams were ever made for rep-staging as
a few I spot checked did not seem to have them. I am not sure if we
are ready for deleting them on this server.

Let me know.

#7

Actually this log looks right Dave. This script skips over any MP4 datastreams and only attacks obsolete FLV and MOV.

#8

Dave,

Thanks. Since the MP4 creation and addition process was manual it was never done on the staging system. Looking at the small number of of staging objects that are typeOfResource movingImage I think we can proceed with pruging the FLV and MOV. Some will still work, have MP4, which should be fine for release testing.

#9

Run on rep-staging in realrun mode. Output is the same as dryrun
with the only difference being:

< Reading data from rep-staging.libraries.rutgers.edu for dryrun run...
---
> Reading data from rep-staging.libraries.rutgers.edu for realrun run...

#10

Status:active» Moved to JIRA

Back to top