Bratwurst

Anyone know how to do something like this? Maybe through a batch file?

1. identify all files within a folder (including subfolders) that are over 250MB
2. Run through the directory of files and zip each identified into a .zip file with the same name, including the file extension (ext: myfile.dbf ==> myfile_dbf.zip)
3. verifies that the zip archive is valid
4. delete the original files which was zipped (upon a successful verification in step 3)

ideally i'd like to run multiple occurrences of such a script/app, or have the app/script take advantage of as much horsepower as possible and process multiple files in parallel.

__________________
40

ratbastid

Shame you're talking batch file. I could write it for you in thirty seconds in Bash or Perl. Especially Perl, it's like a two-liner. But you probably wouldn't want to install Cygwin just for this...

cyrnel

A few zippers have scripting methods included. I don't know them so resort to ugliness.

Copy to compressbigstuff.bat:
Scary but functional. It only compresses 250MB txt files as is. Assumes 7za (http://www.7-zip.org/) is in your path. (BTW, 7zip is very efficient compared to rar, winzip, etc.)

Change the "(*.txt)" to "*" for everything, or allow passing in your own directory with %1, etc. You could have it follow directories by changing "for" to "for /R", but I wasn't that adventurous. (see "for /?" from the command line)

7zip can be multithreaded if you ask it to be with some compression types. I wouldn't bother forcing multiple instances or it'll be slower from overhead and thrashing, and force file fragmentation.

__________________
There are a vast number of people who are uninformed and heavily propagandized, but fundamentally decent. The propaganda that inundates them is effective when unchallenged, but much of it goes only skin deep. If they can be brought to raise questions and apply their decent instincts and basic intelligence, many people quickly escape the confines of the doctrinal system and are willing to do something to help others who are really suffering and oppressed." -Manufacturing Consent: Noam Chomsky and the Media, p. 195

Bratwurst

Hey guys thanks a ton! I'll try this Cyrnel, and Ratbastid, I don't mind using perl. I have perl installed on a different machine for MRTG so I can probably install it again elsewhere. I'm not very learned in the finer points of it, but I'm sure I can get it working.

__________________
40

Bratwurst

I installed 7-zip, but I think the above about path is causing an error. I'm getting this error when I try to run it:

M:\test\FL\develop>zip.bat
'7za' is not recognized as an internal or external command,
operable program or batch file.

Do I need to do something further to get this to run?

__________________
40

Bratwurst

OK I changed 7za to 7zG and it runs now, makes the file for the gz when testing, but 7zip doesn't actually run and create the file as far as I can tell?

__________________
40

cyrnel

You need the command-line version.

Can't sidetrack to get you the url. I'm at a customer's right now & just snuck a tfp read in. I'll be back soon.

__________________
There are a vast number of people who are uninformed and heavily propagandized, but fundamentally decent. The propaganda that inundates them is effective when unchallenged, but much of it goes only skin deep. If they can be brought to raise questions and apply their decent instincts and basic intelligence, many people quickly escape the confines of the doctrinal system and are willing to do something to help others who are really suffering and oppressed." -Manufacturing Consent: Noam Chomsky and the Media, p. 195

Bratwurst

No problem thanks for the tip - I'll see if I can track it down

__________________
40

Bratwurst

I just tried it with the cmd line version and it worked! kick ass!

__________________
40

Bratwurst

Cyrnel, I ran it with the "for /R" and it started going through the sub directories like you said, however, it put the resulting compressed file into the same directory that I ran the batch file from. Is there a way to put a wildcard in there so the files remain in the original directories?

I wish I knew this as well as you guys do.

__________________
40

cyrnel

Sorry about that. Originally tried your "name-ext.zip" preference and left the baggage to trip over itself.

BTW, if you look at help for these things it's all self explanatory. At least you'd recognize what's going on. Stringing it together is just a matter of familiarity.

This old DOS stuff isn't something to be impressed by. Go for PERL or other if you have real work to do. It'll mean fewer migraines.

__________________
There are a vast number of people who are uninformed and heavily propagandized, but fundamentally decent. The propaganda that inundates them is effective when unchallenged, but much of it goes only skin deep. If they can be brought to raise questions and apply their decent instincts and basic intelligence, many people quickly escape the confines of the doctrinal system and are willing to do something to help others who are really suffering and oppressed." -Manufacturing Consent: Noam Chomsky and the Media, p. 195

Bratwurst

I'm really sorry I didn't write back sooner on this, I'm just now getting back to this project (been a rough couple months here at work!).

Just ran the script (well it is making its way through the directories I setup for testing as I write this) and it seems to be working well.

Do you guys think it is worth it to run this through Perl? I really don't mind installing it if it would be a lot better. Like I said before, I had Perl installed on a different machine to run MRTG and I am looking to maybe do RRDTool soon too.

Thanks again for the script - I'll let you know how it ends up.

__________________
40

cyrnel

Almost all of the real work is in zipping so Perl wouldn't necessarily be better for this application. But, you would be learning a tool that works everywhere and has far more power. Batches are good for quick and dirty Windows stuff, and it's always "installed", but they're horrible for complexity or if you need user interaction.

Spend a little time at http://learn.perl.org then convert the above to Perl.

__________________
There are a vast number of people who are uninformed and heavily propagandized, but fundamentally decent. The propaganda that inundates them is effective when unchallenged, but much of it goes only skin deep. If they can be brought to raise questions and apply their decent instincts and basic intelligence, many people quickly escape the confines of the doctrinal system and are willing to do something to help others who are really suffering and oppressed." -Manufacturing Consent: Noam Chomsky and the Media, p. 195

KnifeMissile

If you want to go the PERL route (and it's not a bad route to go), you don't need to install Cygwin. Indeed, I've never liked that environment. You can get a version of PERL that's native to Win32, called Active PERL. Now that I'm looking at the page, it appears they have other "active" langauges, like Python and Ruby, which is interesting... Anyway, if you're looking to install Perl for your Windows system, I highly recommend this one! This is the version of PERL that I use, before I learned Ruby...

Which brings me to my next topic, have you considered something a little more powerful than even PERL, like Python or Ruby? I prefer Ruby myself, so I'll advocate that, but Python is just as powerful so I can't fault you for choosing it.
Just giving you some options...