How to FTP a large number of files? - Tilted Forum Project Discussion Community

sailor · 10-07-2004, 08:11 PM

Heres the situation: I have a friend with a corrupted Windows install that he would like to get data off of. No problem: I fire up Knoppix, find everything, and get ready to transfer it to one of my boxen. Thing is, I cant quite figure out the best way to do it. As far as I can tell, Knoppix doesnt include a graphical FTP utility, and I cant figure out how to send a large number of files and folders. I tried the mput command, but it wont create the folders, and Im not about to go make them all by hand. I thought about just making a tar with all the files he needs and then sending that, but I cant write to the disk to have somewhere to put it: the disk is formatted NTFS. Any suggestions? I tried SSH/SCP, but (I think) because of the disparity in processor speeds, his was choking trying to encode/decode everything while sending it, leading to corrupted files and a failed transfer. Id really like to get the FTP working, but I cant for the life of me figure out a good way to send that much stuff with the CLI based ftp...

bendsley · 10-07-2004, 08:24 PM

What I would probably tell you to do is start an SSH server on the knoppix cd. I believe this comes with Knoppix by default. Once you are able to SSH in to the machine, install a program on the Windows machine you are wanting to go to called WinSCP. It's basically graphical file transfer over SSH.

You can set the program up to look like Windows Explorer format or Norton Commander format (which is easier). Copy and transfer the files to where you need.

I work with linux boxes at work quite a bit and find this a very easy way to transfer files. Hope this helps.

sailor · 10-08-2004, 07:32 AM

Thats exactly what I tried first, but unfortunately it kept failing. I think the reason is the disparity in processors--my Athlon64 3000 was running at about 65% to decode the stuff, I doubt his 3 year old laptop was keeping up. Files kept corrupting and screwing up the transfer

sailor · 10-08-2004, 04:19 PM

Anyone? This is bugging the hell out of me...

sailor · 10-10-2004, 04:39 PM

OK, one more bump...

belkins · 10-10-2004, 07:15 PM

Can you attach an external hard drive and copy the files/folders?

SiN · 10-10-2004, 08:38 PM

sailor - my bf thinks he can answer your Q ...

he'll be around when he gets home from work (9+ hours from now) so maybe we can get it figured out then.

schnable · 10-11-2004, 12:01 AM

Quote:

Originally Posted by sailor

Thats exactly what I tried first, but unfortunately it kept failing. I think the reason is the disparity in processors--my Athlon64 3000 was running at about 65% to decode the stuff, I doubt his 3 year old laptop was keeping up. Files kept corrupting and screwing up the transfer

This sounds strange. Are you sure that the hard drive isn't trashed and the poor performace and corrupt data are the result of this?

Try using a USB memory key or some external storage device and transfer some test files. If they are corrupt as well, they hard drive might be screwed.

If the HD is in poor shape, you may be able to run a data recovery program on the drive. I can't make any recommendations unfortunately because I haven't been doing IT in a while. Anyone?

sailor · 10-11-2004, 04:26 AM

Quote:

Originally Posted by belkins

Can you attach an external hard drive and copy the files/folders?

Unfortunately, the only external drive I have is formatted NTFS, so I cant write to it from Knoppix.

Ill try the USB solution today. Not the best solution, but it just might work. Gonna be fun umount/mounting it a bunch though

Sin, thanks for the help... Ill keep watching for his reply

WildChild · 10-11-2004, 12:58 PM

Hi, I'm SiN's boyfriend.
Aight, here goes nothing.
First of all, I see how SCP could be excessively slow in your case, with the (probably slow) laptop and a high-speed link (where you would expect more), but then again getting CRC errors on files doesn't make sense.
Let's just forget about that, it wasn't meant for this kind of use anyway.
There's esentially two (three) options I see.
1) Setup some stupid ftp server (e.g. vsftpd. stupid = simplicity in this case) on either machine. Preferrably on the target machine. This should be simple enough for any distro. Heck even Knoppix comes with vsftpd afaik.
Then use ncftp on the client (the `broken') machine. Keyword: "mput -r"
2) Setup samba on the target machine, make one share, guest = yes; browseable = yes; write users = youruser;...
This could be an example (this is taken from a working configuration with minor modifications):

Code:

[yoursharenamehere]
        path            = /dump
        comment         = backup dump
        browseable      = yes
        writeable       = yes
        create mask     = 0777
        directory mask  = 0777
        force user      = yourunixuserhere
        force group     = yourunixuserhere
        hide dot files  = yes
        guest ok        = yes
        unix charset    = iso8859-15
        display charset = iso8859-15
        client codepage = 850

Don't forget to sync/add samba users on the server using smbpasswd (as always, man is your friend)
3) lather, rinse, repeat, just with nfs. But here is more involved, you'd need user-space-daemon and a locking daemon - installation differs from distro to distro, so unless you know what you're doing, I'd recommend one of the above as a quick'n'dirty hack.

If you go for 2 or 3, either use cp -r or mc to control what you want to copy. mc does subdir tagging afaik, so it should be easy enough.

In any way, don't expect speed miracles here, the ntfs driver is not the speediest compared to `real, native' (no flaming please) fs drivers - but all of the above should be solid ways to get you from a to b without too much hassle.

enjoy

if need be, SiN can give you my email.

WildChild · 10-11-2004, 01:02 PM

An additional thought: if everything else fails, you could use some kermit client + a parallel laplink cable, or one of the thousands of interlink apps.

I probably would have taken out the hd in the first place and put it in the target machine, since I'm a lazy tard and I hate having to wait for things

sailor · 10-11-2004, 03:18 PM

Excellent, thanks for the help. Now if I could just get yalls help on my german skills

One question: mput -r should put/create directories as well? Some of the stuff that has to go over consists of some pretty nasty trees that I dont want to make manually.

10-07-2004, 08:11 PM	#1 (permalink)
sailor beauty in the breakdown Location: Chapel Hill, NC	How to FTP a large number of files? Heres the situation: I have a friend with a corrupted Windows install that he would like to get data off of. No problem: I fire up Knoppix, find everything, and get ready to transfer it to one of my boxen. Thing is, I cant quite figure out the best way to do it. As far as I can tell, Knoppix doesnt include a graphical FTP utility, and I cant figure out how to send a large number of files and folders. I tried the mput command, but it wont create the folders, and Im not about to go make them all by hand. I thought about just making a tar with all the files he needs and then sending that, but I cant write to the disk to have somewhere to put it: the disk is formatted NTFS. Any suggestions? I tried SSH/SCP, but (I think) because of the disparity in processor speeds, his was choking trying to encode/decode everything while sending it, leading to corrupted files and a failed transfer. Id really like to get the FTP working, but I cant for the life of me figure out a good way to send that much stuff with the CLI based ftp... __________________ "Good people do not need laws to tell them to act responsibly, while bad people will find a way around the laws." --Plato

10-07-2004, 08:24 PM	#2 (permalink)
bendsley Professional Loafer Location: texas	What I would probably tell you to do is start an SSH server on the knoppix cd. I believe this comes with Knoppix by default. Once you are able to SSH in to the machine, install a program on the Windows machine you are wanting to go to called WinSCP. It's basically graphical file transfer over SSH. You can set the program up to look like Windows Explorer format or Norton Commander format (which is easier). Copy and transfer the files to where you need. I work with linux boxes at work quite a bit and find this a very easy way to transfer files. Hope this helps. __________________ "You hear the one about the fella who died, went to the pearly gates? St. Peter let him in. Sees a guy in a suit making a closing argument. Says, "Who's that?" St. Peter says, "Oh, that's God. Thinks he's Denny Crane."

10-08-2004, 07:32 AM	#3 (permalink)
sailor beauty in the breakdown Location: Chapel Hill, NC	Thats exactly what I tried first, but unfortunately it kept failing. I think the reason is the disparity in processors--my Athlon64 3000 was running at about 65% to decode the stuff, I doubt his 3 year old laptop was keeping up. Files kept corrupting and screwing up the transfer __________________ "Good people do not need laws to tell them to act responsibly, while bad people will find a way around the laws." --Plato

10-08-2004, 04:19 PM	#4 (permalink)
sailor beauty in the breakdown Location: Chapel Hill, NC	Anyone? This is bugging the hell out of me... __________________ "Good people do not need laws to tell them to act responsibly, while bad people will find a way around the laws." --Plato

10-10-2004, 04:39 PM	#5 (permalink)
sailor beauty in the breakdown Location: Chapel Hill, NC	OK, one more bump... __________________ "Good people do not need laws to tell them to act responsibly, while bad people will find a way around the laws." --Plato

10-10-2004, 07:15 PM	#6 (permalink)
belkins Tilted Location: Salt Lake City	Can you attach an external hard drive and copy the files/folders?

10-10-2004, 08:38 PM	#7 (permalink)
SiN strangelove Location: ...more here than there...	sailor - my bf thinks he can answer your Q ... he'll be around when he gets home from work (9+ hours from now) so maybe we can get it figured out then. __________________ - + - ° GiRLie GeeK ° - + - ° 01110010011011110110111101110100001000000110110101100101 Therell be days/When Ill stray/I may appear to be/Constantly out of reach/I give in to sin/Because I like to practise what I preach

10-11-2004, 12:58 PM	#10 (permalink)
WildChild Upright	Hi, I'm SiN's boyfriend. Aight, here goes nothing. First of all, I see how SCP could be excessively slow in your case, with the (probably slow) laptop and a high-speed link (where you would expect more), but then again getting CRC errors on files doesn't make sense. Let's just forget about that, it wasn't meant for this kind of use anyway. There's esentially two (three) options I see. 1) Setup some stupid ftp server (e.g. vsftpd. stupid = simplicity in this case) on either machine. Preferrably on the target machine. This should be simple enough for any distro. Heck even Knoppix comes with vsftpd afaik. Then use ncftp on the client (the `broken') machine. Keyword: "mput -r" 2) Setup samba on the target machine, make one share, guest = yes; browseable = yes; write users = youruser;... This could be an example (this is taken from a working configuration with minor modifications): Code: [yoursharenamehere] path = /dump comment = backup dump browseable = yes writeable = yes create mask = 0777 directory mask = 0777 force user = yourunixuserhere force group = yourunixuserhere hide dot files = yes guest ok = yes unix charset = iso8859-15 display charset = iso8859-15 client codepage = 850 Don't forget to sync/add samba users on the server using smbpasswd (as always, man is your friend) 3) lather, rinse, repeat, just with nfs. But here is more involved, you'd need user-space-daemon and a locking daemon - installation differs from distro to distro, so unless you know what you're doing, I'd recommend one of the above as a quick'n'dirty hack. If you go for 2 or 3, either use cp -r or mc to control what you want to copy. mc does subdir tagging afaik, so it should be easy enough. In any way, don't expect speed miracles here, the ntfs driver is not the speediest compared to `real, native' (no flaming please) fs drivers - but all of the above should be solid ways to get you from a to b without too much hassle. enjoy if need be, SiN can give you my email.

10-11-2004, 01:02 PM	#11 (permalink)
WildChild Upright	An additional thought: if everything else fails, you could use some kermit client + a parallel laplink cable, or one of the thousands of interlink apps. I probably would have taken out the hd in the first place and put it in the target machine, since I'm a lazy tard and I hate having to wait for things

10-11-2004, 03:18 PM	#12 (permalink)
sailor beauty in the breakdown Location: Chapel Hill, NC	Excellent, thanks for the help. Now if I could just get yalls help on my german skills One question: mput -r should put/create directories as well? Some of the stuff that has to go over consists of some pretty nasty trees that I dont want to make manually. __________________ "Good people do not need laws to tell them to act responsibly, while bad people will find a way around the laws." --Plato