Save a site... - Tilted Forum Project Discussion Community

eyeronic · 06-05-2003, 10:49 PM

Is there a way/program to save a site(links and everything intact) to be used offline?
Would this work with the TFP?
How big do you think the TFP is in Gigs?

MacGnG · 06-06-2003, 01:04 PM

there aren't any programs i know of that would work with TFP because it uses cookies, etc.. but there are MANY programs that can be used to simply download everything from a URL, imput url and it just starts to download any and all links from it (better progz have better configs, etc..) but u get the idea

try download.com or something like that

Stiltzkin · 06-06-2003, 03:29 PM

You know, this reminds me of some site I found a while ago operated by some university. They have a seriously massive database cataloging the entire internet for each year, so you can actually go back and see what the internet looked like in say, 1997. I know I posted about it at the http://www.fanimenation.com/ forums a looong time ago. Now I'm going through all the threads to try to find it.

Redirect · 06-07-2003, 07:13 AM

depending on how much you're actually trying to save, try going to FILE then SAVE AS.........i know it will save the current page and all graphics on it, not sure if you will be able to do a WHOLE site. this is my second day here and as far as i know TFP is using
some massive space....that TT board is pretty stocked up with graphics.

ah yes..........check this link
http://download.com.com/3000-2381-10...l?tag=lst-0-17

MSD · 06-07-2003, 07:09 PM

The site that archives the whole internet is www.archive.org

It only catches the front page of TFP. As for saving to disk, I recall Sty saying last week that the database passed the 2GB mark. The problem that you would probably have is that certain information (hidden email addresses in TFP, Admin areas of most sites, etc.) are set to restricted viewing rights. Anything that tried to save everything to disk would get horribly confused by the TFP, and also suck down a whole hell of a lot of bandwidth (which does not make administrators happy. )

eyeronic · 06-08-2003, 12:06 PM

MrSelfDestruct: I just started thinking about this because I was rememebering how sad the loss of the last TFP was. Is there a backup system this time? A backup server? The whole thing is only 2 gigs?

geeza · 06-21-2003, 01:44 AM

Great tip guys thanks

etrick · 06-25-2003, 09:49 PM

Agreed. Awesome, very useful.

giblfiz · 07-05-2003, 06:22 PM

I don't know what I would do in the windows world, but if you can get your hands on a unix running computer (not to hard in these days of linux and free software) the program you want is called curl (wget would also work but not as well for this task) you would need to play with it a little so that it sent the right cookies and post data, but all of that can be done pretty easily through command line options. In theory you could get everything that you have accsess to as a user, it would probably take about 2 hours of fiddleing with to do. The other option is just to ask the admin to send you a copy of the sight ziped up in one big file and then encripted with a public key so you couldn't go rooting around thru private data. Then the admin just needs to find somewhere safe to keep his (private) key and he can recover from you whenever he wants. Aw hell screw public key, he could just use a cypher for it and that would work plenty fine, just never gives you the cypherkey.

nulltype · 07-07-2003, 11:09 AM

Under windows, you can use the Windows port of wget to do something like this.

http://space.tin.it/computer/hherold/

Just read through the manual. The administrators of the forum might not like you much if you tried to download the entire forum, but it should be possible. You can also use rate limiting, and you should be able to save the pictures. Also, if you use Netscape/Mozilla you should be able to use the cookies.txt generated by the browser to allow wget to act as if it is logged in. I am not sure if you can do that with IE.

mvassek · 07-12-2003, 05:21 PM

I use webreaper it is great and it allows you to get the information in threads you only go into the site as deep as you want to.

I don't remember the site I got it from but it was free.

07-15-2003, 09:54 AM

Quote:

Originally posted by Redirect
....that TT board is pretty stocked up with graphics.

Remember newbies. Most of the images on the titty board are stored on peoples private hosts, so the overall size of the TFP isn't incredibly large. It's big, but it's not terabyte big.

MichaelFarker · 09-22-2003, 09:38 PM

WinHTTrack is free and works well.

aryan · 10-03-2003, 05:12 PM

it would be cool to have a program like that

AxelF · 10-04-2003, 03:24 PM

I used WebCopier to download a site about Prague to my laptop before I went there. It was very useful in the hotel room.

BUT it took me DAYS of downloading on my broadband. DAYS! The project will not be letting you browse it until it is finnished or you stop it. Beacause not until then will it convert all the links to local links, which again took hours. But it worked, and it took gigs of space. (turned out to include tons of pictures of the local hookers that I could have been without)

NOTE: a lot of owners of sites do not like (understatement) you to use these robot downloaders and may spank you for it. I am not sure that it would be liked if you try to image this site locally... I would not risk my membership here trying. It may be against the rules in the charter if I am not remembering wrong. So ask first!

arcane · 10-05-2003, 03:43 AM

i use wget

google for it

chewybaca96 · 10-08-2003, 06:35 PM

WinHTTrack website copier is a good one.

http://www.httrack.com

Nazggul · 10-09-2003, 03:26 PM

Hell I think Frontpage had a feature like that way back when. Not that I would recommend buying it. I remember using it back in the mid to late 90's sometime.

ratbastid · 10-11-2003, 07:09 PM

I could whack ya up something in Perl in about half an hour.

06-05-2003, 10:49 PM	#1 (permalink)
eyeronic I run E. Location: New York	Download a site... Is there a way/program to save a site(links and everything intact) to be used offline? Would this work with the TFP? How big do you think the TFP is in Gigs? __________________ I hold with those that favor fire. Last edited by eyeronic; 06-06-2003 at 05:37 AM..

06-06-2003, 03:29 PM	#3 (permalink)
Stiltzkin Junkie	You know, this reminds me of some site I found a while ago operated by some university. They have a seriously massive database cataloging the entire internet for each year, so you can actually go back and see what the internet looked like in say, 1997. I know I posted about it at the http://www.fanimenation.com/ forums a looong time ago. Now I'm going through all the threads to try to find it. __________________ The most important thing in this world is love.

06-07-2003, 07:13 AM	#4 (permalink)
Redirect Loser Location: Chicago	depending on how much you're actually trying to save, try going to FILE then SAVE AS.........i know it will save the current page and all graphics on it, not sure if you will be able to do a WHOLE site. this is my second day here and as far as i know TFP is using some massive space....that TT board is pretty stocked up with graphics. ah yes..........check this link http://download.com.com/3000-2381-10...l?tag=lst-0-17 Last edited by Redirect; 06-07-2003 at 07:16 AM..

06-08-2003, 12:06 PM	#6 (permalink)
eyeronic I run E. Location: New York	MrSelfDestruct: I just started thinking about this because I was rememebering how sad the loss of the last TFP was. Is there a backup system this time? A backup server? The whole thing is only 2 gigs? __________________ I hold with those that favor fire.

07-05-2003, 06:22 PM	#9 (permalink)
giblfiz is a shoggoth Location: LA	I don't know what I would do in the windows world, but if you can get your hands on a unix running computer (not to hard in these days of linux and free software) the program you want is called curl (wget would also work but not as well for this task) you would need to play with it a little so that it sent the right cookies and post data, but all of that can be done pretty easily through command line options. In theory you could get everything that you have accsess to as a user, it would probably take about 2 hours of fiddleing with to do. The other option is just to ask the admin to send you a copy of the sight ziped up in one big file and then encripted with a public key so you couldn't go rooting around thru private data. Then the admin just needs to find somewhere safe to keep his (private) key and he can recover from you whenever he wants. Aw hell screw public key, he could just use a cypher for it and that would work plenty fine, just never gives you the cypherkey. __________________ Use the star one and you'll be fighting off the old ones with your bare hands -A Shoggoth on the Roof

06-06-2003, 01:04 PM	#2 (permalink)
MacGnG Know Where!	there aren't any programs i know of that would work with TFP because it uses cookies, etc.. but there are MANY programs that can be used to simply download everything from a URL, imput url and it just starts to download any and all links from it (better progz have better configs, etc..) but u get the idea try download.com or something like that

06-07-2003, 07:09 PM	#5 (permalink)
MSD The sky calls to us ... Super Moderator Location: CT	The site that archives the whole internet is www.archive.org It only catches the front page of TFP. As for saving to disk, I recall Sty saying last week that the database passed the 2GB mark. The problem that you would probably have is that certain information (hidden email addresses in TFP, Admin areas of most sites, etc.) are set to restricted viewing rights. Anything that tried to save everything to disk would get horribly confused by the TFP, and also suck down a whole hell of a lot of bandwidth (which does not make administrators happy. )

06-21-2003, 01:44 AM	#7 (permalink)
geeza Crazy	Great tip guys thanks

06-25-2003, 09:49 PM	#8 (permalink)
etrick Upright	Agreed. Awesome, very useful.

07-07-2003, 11:09 AM	#10 (permalink)
nulltype Crazy	Under windows, you can use the Windows port of wget to do something like this. http://space.tin.it/computer/hherold/ Just read through the manual. The administrators of the forum might not like you much if you tried to download the entire forum, but it should be possible. You can also use rate limiting, and you should be able to save the pictures. Also, if you use Netscape/Mozilla you should be able to use the cookies.txt generated by the browser to allow wget to act as if it is logged in. I am not sure if you can do that with IE.

07-12-2003, 05:21 PM	#11 (permalink)
mvassek Crazy Location: East Tennessee	I use webreaper it is great and it allows you to get the information in threads you only go into the site as deep as you want to. I don't remember the site I got it from but it was free.

09-22-2003, 09:38 PM	#13 (permalink)
MichaelFarker 2+2=5? Not again! Location: Dallas, Texas	WinHTTrack is free and works well.

10-03-2003, 05:12 PM	#14 (permalink)
aryan Upright	it would be cool to have a program like that

10-04-2003, 03:24 PM	#15 (permalink)
AxelF Crazy Location: Europe	I used WebCopier to download a site about Prague to my laptop before I went there. It was very useful in the hotel room. BUT it took me DAYS of downloading on my broadband. DAYS! The project will not be letting you browse it until it is finnished or you stop it. Beacause not until then will it convert all the links to local links, which again took hours. But it worked, and it took gigs of space. (turned out to include tons of pictures of the local hookers that I could have been without) NOTE: a lot of owners of sites do not like (understatement) you to use these robot downloaders and may spank you for it. I am not sure that it would be liked if you try to image this site locally... I would not risk my membership here trying. It may be against the rules in the charter if I am not remembering wrong. So ask first! __________________ Coffee

10-05-2003, 03:43 AM	#16 (permalink)
arcane Insane Location: SLC, UT	i use wget google for it __________________ <Arcane> so if you banged 2000 chicks then at least one had a pen0r?

10-08-2003, 06:35 PM	#17 (permalink)
chewybaca96 We can't stop here! This is bat country! Location: SL,UT	WinHTTrack website copier is a good one. http://www.httrack.com __________________ Brian: “Ok, all we’ve gotta do is find the American Embassy, and they’ll help us get home” Stewie: “Home? I have no intention of returning to that disgusting hovel with that intolerable woman, that fat slob, and that insufferable dog… Oh, you’re right here aren’t you? Oh well, I stand by it."

10-09-2003, 03:26 PM	#18 (permalink)
Nazggul Psycho Location: San Francisco	Hell I think Frontpage had a feature like that way back when. Not that I would recommend buying it. I remember using it back in the mid to late 90's sometime. __________________ "If something has to give then it always will." -- Editors

10-11-2003, 07:09 PM	#19 (permalink)
ratbastid Darth Papa Location: Yonder	I could whack ya up something in Perl in about half an hour.