Clustering and Linux - Tilted Forum Project Discussion Community

Toecutter · 06-23-2005, 01:41 PM

I am in the process of building a SAN cluster using Suse 9.0. I have scoured the web for resources and have had my share of "trial and error" trying to get this dog to hunt. Could anyone recommend an "easy" how to in building culsters. I have d/l Mandrake CLIC, but that realy wasn't what i am looking for. I am willing to give openMosix a try again, I d/l the rpm for that, but from what i read, i need to get off my butt and compile it. BTW, i'm a tech for a school district in montana, so funds are tight. Any suggetions will be appreciated and thank you in advance!

skaven · 06-23-2005, 07:20 PM

What you're probably looking for is called a "clustered file system". There are lots of free ones out there. OpenAFS is one that scales up to enterprise-sized clusters and I believe even supports caching over WANs. Lustre is another one, but I'm not sure if it's free. Just google around for "clustered file system" and you'll get some hits.

Is there any particular reason you need to use multiple machines for this, though? Unless you're expecting to be doing more than 60Mb/sec consistent throughput, why not just build one big fileserver and not worry about the whole cluster side of things? A modest Opteron-based system with a couple GB of RAM, a gigabit network card, a few SATA cards, some 400GB SATA drives, and good ol' software RAID-5 and you can have a very nice file server for not much money.

If this has to be ultra-high performance, then obviously a CFS is the way to go. Don't knock the benefits of a giant stack of SATA drives, though. With 3 4-port SATA cards, 12 400GB disks (11 useable after one parity drive), that's 4.4Tb of space!

Latch · 06-23-2005, 11:05 PM

We used Redhat's GFS (in fact, I think Fedora Core 4 actually comes with it now) at work. It was shit for the way we wanted to use it (with Samba2/3), and I believe it took a while in lots of small writes, but kicked ass in large, single-file writes.

bendsley · 06-24-2005, 05:16 AM

I don't guess I see the need for multiple machines for a SAN though unless you just have a hojillion bytes of data. I don't know if I'd go with SATA drives for enterprise class, but for just data storage they would probably work ok. 3ware makes 4-port, 8-port and 12-port SATA Raid cards. They're not too expensive and neither are SATA drives.

With clustering, you're going to have a lot of overhead, both on the machines and the network. Most clustering setups will use something like Dolphin or Myrinet to connect the machines.

If you go the Dolphin route, you have to get an SCI Interconnect card for each machine, cables to go from them to a BxBAR SCI Switch and then you have to have the switch itself. This is just basic. There is a lot more to this setup.

If you choose the Myrinet route, you will need to buy the Myri-10G cards for each machine. These will support 10G Myrinet/10G Ethernet. You can also go with Myrinet Fibre Cards, but like the rest of the stuff mentioned, they start around $700 each. Then there's the cost of the switches, etc. Myrinet switches start around $4,000 and go up to ~$13,000.

If you want to look for a canned solution, Topspin is one of the best out there. They were recently acquired by Cisco Systems. The Topspin stuff can use Fibre Channel, Ethernet (1g or 10g) or Infiniband.

There is also the iSCSI protocol out there you might research and see if that would benefit your needs.

www.topspin.com - Topspin
www.myri.com - Myricom
www.dolphinics.com - Dolphin ICS

You might look here as well:
http://www.clusterworld.com/

Toecutter · 06-24-2005, 05:41 AM

Thanks for the input, everyone! I agree, it would make sense to build a box w/ a few SATA drive, pref in a RAID 5 config and everyones ideas are great, but this costs money, and being a school dist. that has a hard time getting mill levys, general or tech, i have learned to work with what i have, this is why im going the cluster route. Thanks again!

bendsley · 06-24-2005, 05:46 AM

Do you want to buy machines and then cluster or cluster machines you already have? Are these machines going to be dedicated or will they be in use for other purposes?

Toecutter · 06-24-2005, 12:52 PM

I've already got some boxes that i am using for this, mostly P3's. We use a combo of veritas backup exec and retrospect. i would like to send the retrospect backup sets to the cluster and then burn them to dvd for offline/site storage.

skaven · 06-24-2005, 06:23 PM

If you've already got some boxes for this...I see no reason to try to cluster them together. Just take all the hard drives you can and cram them into one chassis with the fastest system you can get your hands on. Again, unless you need ultra-high performance, a RAID-5 or JBOD setup with a big stack of disks will work just fine without the overhead and complexity of a clustered file system.

06-23-2005, 01:41 PM	#1 (permalink)
Toecutter Tilted Location: Montana	Clustering and Linux I am in the process of building a SAN cluster using Suse 9.0. I have scoured the web for resources and have had my share of "trial and error" trying to get this dog to hunt. Could anyone recommend an "easy" how to in building culsters. I have d/l Mandrake CLIC, but that realy wasn't what i am looking for. I am willing to give openMosix a try again, I d/l the rpm for that, but from what i read, i need to get off my butt and compile it. BTW, i'm a tech for a school district in montana, so funds are tight. Any suggetions will be appreciated and thank you in advance!

06-24-2005, 05:16 AM	#4 (permalink)
bendsley Professional Loafer Location: texas	I don't guess I see the need for multiple machines for a SAN though unless you just have a hojillion bytes of data. I don't know if I'd go with SATA drives for enterprise class, but for just data storage they would probably work ok. 3ware makes 4-port, 8-port and 12-port SATA Raid cards. They're not too expensive and neither are SATA drives. With clustering, you're going to have a lot of overhead, both on the machines and the network. Most clustering setups will use something like Dolphin or Myrinet to connect the machines. If you go the Dolphin route, you have to get an SCI Interconnect card for each machine, cables to go from them to a BxBAR SCI Switch and then you have to have the switch itself. This is just basic. There is a lot more to this setup. If you choose the Myrinet route, you will need to buy the Myri-10G cards for each machine. These will support 10G Myrinet/10G Ethernet. You can also go with Myrinet Fibre Cards, but like the rest of the stuff mentioned, they start around $700 each. Then there's the cost of the switches, etc. Myrinet switches start around $4,000 and go up to ~$13,000. If you want to look for a canned solution, Topspin is one of the best out there. They were recently acquired by Cisco Systems. The Topspin stuff can use Fibre Channel, Ethernet (1g or 10g) or Infiniband. There is also the iSCSI protocol out there you might research and see if that would benefit your needs. www.topspin.com - Topspin www.myri.com - Myricom www.dolphinics.com - Dolphin ICS You might look here as well: http://www.clusterworld.com/ __________________ "You hear the one about the fella who died, went to the pearly gates? St. Peter let him in. Sees a guy in a suit making a closing argument. Says, "Who's that?" St. Peter says, "Oh, that's God. Thinks he's Denny Crane."

06-24-2005, 05:46 AM	#6 (permalink)
bendsley Professional Loafer Location: texas	Do you want to buy machines and then cluster or cluster machines you already have? Are these machines going to be dedicated or will they be in use for other purposes? __________________ "You hear the one about the fella who died, went to the pearly gates? St. Peter let him in. Sees a guy in a suit making a closing argument. Says, "Who's that?" St. Peter says, "Oh, that's God. Thinks he's Denny Crane."

06-23-2005, 07:20 PM	#2 (permalink)
skaven Insane Location: Austin, TX	What you're probably looking for is called a "clustered file system". There are lots of free ones out there. OpenAFS is one that scales up to enterprise-sized clusters and I believe even supports caching over WANs. Lustre is another one, but I'm not sure if it's free. Just google around for "clustered file system" and you'll get some hits. Is there any particular reason you need to use multiple machines for this, though? Unless you're expecting to be doing more than 60Mb/sec consistent throughput, why not just build one big fileserver and not worry about the whole cluster side of things? A modest Opteron-based system with a couple GB of RAM, a gigabit network card, a few SATA cards, some 400GB SATA drives, and good ol' software RAID-5 and you can have a very nice file server for not much money. If this has to be ultra-high performance, then obviously a CFS is the way to go. Don't knock the benefits of a giant stack of SATA drives, though. With 3 4-port SATA cards, 12 400GB disks (11 useable after one parity drive), that's 4.4Tb of space!

06-23-2005, 11:05 PM	#3 (permalink)
Latch In Your Dreams Location: City of Lights	We used Redhat's GFS (in fact, I think Fedora Core 4 actually comes with it now) at work. It was shit for the way we wanted to use it (with Samba2/3), and I believe it took a while in lots of small writes, but kicked ass in large, single-file writes.

06-24-2005, 05:41 AM	#5 (permalink)
Toecutter Tilted Location: Montana	Thanks for the input, everyone! I agree, it would make sense to build a box w/ a few SATA drive, pref in a RAID 5 config and everyones ideas are great, but this costs money, and being a school dist. that has a hard time getting mill levys, general or tech, i have learned to work with what i have, this is why im going the cluster route. Thanks again!

06-24-2005, 12:52 PM	#7 (permalink)
Toecutter Tilted Location: Montana	I've already got some boxes that i am using for this, mostly P3's. We use a combo of veritas backup exec and retrospect. i would like to send the retrospect backup sets to the cluster and then burn them to dvd for offline/site storage.

06-24-2005, 06:23 PM	#8 (permalink)
skaven Insane Location: Austin, TX	If you've already got some boxes for this...I see no reason to try to cluster them together. Just take all the hard drives you can and cram them into one chassis with the fastest system you can get your hands on. Again, unless you need ultra-high performance, a RAID-5 or JBOD setup with a big stack of disks will work just fine without the overhead and complexity of a clustered file system.