The following is a chat with JonathanD regarding IBM Bladecenter technology. He is a certified IBM BladeCenter expert
[Attendee1] So, how can I get one for $10/mo?
[@JonathanD] Attendee1: become an ibm business partner doing more than $100,000 a year in IBM sales, then apply for a development lease.
[Attendee2] Well, blade centers look interesting, but some of the information is moderately, er, confusing?
[@JonathanD] Anything in particular, Attendee2?
[Attendee2] Like, specifically how networking works with the backplane
[@JonathanD] I’ll summarize briefly, then you can ask questions, ok?
[@JonathanD] in a normal configuration, we’ll put 2 switch modules in a chassis.
[@JonathanD] These switches go in bays 1 and 2
[@JonathanD] regardless of the chassis
[@JonathanD] bays 1 and 2 are essentially hard wired to a network connection on each blade
[@JonathanD] each blade has to built in NICs
[@JonathanD] it’s actually a single dual port NIC, physically
[@JonathanD] first port goes to bay 1
[@JonathanD] second port goes to bay 2
[@JonathanD] so thats the baseline. Each blade connects to bay 1 and 2 at least once.
[Attendee2] So, the switch in the bay determines the speed of networking between blades, not the speed of the backplane?
[@JonathanD] Attendee2: yes. The backplane is basically just wiretraces.
[@JonathanD] theoretical limit is 10GB
[@JonathanD] however, all blades, to date, have only 1GB nics built in
[@JonathanD] now, I said at least once, because a 2 slot blade has at least the potential for 2 connections to each switch
[@JonathanD] depending on the blade, how it’s configured, and a few other things, you’ll have 1 or 2, to each
[Attendee2] So, if you wanted to add 10gb or infiniband or whatever you’d have to put in a switch for it and a module in every blade
[@JonathanD] Attendee2: essentially yes
[@JonathanD] I’m going to speak briefly on infiniband and what it can offer in a moment
[@JonathanD] just high level.
[@JonathanD] now, the enterprise chassis (just called bladecenter) has 4 IO module slots
[@JonathanD] as we discussed , 1 and 2 are dedicated to ethernet
[@JonathanD] externally, they can be fiber or copper.
[@JonathanD] the other 2 can be a wide variety of things
[@JonathanD] fiber channel, SAS, more ethernet, infiniband
[@JonathanD] in the newer H chassis, there are all-told 10 slots for different IO options.
[@JonathanD] one of these is a high speed infiniband switch.
[@JonathanD] with the infiniband switch, you can do some interesting things…
[@JonathanD] not much support exists yet for infiniband, so what we have is a special adapter that sits in the blade and talks to the infiniband switch (preferably, 2 of them for redundancy, same as ethernet)
[@JonathanD] this gives us, on top of our 2 built in GBE connections, 6 additional “virtual” connections on infiniband fabric, to each blade.
[@JonathanD] with the infiniband switch, each of these 6 can be either Fiber channel OR 10GBE
[@JonathanD] and it can vary by blade.
[@JonathanD] Attendee2: does that answer your question, and do you have any more?
[Attendee1] Do cross-blade ethernet communications exit the ethernet I/O ports or does it handle ethernet requests between blades internally?
[Attendee1] Basically, can the ethernet be unplugged external to the bladecenter and still have ethernet between the blades themselves?
[@JonathanD] Attendee1: absolutely
[@JonathanD] the switches in a blade chassis are real switches, most of them with vlan capability
[Attendee3] Why do blades talk to each other? do they all pool into one server or something? like one blade can be a file server, one an email server, one a Web server, etc? sorry for a n00b question
[@JonathanD] but yes, often folks will put a substantial infrastructure inside a single chassis
[Attendee2] er, yeah it did (sorry thought I answered)
[@JonathanD] that doesn’t need to exit the chassis at all except for the end user responses
[@JonathanD] so your web server, app server, and db server (or cluster) can live inside the chassis.
[Attendee3] So I could have like 20 blades, all doing different things, but it looks like one server to an end user? Wow.
[Attendee4] when i look at photos of the H Chassis it looks closed off from the back, so if i put an expansion module on a blade to put in a PCI card or something, where will the connection sit for that card ?
[@JonathanD] furthermore, several of the switches are capable of load balancing with no additional hardware.
[@JonathanD] Attendee4: there are fillers on the back of a BC H, you take them out to install a switch module
[@JonathanD] that switch module has external ports
[Attendee1] JonathanD: What’s the network interconnection speed between blades?
[Attendee4] perfect !
[@JonathanD] the number and type of ports vary, based on what you choose.
[@JonathanD] Attendee1: up to 40Gbit per blade, depends entirely on the switches installed.
[Attendee4] i would just love, for redudnancy, to have my PBX sit on a blade, so i would need to add my T-1 interface card to the blade
[Attendee1] JonathanD: Does it only depend on the switch, or does the blade have to support that speed too?
[@JonathanD] Attendee1: the adapter in the blade does, yes.
[Attendee3] 40Gbit/sec… wows
[@JonathanD] much like a 1Gb switch does no good if you have a 10Mbit nic
[Attendee1] JonathanD: What’s the oldest that still supports 40Gbit?
[Attendee1] (stock from IBM)
[ttender3 still has 100Mbit network…with some nodes only 10Mbit
[@JonathanD] the HS21s, almost all of them, have support for infiniband to bays 7-10 which supports that 40GBit
[Attendee1] Also, does that interconnect show as a separate network card?
[Attendee3] Infiniband = unlimited speed?
[Attendee3] i.e. “infini”band?
[@JonathanD] Attendee1: that would be an infiniband connect, which in this case you would probably configure as virtual 10GbE adapters.
[@JonathanD] you can actually exceed that 40Gbit in some cases
[@JonathanD] and as the midplane is basically just traces, thats not a hard value.
[@JonathanD] it could increase as new switching technology becomes available.
[Attendee1] The 40Gbit is per blade or overall?
[@JonathanD] Attendee1: thats the current max that can be delivered to a single blade
[@JonathanD] within the chassis, that would be point to point.
[Attendee1] Is that full duplex or half duplex speed?
[@JonathanD] which makes it excellent for vmotion, for example.
[Attendee1] (as in utilizing both send/receive to get 40Gbps, or are both @ 40Gbps) ?
[@JonathanD] thats half, so you can push 40 in both directions.
[@JonathanD] across 4 interfaces or virtual interfaces.
[@JonathanD] in short, particuarly with infiniband, you have *lots* of bandwidth.
[Attendee3] I used to think of Mbits like mph…10 is slow and sucky and 100 was really fast.
[Attendee3] Now we have 40Gbit!?
[Attendee3] Is this available for regular non-blade servers?
[Attendee3] the most I’ve seen is 1Gbit, honestly.
[@JonathanD] Attendee3: infiniband is, yes. I don’t know if the virtual IO tech is.
[Attendee1] So the virtual network adapter you mentioned. How is this configured?
[Attendee3] “the virtual IO tech” means what?
[Attendee1] You install a driver in Windows, and configure this on the blade center level?
[@JonathanD] Attendee1: my understanding is that it’s configured in the cards BIOS
[@JonathanD] so you can do it pre-install
[Attendee1] I see.
[@JonathanD] and much like the qlogics, once the machine is up you can install software to manage it without a reboot.
[@JonathanD] it can also be assigned from within the managemnet module, and even scripted.
[Attendee1] So you could have upto 4 of these virtual cards or can you over allocate the bandwidth, and it’ll do best effort routing?
[Attendee1] (Per blade)
[@JonathanD] it will drive to the limit of the infiniband, yes.
[@JonathanD] you can configure 10GbE interfaces, and 4GB FC interface.
[Attendee1] So other than OS limits, how many network interfaces can you create?
[Attendee1] Per blade.
[@JonathanD] includeing the 2 built in
[@JonathanD] 6, within infiniband
[Attendee3] and what pretel do you *do* with 6 virtual NICs?
[Attendee1] So, each blade has 2 which are connected to the external ethernet I/O ?
[Attendee3] Attendee1: sounds like it
[@JonathanD] Attendee1: to switch bays 1 and 2, yes.
[@JonathanD] the “default” ethernet.
[@JonathanD] those are part of the system board and non-removable.
[Attendee1] What’s the max interconnect speed of HS20’s ?
[@JonathanD] akin to the built in nics on an xseries.
[@JonathanD] Attendee1: depends 🙂
[Attendee3] do these suckers have PCIe or AGP support?
[@JonathanD] if you have fiber channel, you’ll be using your 1 and only expansion slot for that.
[@JonathanD] Attendee3: as they don’t have slots like that, no. But at a technical level, yes.
[Attendee1] Attendee3: It’s not designed for high-speed graphics cards.
[Attendee1] (for games).
[@JonathanD] the adapters in them are PCI-X or PCI express, in the newer
[@JonathanD] to answer Attendee4’s question
[Attendee3] JonathanD: ahh okay
[@JonathanD] you can purchase a sidecar, that takes a slot
[@JonathanD] which allows you to install 1 or 2 PCI cards.
[@JonathanD] these would be accesible at the front of the blade chassis.
[@JonathanD] Attendee4: it’s an uncommon need, only there for rare uses.
[Attendee4] i dont know if i want this nice “all-in-one” enclosure to have 2 cables running off the front of the chassis
[@JonathanD] I have to wind this up in the next few minutes.
[@JonathanD] any more questions, right now?
[Attendee1] JonathanD: Okay, so HS20, I’m still not sure what the interconnect speed is between blades.
[@JonathanD] Attendee1: ahh, sorry.
[@JonathanD] if you forgo fiber, you can put a 10GbE in modules 3 and 4
[Attendee1] Does the HS20 still do the virtual network thing too?
[@JonathanD] for 20GbE, + the 2 base.
[@JonathanD] Attendee1: afraid not, it does not have the slots for it.
[Attendee1] So modules 3 & 4 would be used internally, or what?
[@JonathanD] Attendee1: internally and externally
[@JonathanD] with 1 or more 10GbE connections to external
[Attendee1] Okay, a module services how many blades? All of them?
[@JonathanD] or has the potential to
[@JonathanD] if, for example, you have san, and one doesn’t need san
[Attendee1] So redundancy aside, you could just do one module for network, right?
[@JonathanD] you don’t have to put an adapter in it.
[@JonathanD] Attendee1: yes, and some do.
[Attendee4] JonathanD: i think i need to know more about the storage, so for your next get together we should do that !!
[Attendee1] Yes, what’s the recommended way to add storage to these things?
[Attendee3] I’m still 100% confused about virtual networking.
[@JonathanD] there was some talk of a hybrid card that would fit the existing slot in an hs20 and attach to the infiniband fabric.
[Attendee3] SCSI, I’d assume.
[@JonathanD] but there was not enough customer interest in it, unfortunatly.
[@JonathanD] Attendee3: doubtful.
[@JonathanD] Attendee1: I think we’ll have to revisit storage.
[Attendee3] IEEE 1394?
[@JonathanD] I’m gonna have to get on a customer call.
[Attendee1] JonathanD: K.
[@JonathanD] Attendee3: FC or SAS
[Attendee4] well theres no way i could afford the fibre, so i will definately be going with the SAS
[Attendee1] JonathanD: Thanks.
[Attendee3] Yes, thank you JonathanD.
[@JonathanD] no problem.
[@JonathanD] hope this was helpful for all involved.