Background: Inside Intel’s Tera-scale project

Santa Clara (CA) – Last week, Intel released eight technical papers providing details about its Tera-scale project. TG Daily had an opportunity to discuss the technology with Jerry Bautista, director of technology management at Intel. Could Tera-scale become the x86 killer?

The Tera-scale project is currently has over 100 separate teams work on it. Intel is working on everything from electrical foundations all the way up to the software. Some of the research Bautista was able to share with us indicated how powerful this project is and why Intel is throwing so many resources at it. Read More on TG Daily

Freedom9 freeView IP 100 IP KVM Switch Review

Introduction

I have received for review the IP KVM switch called freeView IP 100 (KVM-01IA) made by Freedom9.

This switch is a single port IP KVM switch which means you can control the computer over the Internet or your local network even during the BIOS and CMOS settings screens.

In a home setting attaching a monitor to your computer may seem inconvenient but not out of the question. However, in some data center type settings or where the servers or computers may be too remote to easily do this the IP KVM may be an ideal solution for that kind of a situation. There’s not much that can compare to the frustration of working with a computer that has been working for many months or years and then suddenly was not work and your remote screen tools or desktop tools will not allow you to access it because the computer won’t boot and you won’t be able to see what’s on the screen. No longer will you be victim to a remote machine stuck at a “Press F1 to continue” screen!

In an ideal world your video, keyboard, and mouse would all be abstracted to the level that remoting them over TCP/IP would be a relatively easy thing and would even be built into modern computers. Until this utopian world appears products like IP KVM switches will always have an audience.

Box Contents

Inside the box the contents are rather well packed. The main unit sits between a sandwich of foam.

You may notice that there is no printed manual the manual exists only in electronic form on the CD.

Package contents include Freedom9 freeView 100, 5V 2.5A powerbrick, 6′ 9pin serial (null modem) cable, 6′ Cat5e patch cable, 3′ host-KVM straight-through cables, 8.5″ wide rackmount bracket, screws, 6′ USB cable.

Host connection side has the power, USB, 15pin VGA, PS/2 keyboard, PS/2 mouse, and pinhole factory reset button.

Close observation of the KVM switch shows that there is not only a connection for the host computer but also the console which means that you will not sacrifice your keyboard monitor and mouse access if you need access at the actual machine using the standard method.

The console connection side has 10/100 network connection, 9pin serial connection, 15pin VGA, PS/2 keyboard, and PS/2 mouse.

CD Contents

The following files are found on the CD which accompanies the product in the box.

freeView_IP_100_Manual.pdf
freeView_IP_100_QIG.pdf
freeView_IP_800_Manual.pdf
freeView_IP_800_QIG.pdf
KVMSetup.exe
KVMVisionViewer.exe

Additionally this product is not bundled with any PDF readers you will need to download one from the Internet.

Supported Video Modes

The following is a table of the supported video modes.

Resolution (x, y) Refresh Rates (Hz)
640 x 350 70, 85
640 x 400 56, 70, 85
640 x 480 60, 67, 72, 75, 85, 90, 100, 120
720 x 400 70, 85
800 x 600 56, 60, 70, 72, 75, 85, 90, 100
832 x 624 75
1024 x 768 60, 70, 72, 75, 85, 90, 100
1152 x 864 75
1152 x 870 75
1152 x 900 66
1280 x 960 60
1280 x 1024 60, 75

The manual also contains the following warning, “Do not use custom video settings besides that are not listed in the following table. Otherwise the freeView IP 100 may not be able to detect them.”

Highlights

A highlight of features as they stood out to me:

  • Remote IP KVM ability (obvious)
  • Remote mount/drive redirection (discussed more later)
  • Highly configurable (almost too configurable)
  • Management tool to manage a collection of these devices from a single screen.
  • Documentation specifically mentions attaching it infront of a larger KVM.

Documentation

I will spare you from a complete copy/paste of the manual, but it is available from the support site.

The manual lacks quality control, but is packed full of useful, and reasonably documented procedures.  Some areas with limited documentation are the modem connection ability, and the telnet pass-through to an RS-232 capable switch.

From reviewing the documentation it appears this documentation was originally created in German, and translated. The translation went well, but there are some minor loose ends like cancel buttons which still had German on them.

First Impressions

The process of removing the product from the box was enjoyable. It was a product which looked like someone cared enough to package it well. Reading over the documentation was somewhat exciting as the potential of the product become apparent. The excitement stopped there.

Problem #1 – after first connecting the product, and testing it out, I had the control window open, and went to do some other things. When I got back, there was an error which at first seemed like the product would not accept connections. After rebooting my client computer, it connected, which was a relief since it shows the product was in the client software.

Problem #2 – the default firmware on the product would not access any pre-OS screens, including startup, press any key to boot from CD, etc. This for me is a major deal breaker.

Annoyance #1 – Only registered users may download firmware updates.

Problem #3 – the firmware is the latest already which bodes ill news.

Problem #4 – Auto-adjust seems to work less wonderfully than the name implies. I got the following after hitting it a few times

Continue reading for my support experience

Support

Calling support was relatively painless. Everyone who picked up the phone spoke english, and spoke it well. No sensations of calling third-rate support centers in the middle of some foreign country.

Speaking with Ian, he directed me to some interesting areas. Using the Web Interface, this set of options made it apparent what could have been wrong with Problem #2 on the prior screen. Some computers don’t like using USB input devices pre-boot. I checked and the USB Legacy setting was set, but I still think it had something to do with the USB being used for the keyboard input.

Today problem #2 seemed resolved before I spoke to Ian, because as I was testing it to tell him about it, the keyboard was working in the BIOS screens which it was not. He told me that the unit periodically resets portions of itself depending on the scenario. Considering the computer it was connected to was off overnight, perhaps it “fixed itself” via one of those resets.

Ian suggested I also update my Java version

I didn’t make him wait on the phone, but after updating, it now shows this:

Receiving numerous questions, concerns, pointing out mistakes, Ian seemed to take each in stride, not becoming irritable or upset. Freedom9 has done well to hire him.

Coming soon, my experiences using remote media

Test Setup #1

IBM xSeries 345

  • 2×2.8Ghz Xeon
  • 4GB RAM
  • 6×146.8GB HD
  • CD-ROM
  • Floppy
  • Single USB Port
  • CentOS 4.4

Experience

Mouse interaction w/ CentOS in KDE seemed erratic, similar to using the Remote Supervisor Adapter to access the RAID management tool from IBM. When I first used the freeView 100, it would not allow me to access any of the pre-boot screens using the keyboard or mouse. I think this was related to USB being used for the inputs, and not really well supported by the BIOS, since I had USB Legacy enabled and it didn’t seem to be helping. However, later it “cleared up”. I switched it to PS/2, and it worked well then too.

This server did not seem to be able to USB boot, but when I attached a USB -> IDE adapter and tried a Knoppix CD on a DVD drive I had, it didn’t boot either, and none of the boot options had USB. I think on this one, I was just out of luck for that.

The USB device was detected by CentOS after booting though

Test Setup #2

Dell Inspiron 8600

  • 1.4Ghz Pentium M
  • 2GB RAM
  • 80GB HD
  • DVD Writer
  • 2xUSB Port on docking station
  • Docking Station VGA

Experience

The laptop had about the worst experience. The mouse movement was erratic, no matter how many times I synced it, or how many mouse options I tried, how many times I reset the Keyboard/Mouse, or the entire device, it didn’t work right. The keyboard worked, but the mouse didn’t really work right.

Additionally, the laptop didn’t seem to want to output the screen to the monitor through the VGA port during the startup, at least when the laptop screen was also visible. After starting, I was able to see, and when the monitor was blank, the soft keyboard and IP KVM keyboard access did indeed work.

Below is an unmodified example of the image quality

Considering it’s only supposed to be for administration, it’s probably not critical to have super quality. Additionally, the bytes in/out indicate this product could be used over dialup speeds.

IBM BladeCenter and DS4000 Question and Answer session

The following is the modified chat log portion of a live demo for an IBM BladeCenter and DS4000 Question and Answer session. JonathanD led the session.

[@JonathanD] I’m a certified expert in IBM bladecenter.

[@JonathanD] I’ve installed several dozen DS4000 units.

[Attendee2] JonathanD: mind if sidetrack some of your DS4000 time torward the DS3200 ?

[@JonathanD] Attendee2: I’d prefer to do that one next week, after I’ve installed one

[@JonathanD] I’d really need Brian here, if we want to do it today

[@JonathanD] and I don’t think he’s available right now.

[Attendee2] but most of my questions about LUNs and arrays would be the same anyway right ?

[@JonathanD] Attendee2: yeah, they should be.

[@JonathanD] everyone in VNC?

[@JonathanD] So, does anyone need to know more about the basics… what a SAN is?

[Attendee3] I don’t, mostly.

[@JonathanD] I’ll go over some of the very basic stuff anyway, just as a refresh and in case anyone needs it.

[Attendee2] JonathanD: is this sotrage software the same, regardless of the model ?

[@JonathanD] Attendee2: for the 4000 series, yes.

[@JonathanD] this is what you would see.

[Attendee2] ok it looks very similar to the DS3400 software doc

[@JonathanD] so, what makes a SAN different from a NAS…

[@JonathanD] Attendee2: it is

[@JonathanD] a NAS is a network share, basically, like a windows network share. different machines can use it, and store things there. You can use it with normal networking hardware, nothing special.

[Attendee3] SAN attaches via high speed fiber to a server, whereas a NAS connects via Ethernet?

[Attendee1] Generally.

[@JonathanD] A SAN, generally, is seperate, on it’s own network, and doesn’t have shares.

[@JonathanD] It has LUNs

[@JonathanD] which are assigned to hosts.

[@JonathanD] a LUN is a Logical unit number. to a server, a LUN is a physical disk.

[@JonathanD] on a scsi bus, each drive is a LUN.

[Attendee2] can any one disk (or array ) be split into multiple LUNs ?

[@JonathanD] Attendee2: yes. We’ll take a look at how that works in a moment.

[@JonathanD] so, a LUN, presented to a host, is functionally the same as a physical scsi disk installed IN that host.

[@JonathanD] thats the single biggest difference between SAN and NAS.

[@JonathanD] so, lets take a look at our san here.

[Attendee1] JonathanD: Try not to click around without giving at least a sec or two.

[Attendee1] We won’t be able to follow the screen updates.

[Attendee3] but the ds4k series doesn’t use scsi, it uses SATA?

[@JonathanD] this is a demo system. It has 2 drawers, one with 16 SATA drives, one with 10 FC drives.

[@JonathanD] Attendee3: it uses SATA or FC. FC is funtionally equiv to SCSI

[@JonathanD] from a performance perspective

[@JonathanD] but all these disks are connected to the 4000 via fiber

[@JonathanD] so, you can “see” the disks right now, right?

[Attendee2] and to the hosts via …. ?

[@JonathanD] Attendee2: fiber

[@JonathanD] on the right, we have luns.

[@JonathanD] we’re going to look at how you create a LUN and where it comes from

[@JonathanD] everyone ready?

[@JonathanD] I have about 600GB of unconfigured Fibre disk space.

[@JonathanD] as you can see.

[@JonathanD] thats those 7 drives which currently have little purple things under them.

[@JonathanD] you can see here the drive selection and raid options.

[@JonathanD] we’re going to create a 3 disk raid 5, right now.

[@JonathanD] which will give us 135GB, after parity.

[Attendee1] What’s enclosure loss protection ?

[@JonathanD] someone let me know if I’m going to fast for updates.

[@JonathanD] Attendee1: consider the following.

[@JonathanD] You have a DS4000 with 8 enclosures.

[@JonathanD] (that is, disk cabinets)

[Attendee2] like expansion units ?

[@JonathanD] you build your arrays, as 8 disk raid 5 arrays, going vertically down the cabinets

[@JonathanD] yes, Attendee2

[@JonathanD] so disk 1 in each is part of array 1, disk 2, array 2, and so on.

[@JonathanD] if you lose an entire enclosure, somehow, you still are online.

[Attendee1] Interesting.

[@JonathanD] as you have lost only one disk from each array.

[@JonathanD] that is enclosure loss protection

[@JonathanD] losing an enclosure is a RARE thing, but if a customer does have enough enclosures to do this, we will do it.

[@JonathanD] otherwise, it’s considered minor.

[Attendee3] Ah. So that little notification is telling you if those drives are located on different enclosures?

[@JonathanD] Attendee3: well, it’s telling me that TOO many drives from this array are in the same enclosures.

[@JonathanD] you can do loss protection with only 2 enclosures, too.

[@JonathanD] with raid 10

[Attendee3] makes sense. You’d need three enclosures to do raid 5

[@JonathanD] so now,wwe’re looking at the next screen. here, we define the lun size and name.

[@JonathanD] Attendee3: correct.

[@JonathanD] enclosures failing is pretty unlikely, anyway.

[@JonathanD] each has redundant power, and 2 ESMs, which are the devices that connect the drives to the controllers

[Attendee1] Okay, so here you are defining logical/virtual disks carved out of the array you just created?

[@JonathanD] Attendee1: exactly.

[@JonathanD] lets make 2

[@JonathanD] so, an array can be used by more than one host.

[Attendee2] thats perfect

[Attendee1] So you could if you wanted, make the whole thing 1 giant RAID 5 + hot spares, and carve luns out for various uses, right?

[@JonathanD] Attendee1: yes, but that would be a bad idea for more than say, 8 drives.

[Attendee1] Why?

[@JonathanD] Attendee1: raid 5 rebuild time increase rapidly past 8 drives.

[Attendee1] Really?

[@JonathanD] a rebuild that took 2 hours on 8 disks could take 8 hours on 10

[Attendee1] So like a 20 drive RAID 50, or RAID 10 would be okay right?

[Attendee2] thats really what i wanted to do was 2 RAID5 arrays and split out 3 LUNs per array to take care of my 6 blades

[@JonathanD] Attendee1: both would probably be fine.

[Attendee3] seems it’d be a better idea to create multiple raid 5 arrays and carve them out as needed.

[@JonathanD] so what were looking at now is LUN characteristics

[@JonathanD] the segment size, cache prefetch, etc.

[Attendee1] The radio buttons do what? Auto suggest geometry?

[@JonathanD] Attendee1: yes.

[@JonathanD] we generally go with cache read prefetch on, and 256 for segment size for mixed workloads

[@JonathanD] unless you have an app making small writes, it’s a good fit

[Attendee1] So like a webserver, you might use smaller segments, right?

[@JonathanD] controller ownership defines which controller owns that lun.

[@JonathanD] Attendee1: a webserver with lots of small files, sure.

[@JonathanD] 64K might be good

[Attendee1] Where webpages are typically under 20k.

[Attendee2] ok well what about something like an exchange DB ? that writes a lot of small things but large things consistantly too

[@JonathanD] 256, Attendee2

[@JonathanD] best for mixed loads like that

[@JonathanD] 128 would be acceptable too

[Attendee2] so i dont really want to use the DB radio ?

[@JonathanD] you can change it, after install

[@JonathanD] you can, Attendee2

[@JonathanD] it picks 128, which would be fine

[@JonathanD] generally you want “map later” here.

[@JonathanD] mapping to default only applies on DS4000 connected to a single server or cluster

[Attendee1] What’s a storage partition?

[@JonathanD] Attendee1: a storage parition is a segmentation of the storage system.

[@JonathanD] without ANY, all hosts see all luns.

[Attendee1] So, it’s used to divvy up luns for connecting computers?

[@JonathanD] right.

[@JonathanD] basically

[@JonathanD] the 4000 supports up to 64

[@JonathanD] a cluster with shared disk uses 1, not 1 per node.

[@JonathanD] since all nodes need the shared disk

[@JonathanD] so, we have 2 luns now

[Attendee2] Did you mean to put that on a diff controller ?

[Attendee2] I think you used A for the first one.

[@JonathanD] yes.

[Attendee2] why?

[@JonathanD] it automates back and forth

[@JonathanD] ideally, you want to balance between them.

[@JonathanD] you want equal load on each controller

[Attendee2] so if i have 4 LUNs in an array i want 2 of them to be on one controller and 2 on the other

[@JonathanD] yes.

[Attendee2] well i think ill end up with the single controller 3200 so i shouldnt have to worry about that !

[@JonathanD] (this is the san switch)

[Attendee3] Ok, what are you doing here?

[@JonathanD] well, typically we would do zoning here.

[@JonathanD] we don’t need to, because they are already there.

[Attendee3] define zoning for me, please.

[@JonathanD] zoning determines which blades and servers are allowed to see which storage systems, and tapes, and other FC things.

[@JonathanD] it’s similar to VLANs, in ethernet networking.

[Attendee1] If I remember correctly fiber channel is an explicitly defined network if you will.

[Attendee1] Not automatic like normal TCP/IP networks.

[@JonathanD] as you can see here, our HS21 blade can see DS4700_A1

[@JonathanD] which is port 1 on controller A of the 4700

[@JonathanD] here we have the masks.

[Attendee1] Can you also do RADIUS or other authentication?

[@JonathanD] no, theres no provision for that in fiber channel

[Attendee1] Is each blade in a chassis an available endpoint?

[@JonathanD] yes.

[@JonathanD] you can see here, we have an HS21 and an LS21 blade, each with it’s own ports defined

[Attendee2] and the hosts will autodetect from the chssis to the SAN switch ?

[@JonathanD] Attendee2: basically. once the zone is in place.

[@JonathanD] the 4000 sees the ports, you tell the 4000 which ports belong to which host

[Attendee1] Okay, what are we doing now?

[@JonathanD] note: this blade has NO internal disks

[@JonathanD] now, we’re going to put our new luns on a blade 🙂

[@JonathanD] and a windows boot lun, as well

[@JonathanD] I’m just taking off the existing linux boot lun and shared lun

[@JonathanD] so, we just added the lun “HS21_Lun0”

[Attendee1] gvstg01 is what kind of device?

[@JonathanD] as LUN0

[Attendee1] Is that the switch?

[@JonathanD] a ds4700

[@JonathanD] no, it’s the storage system.

[Attendee1] Okay, so it’s the FC controller we configured earlier?

[@JonathanD] right, we’re still on it now

[Attendee1] Host HS21 is the chassis w/ the blade we are trying to setup right?

[@JonathanD] so, I’m adding the 2 luns we created earlier

[@JonathanD] HS21 is the specific blade, in the chassis

[@JonathanD] in this case, blade 2

[Attendee1] Okay, so the specific blade.

[@JonathanD] I just added the 2 luns we created, as luns 1 and 2

[@JonathanD] and you can see them in the list here.

[@JonathanD] except I added them to the wrong host 🙂

[Attendee3] changing them seems easy enough.

[Attendee3] did you do that on purpose? 🙂

[@JonathanD] hehe, catch on quick, Attendee3 🙂

[@JonathanD] so here is our HS21

[Attendee3] I’m wise to the sneaky demo guy tricks..

[@JonathanD] and we’re going to hope that nobody changed whats installed on that Lun0

[@JonathanD] cause if they did we’ll have to install windows 🙂

[@JonathanD] tis the danger of an open lab 🙂

[Attendee1] So, does this device support snapshots of luns?

[@JonathanD] you can see we have 3 luns, the 3 we just added.

[@JonathanD] Attendee1: you can make a flashcopy, yes.

[@JonathanD] it’s 100% space though.

[@JonathanD] and it’s a cost option

[Attendee3] What is a flashcopy?

[@JonathanD] duplicates a lun, Attendee3

[@JonathanD] you can duplicate it to a DS4000 elsewhere, too

[@JonathanD] I just configured us to boot from lun 0

[@JonathanD] if all is well, we should be able to reboot into windows.

[Attendee2] so say i have a LUN with server 2k3 on it, i can just copy that to another LUN that i intend on using for another host for the same reason

[@JonathanD] you wouldn’t copy to a lun, a new one would be crated by the copy process

[Attendee1] You’ll need to sysprep.

[Attendee3] generate new ssids and all that, but yes.

[@JonathanD] Attendee2: there are better ways of doing it, in any event.

[@JonathanD] it’s primarily a DR feature

[Attendee2] well it would be killer if i could use 1 LUN for all hosts’ boot partition !

[@JonathanD] caching!

[@JonathanD] Attendee2: can’t do that, windows doesn’t support sharing disks like that.

[Attendee2] JonathanD: i didnt think i could, just for registry issues alonew

[@JonathanD] yet, anyway

[@JonathanD] (This is the switch in the chassis)

[@JonathanD] you’ll notice the interface is nearly identical

[Attendee3] So, things blinked past sorta quickly for me. To set up the LUN as a bootable disk, you used the utility on the blade?

[Attendee1] shrink the top console bar too.

[@JonathanD] can’t change it without kicking everyone out.

[@JonathanD] Attendee3: yes.

[@JonathanD] Attendee3: basically like setting boot order.

[@JonathanD] you can say “use this, then use this”

[@JonathanD] as you would with an addon raid card.

[Attendee1] That was part of the setup function in the QLogic controller attached to the chassis for the blades right?

[@JonathanD] the qlogic is the HBA, in the blade.

[@JonathanD] and theres our disks.

[Attendee1] So, diskless blades.

[Attendee3] Cool, so from this point, you just treat it like a normal disk.

[@JonathanD] yup

[Attendee1] Now, let’s say this Blade #2 kicks the bucket.

[Attendee1] How do you fix it?

[@JonathanD] yeah, I just formatted

[@JonathanD] Attendee1: remember when I moved it from the LS21 to the HS21?

[Attendee3] can you just slide in a new blade, reassign the luns and run with it?

[@JonathanD] Attendee3: exactly.

[Attendee1] Nice, so how much to buy this technology?

[@JonathanD] a DS4700 is gonna cost you around 40k, usually

[Attendee1] And the bladecenter + blades is how much?

[@JonathanD] the expensive part there is the SAN switches.

[@JonathanD] they’re 15k. each. for 2.

[@JonathanD] which is more than the chassis itself.

[@JonathanD] so it’s not cheap 😛

[Attendee2] Attendee1: the chassis is like $5k and the blades range from $1000 – $20,000 (for that 4x dual core AMD)

[@JonathanD] right, Attendee2

[Attendee3] So, odds are good I’ll have to stick with my HP Storageworks SA1000

[@JonathanD] how does that connect, Attendee3?

[Attendee3] fiber

[Attendee3] It was already set up when I got here, the idea was to have centralized storage for a pair of production servers, however, it never worked out that way, so now it’s just being used as a glorified external drive.

[Attendee3] It’s got four 72gig U320 drives in it.

[@JonathanD] 47K for a DS4700 with 14 500GB sata.

[@JonathanD] 7K of that was for an AIX kit for it

[@JonathanD] so 40k, yup

[@JonathanD] Now, 3400 which can get you about 3TB of storage can be had for around 15K, *WITH* the 3TB in it.

[@JonathanD] thats 3TB usable, after parity, hot spare, etc.

[Attendee1] Does IBM sell equipment without support contracts, or do they require support contracts?

[@JonathanD] almost anything you can get without support.

[@JonathanD] I’m pretty sure you can’t get a zSeries without it, though.

[@JonathanD] keep in mind, this is in an array that is still building 🙂

[Attendee3] Well, thanks for the demo, that was really informative. 🙂

[@JonathanD] Attendee1: what performance did you get pre-build?

[Attendee1] Like 8MB/sec or something.

[@JonathanD] so, pretty competative ehh? 😉

[@JonathanD] no problem, Attendee3

[@JonathanD] hope everyone enjoyed it.

[@JonathanD] oh, one more thing.

[@JonathanD] Attendee1: there is your perf, on just 3 drives, degraded.

[Attendee3] good demo. I’m assuming the LUN concepts will translate to other SAN devices?

[@JonathanD] yup

[Attendee1] JonathanD: Thanks, I think I learned a bunch.

IBM ServeRaid SCSI Performance Bottleneck – Part 3

On a recommendation from a friend, I was asked to run iometer to do some benchmarks, to see how well the server would handle server loads instead of just sequential read/writes.

While I realize IOps is a rather arbitrary benchmark, I think if you keep block sizes the same, and the test conditions as close to the same as you can, then it at least has relative perfomance implications. For more details on this subject, read “What Is Good Performace” – Brief by Imperial Technologies

Using 32KB block sizes, 1 worker, on an idle server, 2 processor w/ hyper threading enabled on both processors.

Updated: 2007-03-24 05:52 EDT Added benchmark results of 5×146.8GB 15k RPM HD’s in RAID 5EE (striped + hot spare striped in)

Updated: 2007-03-29 07:45 EDT Added benchmark results of 6×146.8GB 15k RPM HD’s in RAID 5EE (striped + hot spare striped in)

Updated: 2007-03-29 11:27 EDT Added benchmark results of 6×146.8GB 15k RPM HD’s in RAID 0 (yes I checked if something was borked, I used the same install for all my last benches, just did a restore.), Also added 6×146.8GB 15k RPM HD’s in RAID 10 (3xRAID1, then RAID 0 them together)

Updated: 2007-04-31 10:48 EDT Added many benchmarks, and revised the naming conventions

Naming convention, starting with boot array
<array type>-<Raid level>-<NumDisksxDiskSize@DiskSpeed>[, ...] - <Number of partitions in test>[S if synchronized[? if I don't remember]][B if include boot partition] [Additional notes]

Assumption, each partition given it's own worker, all tests against ServeRaid 6i unless otherwise noted
... is used if the array configuration is the same as the last one mentioned.

I found out some interesting things:

  1. Make sure your high performance loads are *NOT* reliant on the boot partition for read/writes
  2. Make sure you synchronize any array mode that needs it before you start benchmarking or it’ll perform in degraded mode (AKA bad performance). For the ServeRaid 6i, just load up the ServeRaid Manager software, and it’ll say when it’s synchronized.
Setup IOps Read IOps Write IOps
RAID-0-2×146.8GB@15kRPM, … – 2 6636.96 4905.30 1731.66
RAID-0-2×146.8GB@15kRPM – 1B 2927.29 2161.56 765.73
RAID-0-2×146.8GB@15kRPM – 1 5287.23 3911.03 1376.20
RAID-0-2×146.8GB@15kRPM, …, … – 4B 5366.80 3972.18 1394.62
RAID-0-2×146.8GB@15kRPM, …, … – 3 6967.94 5153.94 1814.00
RAID-0-2×146.8GB@15kRPM, RAID-0-4×146.8GB@15kRPM – 1 5205.37 3852.88 1352.49
RAID-0-2×146.8GB@15kRPM, RAID-5EE-4×146.8GB@15kRPM – 1 1007.43 746.18 261.26
RAID-0-2×146.8GB@15kRPM, RAID-5EE-4×146.8GB@15kRPM – 1S 5311.35 3933.50 1377.86
RAID-5EE-6×146.8GB@15kRPM – 1SB 3532.33 2612.30 920.03
RAID-5EE-6×146.8GB@15kRPM – 1S 5354.84 3966.74 1388.09
RAID-5EE-6×146.8GB@15kRPM – 2S 6993.74 5173.16 1820.58
RAID-5EE-6×146.8GB@15kRPM – 3S 7214.27 5338.27 1876.01
RAID-5EE-6×146.8GB@15kRPM – 4S 7360.53 5444.32 1916.21
RAID-5EE-6×146.8GB@15kRPM – 4S 1024 sector test size 7352.68 5437.56 1915.12
RAID-5EE-6×146.8GB@15kRPM – 5S 7061.73 5225.83 1835.90
RAID-10-6×146.8GB@15kRPM – 1S?B 3119.32 2307.67 811.65
RAID-0-6×146.8GB@15kRPM – 1B 288.84 213.99 74.84
RAID-5EE-6×146.8GB@15kRPM – 1S?B 2923.77 2160.64 763.13
RAID-5EE-5×146.8GB@15kRPM – 1S?B 2920.75 2161.85 758.90
RAID-0-2x36GB@10kRPM – 1B 2756.12 2039.76 716.36
JBOD-0-1x36GB@10kRPM – 1B Onboard LSI 140.94 104.65 36.29
Setup MBps Read MBps Write MBps
RAID-0-2×146.8GB@15kRPM, … – 2 207.40 153.29 54.11
RAID-0-2×146.8GB@15kRPM – 1B 91.48 67.55 23.93
RAID-0-2×146.8GB@15kRPM – 1 165.23 122.22 43.01
RAID-0-2×146.8GB@15kRPM, …, … – 4B 167.71 124.13 43.58
RAID-0-2×146.8GB@15kRPM, …, … – 3 217.75 161.06 56.69
RAID-0-2×146.8GB@15kRPM, RAID-0-4×146.8GB@15kRPM – 1 162.67 120.40 42.27
RAID-0-2×146.8GB@15kRPM, RAID-5EE-4×146.8GB@15kRPM – 1 31.48 23.32 8.16
RAID-0-2×146.8GB@15kRPM, RAID-5EE-4×146.8GB@15kRPM – 1S 165.98 122.92 43.06
RAID-5EE-6×146.8GB@15kRPM – 1SB 110.39 81.63 28.75
RAID-5EE-6×146.8GB@15kRPM – 1S 167.34 123.96 43.38
RAID-5EE-6×146.8GB@15kRPM – 2S 218.55 161.66 56.89
RAID-5EE-6×146.8GB@15kRPM – 3S 225.45 166.82 58.63
RAID-5EE-6×146.8GB@15kRPM – 4S 230.02 170.13 59.88
RAID-5EE-6×146.8GB@15kRPM – 4S 1024 sector test size 229.77 169.92 59.85
RAID-5EE-6×146.8GB@15kRPM – 5S 220.68 163.31 57.37
RAID-10-6×146.8GB@15kRPM – 1S?B 97.48 72.11 25.36
RAID-0-6×146.8GB@15kRPM – 1B 9.03 6.69 2.34
RAID-5EE-6×146.8GB@15kRPM – 1S?B 91.37 67.52 23.85
RAID-5EE-5×146.8GB@15kRPM – 1S?B 91.27 67.56 23.72
RAID-0-2x36GB@10kRPM – 1B 86.13 63.74 22.39
JBOD-0-1x36GB@10kRPM – 1B Onboard LSI 4.40 3.27 1.13
Setup Average Response Time Maximum Response Time
RAID-0-2×146.8GB@15kRPM, … – 2 0.30 18.37
RAID-0-2×146.8GB@15kRPM – 1B 0.34 18.42
RAID-0-2×146.8GB@15kRPM – 1 0.19 18.24
RAID-0-2×146.8GB@15kRPM, …, … – 4B 0.74 19.22
RAID-0-2×146.8GB@15kRPM, …, … – 3 0.43 18.62
RAID-0-2×146.8GB@15kRPM, RAID-0-4×146.8GB@15kRPM – 1 0.19 18.47
RAID-0-2×146.8GB@15kRPM, RAID-5EE-4×146.8GB@15kRPM – 1 0.99 30.16
RAID-0-2×146.8GB@15kRPM, RAID-5EE-4×146.8GB@15kRPM – 1S 0.19 18.40
RAID-5EE-6×146.8GB@15kRPM – 1SB 0.28 18.49
RAID-5EE-6×146.8GB@15kRPM – 1S 0.19 18.09
RAID-5EE-6×146.8GB@15kRPM – 2S 0.29 18.31
RAID-5EE-6×146.8GB@15kRPM – 3S 0.41 18.45
RAID-5EE-6×146.8GB@15kRPM – 4S 0.54 18.70
RAID-5EE-6×146.8GB@15kRPM – 4S 1024 sector test size 0.54 18.70
RAID-5EE-6×146.8GB@15kRPM – 5S 0.71 18.71
RAID-10-6×146.8GB@15kRPM – 1S?B 0.32 18.50
RAID-0-6×146.8GB@15kRPM – 1B 3.46 44.39
RAID-5EE-6×146.8GB@15kRPM – 1S?B 0.34 18.59
RAID-5EE-5×146.8GB@15kRPM – 1S?B 0.34 18.37
RAID-0-2x36GB@10kRPM – 1B 0.36 20.59
JBOD-0-1x36GB@10kRPM – 1B Onboard LSI 7.09 26.09
Setup % CPU Utilization % User Time % Privileged Time % DPC Time % Interrupt Time
RAID-0-2×146.8GB@15kRPM, … – 2 5.14 0.37 4.79 0.75 1.93
RAID-0-2×146.8GB@15kRPM – 1B 3.84 0.19 3.65 0.68 1.02
RAID-0-2×146.8GB@15kRPM – 1 3.49 0.44 3.10 0.50 1.10
RAID-0-2×146.8GB@15kRPM, …, … – 4B 6.16 0.46 5.71 0.89 1.21
RAID-0-2×146.8GB@15kRPM, …, … – 3 5.81 0.39 5.43 0.85 1.61
RAID-0-2×146.8GB@15kRPM, RAID-0-4×146.8GB@15kRPM – 1 3.49 0.14 3.38 0.67 1.29
RAID-0-2×146.8GB@15kRPM, RAID-5EE-4×146.8GB@15kRPM – 1 0.78 0.10 0.70 0.08 0.31
RAID-0-2×146.8GB@15kRPM, RAID-5EE-4×146.8GB@15kRPM – 1S 3.73 0.21 3.55 0.59 1.36
RAID-5EE-6×146.8GB@15kRPM – 1SB 3.97 0.28 3.72 0.72 1.28
RAID-5EE-6×146.8GB@15kRPM – 1S 4.28 0.51 3.79 0.58 1.43
RAID-5EE-6×146.8GB@15kRPM – 2S 5.36 0.40 4.99 0.79 1.98
RAID-5EE-6×146.8GB@15kRPM – 3S 6.14 0.51 5.66 0.69 1.88
RAID-5EE-6×146.8GB@15kRPM – 4S 7.05 0.63 6.48 0.78 1.64
RAID-5EE-6×146.8GB@15kRPM – 4S 1024 sector test size 6.86 0.55 6.29 0.78 1.77
RAID-5EE-6×146.8GB@15kRPM – 5S 7.75 0.72 7.07 0.86 1.60
RAID-10-6×146.8GB@15kRPM – 1S?B 4.44 0.15 4.32 1.02 1.29
RAID-0-6×146.8GB@15kRPM – 1B 0.43 0.01 0.45 0.10 0.17
RAID-5EE-6×146.8GB@15kRPM – 1S?B 4.37 0.08 4.30 1.11 1.23
RAID-5EE-5×146.8GB@15kRPM – 1S?B 8.00 0.28 7.74 1.51 2.05
RAID-0-2x36GB@10kRPM – 1B 5.17 1.29 3.89 0.67 1.07
JBOD-0-1x36GB@10kRPM – 1B Onboard LSI 0.34 0.04 0.32 0.04 0.19
Setup Interrupts per Second CPU Effectiveness Packets/Second
RAID-0-2×146.8GB@15kRPM, … – 2 6903.88 1291.11 629.38
RAID-0-2×146.8GB@15kRPM – 1B 5353.16 763.13 411.99
RAID-0-2×146.8GB@15kRPM – 1 5561.51 1513.28 417.54
RAID-0-2×146.8GB@15kRPM, …, … – 4B 2161.98 870.81 1049.38
RAID-0-2×146.8GB@15kRPM, …, … – 3 4821.06 1199.72 841.91
RAID-0-2×146.8GB@15kRPM, RAID-0-4×146.8GB@15kRPM – 1 5478.78 1492.11 416.31
RAID-0-2×146.8GB@15kRPM, RAID-5EE-4×146.8GB@15kRPM – 1 1273.59 1299.26 418.36
RAID-0-2×146.8GB@15kRPM, RAID-5EE-4×146.8GB@15kRPM – 1S 5587.42 1424.44 411.12
RAID-5EE-6×146.8GB@15kRPM – 1SB 7332.83 889.47 413.65
RAID-5EE-6×146.8GB@15kRPM – 1S 5627.52 1251.87 414.72
RAID-5EE-6×146.8GB@15kRPM – 2S 7267.00 1303.97 635.57
RAID-5EE-6×146.8GB@15kRPM – 3S 5148.84 1175.41 837.93
RAID-5EE-6×146.8GB@15kRPM – 4S 3862.49 1044.00 1041.51
RAID-5EE-6×146.8GB@15kRPM – 4S 1024 sector test size 3859.17 1071.43 1046.56
RAID-5EE-6×146.8GB@15kRPM – 5S 2841.27 911.55 1239.27
RAID-10-6×146.8GB@15kRPM – 1S?B 5866.90 702.51 415.26
RAID-0-6×146.8GB@15kRPM – 1B 616.43 666.05 419.56
RAID-5EE-6×146.8GB@15kRPM – 1S?B 5417.13 669.64 415.93
RAID-5EE-5×146.8GB@15kRPM – 1S?B 8079.69 365.18 3072.45
RAID-0-2x36GB@10kRPM – 1B 5057.38 532.62 404.83
JBOD-0-1x36GB@10kRPM – 1B Onboard LSI 688.81 409.75 395.23

By looking at these charts, it’s pretty apparent that while the prior articles ( Part 1 , and Part 2 ) show the RAID card does not match up to sequential speeds of non-RAID use, under server loads, it really shines.

I guess one could say this is a good example of using the right benchmark for the job. Maybe a comparable analogy would be benchmarking 0-60 times for a regular car, or a semi tractor. Take that semi, and put a box car behind it, and do the same for the car, and then we get the same outcome as using iometer.

For those interested: the iometer configuration file I used

As an aside, later today, I should be getting some 15k RPM drives in. This should make for some interesting additions to this mix.

IBM ServeRaid SCSI Performance Bottleneck – Part 2

Interesting times. I got my ServeRaid 6i, installed it, but I’m having similar issues with performance that my friend did with his ServeRaid 4LX in part 1 of this article.

My system is a bit different from my friend.

  • IBM xSeries 345
  • 2GB Registered ECC PC2100 (2x1GB)
  • 2x36GB 10K RPM U320
  • 2×2.8Ghz Xeon w/ Hyperthreading (Prestonia core)

Here’s a sample of the card right after install of the card, and W2k3 Std R2, and some simple patches using 2x36GB 10k RPM U320 hard drives in RAID 0 in 8KB stripe.

This benchmark was done with the drivers which installed without doing anything. Fairly pathetic if you ask me.

Let’s see what we can do to increase the performance. Let’s try getting the drivers and firmware updated to 7.12 for both.

Same spec, but drivers and firmware at 7.12:

Okay, a little better, but not really that much.

Now, I backed up the configuration, and changed the stripe size to 64KB (max on this card). Unfortunately this can’t be done with a bios prompt, but you have to run the bootable ServeRaid manager (Basically a bootable Linux distro).

The average performance is much better, but it’s still all over the place. I’d like to see some normal looking decay. And those valleys into the 30’s worries me.

In a bit, I’ll benchmark single disk performance, and maybe even try testing in Linux.

Updates

Continue to Part 3

IBM ServeRaid SCSI Performance Bottleneck – Part 1

Speaking with a friend of mine on FreeNode.net, he had pointed out an interesting apparent performance bottleneck.

His server while quite old (x232, 8668-54X) had three 18GB 10k RPM drives.

He sent me the following picture which the performance metric seemed out of whack for several reasons.

Am I reading that right? 29.6 MB/sec average read ? My external USB can do that on a parallel ATA 7200 RPM drive. This was THREE 10k RPM SCSI drives in RAID 0.

Another point which caught my eye was the fact that there was no decay in the the performance. Most drives decay in performance as you reach the center.

So, I had him try a larger stripe size. The one above is 8KB stripe, which is a bit small.

RAID 0 with 64KB stripe looks like this:

About 9MB/sec gains, but still a *LOT* less than I would expect. My RAID 0 with 2x320GB 7200 SATA seagates gets around 95MB/sec.

Also, I noticed there’s still no decay in performance.

So, for kicks, I asked him if he wouldn’t mind trying RAID 5 with 64KB stripe size.

Read ended up 2.5MB/sec slower, and still no decay in the performance over the entire partition.

This is really odd. It looks like there’s some sort of bottleneck we may not know about, but considering he has the latest firmware for his card, and latest drivers you would expect to see something much higher.

If anyone has any thoughts, please post some comments.

Update 2007-03-10 09:44AM EST

My friend sent me the following screen shot to verify that he was indeed using the latest drivers from IBM

Also, I was able to find some other people complaining about the same issue.

  1. Bad RAID performance scores at forums.2cpu.com
  2. RAID Guru Help Required at forumz.tomshardware.com

One thing I asked my friend to do is to post some software RAID results.

If you have any helpful information, please post some comments.

Update 2007-03-10 12:58PM EST

I had my friend perform some tests using his three 18GB disks with one disk for booting, and 2 as dynamic RAID 0.

Interestingly HD Tach seems to cut through the Dynamic Disks feature, and benchmark the devices individually causing single drive and RAID 0 to both benchmark like this:

Now that performance is a *LOT* more inline with what I expected to see. Good single drive starting performance, and decay over the drive surface.

Okay, now that we know single drive is outperforming *every* 3 disk RAID he could setup using the RAID card, let’s see how dynamic disks fared using a benchmark tool which did not separate the dynamic disks and test them individually. To do this, we had to use HD_Speed:

Wow, isn’t that something? Software RAID performing about 3x as fast as his original RAID 0 performance using a RAID card.

My friend asked me if this basically meant hardware RAID sucked. I’d have to say no, but it does imply this card does. I’ll be getting my own card from a similar line (ServeRaid 6i) and I’ll benchmark that too. I hope I don’t get similar benchmarks, or I’ll have to sell it to someone else, and use software RAID. I’d rather do a 2 drive RAID 1, and a 4 drive RAID 5 using software RAID if I can’t get at least 150MB/sec sequential reads using the RAID card. I have a feeling I won’t be getting that level of performance with the ServeRaid based on the threads above, and my friend’s experience. Such a pity for IBM to release this flop of a product (I would invite anyone from IBM to set me straight, if I’m missing something).

Again, any comments, please post them.

Update 2007-03-16 09:40AM EST

Added IBM ServeRaid SCSI Performance Bottleneck – Part 2