Lefthand SAN + HP = :(

My work place was an early adaptor of the iSCSI SAN technology. We started with Lefthand Networks back in 2005 and were very pleased with their products and continued to to grow our cluster adding a node or two every year which increased performance and storage capacity. This worked out great since our budget didn't allow us to spend 100K on the other SAN technologies available at the time. We could get the small 1 TB raw capacity NSM 150/160 SATA units for ~13,000 at the time.  We used it for simple stuff like file servers but eventually moved our Exchange 2003 DB's to it after our mail servers local storage got low ( 2 DB's supporting about 2500 mailboxes and it ran like a dream). Thin provisioning and dynamically expanding drive size is pure win ( if you've never used it you really don't know what you're missing).

Eventually we deployed Vmware ESX and our SAN storage was suddenly looking pretty full.  We started another cluster of larger storage units (the NSM 2060 3 TB SATA) for about the same cost per unit ~13,000. These also changed from the custom Lefthand chassis for the hardware to basically a Dell 2950 chassis. We're primarily a Dell server shop so we were thrilled and thought maybe a purchase would happen by Dell.

Fast forward to fall 2008 and HP's purchase of Lefthand.  I'm personally not a huge fan of HP servers so I had some worries about what was going to happen but was optimistic. With any transition there are bound to be some bumps in the road but HP made some exceptionally stupid moves like taking down the Lefthand networks customer portal ( where I got all my tech notes and patches ) before they had an equivalent site up on HP.com.  This came at a particularly bad time for us as we needed to get some patches to complete an upgrade to our cluster. Lefthand tech support was always excellent and you usually talked to the same 4 or 5 people when you called in. They were very helpful during the transition period and still got me the files and info I needed. I got the upgrade completed and forgot about the HP situation until it came time for us to buy another node for the cluster.  Since Dell is a competitor, HP quickly changed the chassis for the node we were using to a DL380. The price of these units continued to drop so this wasn't a real big deal 3 TB raw iSCSI SAN for ~10 K was a pretty sweet deal. Now we have a P4500 as well as the 2060's. We also purchased a cluster for our DR site to get some remote snapshots going, 3 P4300 6 TB SATA nodes still very affordable at about 12K each. While in the testing phase we had a hard disk go bad ( maybe the 2nd disk out of all the nodes in our clusters since we've had the things, which is impressive for SATA disks, and also a bit sad for HP starting out). We also came to find that snapshots would cause performance issues in the cluster if you had too many per volume ( I think we had about 4 per volume on about 30 volumes ), we started having managers go down and I/O issues. Luckily we had enough redundant managers that nothing went unavailable but performance was a real issue until we got the snap shops cleaned up. This was also the first time we had to get in contact with support since HP changed things. We were waiting for call backs from engineers instead of getting right through to tech support like in the good ole days.  The same staff seemed to be there in the end though so our problems were resolved. Once we got a daily snapshot schedule set things seem to run like clockwork.

Most recently we got struck by a firmware problem with the RAID controller in the HP chassis. Again the Lefthand redundancy saved the day when a node went down. A RAID restripe took place for about a day after we flashed the firmware and got the bad unit back up. We also recently noticed we were bumping the performance ceiling on I/O and had to shuffle some stuff around between our two clusters. This brought us to the point of ordering another node this year.

Welcome to confusionville, population me and probably every other customer of Lefthand SANs. Lefthand has always had a weird reseller program called preferred vendor pricing. This basically gives the reseller that you first opened your account with Lefthand through better pricing than other resellers ( unless they apply to change the preferred pricing to their company which than causes your initial vendor to question why they lost the preferred pricing). This creates problems if you ever try to get competitive quotes from multiple vendors. On top of this I came to find out effective 3/31/2010 HP is discontinuing all SATA SAN models and also the 3 TB capacity that our NEW cluster is based on. Had a meeting with our HP rep and our local SAN engineer where they basically told me they're going to offer slower speed SAS drives that are equivalent to the SATA disk drives but since the capacity chassis that I use is no longer offered the best I can do is buy the larger 9 TB chassis and just not use the excess capacity. Grrrrr. Option B is to start another new cluster. This kind of hits at a good time because I'd been considering starting a cluster of SAS storage for our higher I/O apps like VMware and Exchange 2007 ( now running 4 DB's for the 2500 mailboxes) plus we're looking into VDI which I hear can beat the hell out of SAN I/O.

I'm quite frustrated with HP that they've kind of screwed me with my mid level cheap storage so I started looking at Equallogic from Dell. There is some chatter out on the internet comparing these two competing SAN vendors. This site is collecting all the info together and is very helpful if you're trying to make this decision yourself. The management and monitoring tools of EQ I think look better than what Lefthand currently has to offer. EQ has software to collect and allow reporting on performance history whereas lefthand basically just has current performance metrics available in the console but no history.  I'd give my left hand ( lol) to get HP to rewrite the console in something other than java. Maybe make a web interface that is clientless ? It takes like 3 minutes to open and log into my management console as it goes out and collects config info from all of my 14 nodes in one management group. I can't imagine how long it would take in a really large environment.  Price wise the EQ and Lefthand are looking about the same for performance but you may be able to eak out more usable space by changing RAID types in the EQ. As a long time Lefthand user I'm pretty comfortable with the network RAID they use and the redundancy it gives if an entire node goes down. I don't know how I feel about all my SAN being in one box ( even though EQ claim everything is redundant ). I will say that the maintenance on all my Lefthand nodes is getting kind of ridiculously expensive but those 9 NSM 160's are reaching end of life so it might be time to drop it and consolidate to a bigger higher performing disk unit. Below is some pricing I've gotten recently and the supposed I/O that the units provide. Hope this help someone, I'm just kind of brain dumping here to help me make a decision on sticking with the enemy I know or moving to the enemy I don't plus having to migrate all the data to a new SAN and buy another EQ unit for our DR site as well......... this is sounding extra expensive. Ah if only this had happened before we established our DR site.

This pricing may vary as our Dell rep was jumping through some hoops to get us pricing to fit what we had budgeted for ( which was not EQ)
EQ PS4000XV 15K SAS 16x600 GB - ~38 K  ~ 1800 I/O*

EQ PS6000XV 15K SAS 16x600 GB - ~50 K ~ 1800 I/O*

* I'm not quite clear on the differences between the 4000 and 6000 series and didn't get a real clear answer about the I/O provided by the 6000 but logic would dictate that the same number and speed of disks in each unit would provide about the same max I/O

Lefthand Virtual SAN bundle

2 Nodes P4500 G2 12 x 450 GB SAS - ~52K ~ 3400 I/O

You can buy single P4500 to add to this cluster to increase capacity and I/O. HP really seems to stick it to you with the support cost for 3 years for this one being almost 6K vs about 3K for the bundle package listed above.

1 P4500 G2 12x 450 GB SAS - ~34K ~1700 I/O

From a storage perspective it looks like EQ would be the way to go but from an I/O perspective I can get more I/O for my dollar from Lefthand which is not what I was expecting.  The tools and optional RAID configs that EQ offers may make up the difference but my established Lefthand environment and previous investments may overcome my current loathing of HP and keep me as their customer.

Infopath code signing cert and Windows 2003 Standard CA

A developer where I work recently came to me for help generating a code signing certificate he could use in an Infopath 2007 form that he could then publish in our MOSS 2007 environment. We don't have the enterprise version of MOSS 2007 or we could just publish the Infopath form directly onto the sites without users needing the Infopath client installed.

A couple issues popped up while trying to accomplish this. The default code signing template in our Windows 2003 Standard CA didn't allow you to export the private key. I found a MS blog that gave instructions for creating a copy of the code signing template and allowing it to export private keys but when we did this we noticed that it only supported Windows 2003 Enterprise CA's. Rather than deploying another server I called MS support and they were able to help me manually create the certificate through the command line. Here are the steps we used:

1. Create a document called request.inf and place the following code in the file (filling it in with the appropriate info for your org)

----------------------------------------

[Version]
Signature="$Windows NT$

[NewRequest]
Subject = "CN=server.yourdomain.com" ; must be the FQDN of domain controller
EncipherOnly = FALSE
Exportable = TRUE  ; TRUE = Private key is exportable
KeyLength = 1024    ; Common key sizes: 512, 1024, 2048,
 ;    4096, 8192, 16384
KeySpec = 1             ; Key Exchange
KeyUsage = 0xA0     ; Digital Signature, Key Encipherment
ProviderName = "Microsoft RSA SChannel Cryptographic Provider"
ProviderType = 12
RequestType = CMC

; Omit entire section if CA is an enterprise CA
[EnhancedKeyUsageExtension]
OID=1.3.6.1.5.5.7.3.3 ; Code Signing

[RequestAttributes]
CertificateTemplate = codesigning ;Omit  line if CA is a stand-alone CA
SAN="dns=.yourdomain.com&dns=ldap.yourdomain.com"

------------------------------------------------------------------------------

2. On your CA server open a command prompt and issue the following commands

               certreq -new c:\request.inf c:\new.req

               certreq -submit c:\new.req

3. You will be prompted to choose your CA and then to save the certificate. Once it is saved run the following command.

               certreq -accept c:\codesigning.cer

You should now have a certificate with private key that you can use wherever you like.

Install the Cert locally on the CA, then open certmgr.msc, export the cert with private key to a P7B file. This can then be provided to your code monkeys to do with what they will.