Initially, we were looking for a 10GE-iSCSI storage solution that would do synchronous or at least memory-synchronous mirroring of data to a second system and automatic failover. We planned to use the system as storage backend for a few dozens VMs, and wanted the storage to be highly available on a shared IP. Active-Active supported seemed pretty awesome too, and the system should allow seamless failover.
However, some vendors didn’t provide this option, others were too expensive and the project was an open bidding, so we had to be cheap. The only viable options seemed to be building something ourselves or buying Open-E from a distributor that would take care of the hardware part. They were also offering 24/7 support. It sounded pretty good, and our distributor was saying he did a couple of installations with it, so we went for it. The system consisted of:
- 4x 10GE interfaces, 4x 1GB Management interfaces
- LSI Megaraid 9271-4i
- 24x 600GB SAS 15K disks, configured as RAID6
- Supermicro Chassis, IPMI
- E5-2609, 16GB DDR3-1066
- Installed on an USB Stick (the mainboard has an internal slot) by our distributor
However, Open-E now says something which is pretty unfortunate:
We recommend installing the software to a hard drive, SSD, or a small Logical Unit (volume) in your RAID. The following RAID controllers are supported as a bootable media: MegaRAID, Smart Array, 3ware, Adaptec and Areca. We do not recommend installing the software to devices connected via USB, as it can cause system instability.”
After asking the support team, they said that it’s ok to still use USB, they just had lots of problems with failing USB sticks.
Now here is a list of some of the problems we’ve had with the system:
- After upgrading to a newer software version, the system would get stuck in a reboot loop (up10 to up11).
- With another software release, it just wouldn’t boot anymore and was stuck in a black screen, blinking cursor (up11 to up12_b9358). It’s not possible to install on an existing USB stick, you have to null it and re-install.
- We’ve had problems with faulty checksums, even after singling out USB-stick hardware failure (we tried sticks from different vendors, and different sizes). Sometimes the corruption just seemed to happen after reboots.
- Several times after upgrades or reinstalls, you enter your license key, but after some while it switches back to evaluation mode. Once, the cluster stopped working. It seems Open-E support had only “fixed” it for so long. (Update: This may have been fixed in newer versions, as far as I remember it happened mostly with up10.)
- The server headers expose that it’s running Debian Etch on Apache with PHP. (“X-Powered-By: PHP/5.2.0-8+etch13”) – it’s EOL, and doesn’t have any security support.
- Shrinking logical volumes is not possible
- Naming logical volumes names is not possible, they are called lvXXXX and the number increases automatically (Update: this is now on their roadmap)
- When configuring syslog, you can only set an IP, not to a DNS name. Come on, this is 2014, and I want some flexibility. We use DNS everywhere and this is pretty inconvenient.
- Every time I try a new software version (up7, up10, up11, RC1 for up12) I have the feeling that new bugs were introduced 🙁
- Why do I have to have two 10GE-lines between the Systems instead of just one? It’s not possible to create the cluster otherwise.
- The default route of the management interface (or any other network configuration) is not changeable after the cluster has started. If you change anything (network, or default gateway), you have to completely stop your HA-Cluster beforehand, which is very, very stupid
- You can define several ping nodes. If those are unreachable, your cluster will switch over. However, if the second system can’t reach them either, the cluster will completely stop and both systems think they are the replication destination. So you have to stop everything, and change one of the systems to replication source for all replicated storage. This sounds easy, but can take an hour if you have 30 iSCSI targets.
- Support told me, if both “aux” connections are still there, the cluster won’t switch if the ping-nodes are unreachable. This isn’t true, unfortunately.
- No LLDP support
- After setting up a completely new version of Open-E on a server, interfaces suddenly switched. This is pretty annoying and inconvenient without LLDP and 8 network interfaces. You can rename the interfaces, but only with a custom patch supplied by the support.
- SSL Certificates get newly generated on every boot. This is pretty annoying and – in my opionion – undermines the purpose of SSL.
- We’ve found a XSS-Bug in the Webinterface and reported it. The reply was “we already know this” and the issue was closed immediately. We never got any feedback. Well, who needs stuff like CVEs anyways?
- A colleague told me, it was/is possible to download some setup/configuration files without authentication (they contain passwords). I still need to verify this with the latest version.
- Passwords are saved in cleartext. When you’re filing a bug, you need to upload the logfile bundle via HTTP – which means you transmit your passwords over HTTP in cleartext. 🙁
- Failover works like a charm when there are not too many targets connected. Writing to the SAN with 10GBit/s and switch to the secondary system? No problem!
- DO NOT TRY THIS if you have more than a handful of targets! The SCST daemon has a bug (that’s what the support said) and is very unstable – your failover will fail! This is said to be fixed in RC1 for up12,
I’m about to test it.Update: It’s fixed in a way that failover now takes about 15-20 minutes. At some point, it just works again. Sometimes it takes only 5 minutes, then again 20 minutes. Unfortunately, all reads and writes are blocked in the meanwhile and all your VMs will probably have crashed. Also, 20m is not an acceptable time for a failover.
- UPDATE: It’s not possible to replicate a whole VG between two systems. This would make setup much easier and failover times shorter (if you want to use that feature, that is!)
- The WebGUI has great messages like: “Failover cluster could not perform requested operation (error code #Ts1000).” In the log it says: “Failed to start the cluster, some internal error occurred.” Now please guess what might be wrong?
- WebGUI Performance is often pretty bad. Example: create a target, create 32 logical volumes. Go to Configuration -> iSCSI target manager -> Targets, click on your target. It takes a pretty long time until the GUI loads the available logical volumes. I reported this, but it’s probably not getting fixed.
- UPDATE: After the first installation of RC1 for up12, it seems that the apache webserver restarts every few seconds, leading to disconnects, unexpected WebGUI behaviour and you having to accept new SSL certificates all the time. This is mostly gone after a reboot.
- Open-E does not scale. When you have more than a few targets, you will notice iSCSI lockups; logging in to several targets at once? Lockup, timeouts. Trying to re-establish a connection after network outage? Lockups, timeouts. Connecting from several hosts to the iSCSI target? Lockups, timeouts.
(This is said to be fixed in RC1 for up12, I’m about to test it.)Update: Tested, now the iSCSI will only be available if it works fine again – which can take 20m.
- We’ve had several outages because of these problems and troubleshooting them with the support was very hard.
- Our distributor said: “Having several targets is not really supported or tested”. Our question: “Then how am I going to add new VM storage to the cluster, it’s only possible with new targets, if I don’t want to stop an online target and attach new LUNs?” was answered by: “Use one single, big target and store everything in there with the vmware container format. Or use NFS.”. But we use KVM. And we might want CHAP and IP security for the targets.
- Update: It’s really annoying that you can’t add additional LUNs to a target that’s in the cluster. This prevents using that feature as workaround for the stability issues with many targets. Also, it’s a better solution in general. We’ve only tried running with many targets because this important feature is missing!
- If you have to replicate all data to the second system, it takes ages. For some reason, the system only uses 1GBit/s instead of available 10GBit/s. I checked replication settings, speed settings etc., but there was no fix. (There is now one fix in RC1 for “replication speed”, this
might beUpdate: is not fixed)
- The support is horrible sometimes. They sometimes contradicted what other support members had said, or had terrible advice.
- Sometimes the support does not really read what you’re saying and asks extremely stupid questions.
- Their support uses skype for calling you and was extremely hard to understand. One time we decided that chatting was the better choice.
- But once, I had a competent support employee who spotted a problem with one of our network cards pretty quickly! The reason for this bug is still unclear, one of the 10GE ports of one of the cards stopped working with 10GE after an Open-E upgrade. However, he was letting me watch his desktop and was working on another assignment while he waited for me, and was browsing configuration templates for a customer with lots of IPs and passwords in it for several minutes and would’ve given me plenty of time for screenshots…
- 24/7 support cannot use the RIS (remote connection fpr troubleshooting) and if you have Linux on your workstation, it’ll be problematic to let them access the system (they want you to open a teamviewer session or something alike).
- iSCSI performances was really good, when the system worked. It’s not really something that was done by Open-E but by the hardware which is pretty good.
- The API is a really good idea and worked well for me. Unfortunately, some commands are missing for no reason, for example you can create an iscsi volume, but you can’t delete it. Also, “iscsi_target_assign” didn’t work for me, so I filed a bug report. Update: iscsi_target_assign works fine, this problem was caused by another bug in Open-E.
- Update: creating, starting and stopping of replication tasks is missing in the API, which is very annoying, because you can’t automate creating a clustered target.
- It’s not possible to use passworded SSH keys for access
I think Open-E had some pretty good ideas for a good price, like the API and active-active Failover. Unfortunately, I’ve tripped over so many bugs. :/ In the current state, it’s totally unusable if you plan to use more than a handful targets and a few initiators. In my opinion, in it’s current state DSS V7 is definitely not a system you would want to use as VM-Storage. However, given the good pricing, it might turn out to be a great product if the bugs are fixed.
I’ve filed lots of tickets for all of those problems, in total >30 (not all are errors, there are also 6 or so enhancement requests). I hope they’ll be fixed soonish.
I’m currently starting tests with the newest release candidate and will update this post accordingly. I’m done, and I’m not impressed.
Update: What really, really needs to be fixed:
- Allow changing of LUNs for targets that are in the cluster, allowing people to actually add VMs during runtime!
- Replication of whole PV from one system to the other, allowing for stable, fast failover without one replication-job per LV. (Open-E said this is NOT going to be implemented already).
Overall, I have to say I’m really, really, extremely unhappy with Open-E DSS V7.
Das einzige wirklich zuverlässig wirkende an proprietärem Storage, was mir bisher untergekommen ist, ist NetApp.
Ansonsten hört sich das nach was an, was sich (bis auf das active/active) auch mit FreeBSD bauen lässt: HAST FreeBSD ZFS with CARP failover
Nem RAID-Z2 vertrau ich deutlich mehr als nem HW RAID6 ;P
Und mit `zfs send | zfs receive` lässt sich das sogar noch angenehm replizieren (vorzugsweise nach irgendwo off-site).
Leider ist das ‘selbstgefriemelt’, ohne Support…ich brauche aber im Firmenumfeld eine Lösung mit 24/7-Unterstützung.
Warum nicht eine “selbstgefriemelte” Lösung mit HW darunter, die 24/7 hat und einem entsprechenden ISV, der Dir Support auf OS und die Selfmade Lösung gibt.
Ansonsten fand ich auf der VMworld das Thema Software Defined Storage mit Nexenta interessant. Da läuft auch ZFS drunter.
Sollte beides relativ preiswert sein.
Nexenta hatte ich mir auch angeschaut, aber das läßt sich, soweit ich gesehen habe, nicht auf der derzeiten Hardware betreiben.
24/7 für die Hardware haben wir. ISV wäre natürlich eine Idee, ich hatte schonmal bei Linbit gefragt, Support für Debian mit DRBD läßt sich dort einkaufen. Fehlt aber noch das iSCSI… und am Liebsten hat man ja doch alles aus einer Hand.
For the lack of API commands:
I wrote a scraper with a command line client for DSSv6. It should work for v7 or at least be easily adaptable…
code is at https://github.com/ixs/dss_cli
Hi Andreas, I only now saw your message, sorry.
Thanks for pointing that out, but I believe Open-E to be a stinking pile of shit and there’s no way we’re going to use it in production ever again!
We’ve had VERY good results with IBM V3700 which has been running rock-solid for over two years now without any downtime, even when doing live OS-Updates .
Also wir nutzen open-e in produktiver umgebung, wir haben 2 Cluster a 2 Nodes in Active-Active betrieb und jeweils 2 20TB VD`s, die Replikation läuft jeweils über 2 40G Kanäle. Nach aussen hat jedes Node 2 10G Verbindungen. Auf jedem VD laufen mindestens 30-40 VM`s. Auf die beiden Cluster greifen gleichzeitig 14 vSphere Hosts zu.
Solange die Hardware ok und absolut kompatibel ist gibt es kaum Probleme, failover klappt ohne das die VM`s etwas merken. Aber die HARDWARE muss allen Kompatibilitätslisten entsprechen sonst kommt es zu recht merkwürdigen Fehlern.
Die Hardware wurde voll von Open-E supported, wurde via Distributor mit Support bezogen. Aber auch da gab es Probleme, die Systeme überhitzten, daher Probleme mit den Raid-Controllern usw. usf. – auch der Distributor hat sich nicht mit Ruhm bekleckert. Aber das ist eine andere Geschichte…
War das STARLINE ?
Die haben uns am anfang auch ein System gebaut das total überhitzte aber nur weil die Typen den RAID Controller und die Dual 10G Karten nebeneinander installierten obwohl es genug freie Slots gab.
Aber sonst haben wir echt wenig Probleme.
Noch eine empfehlung, am besten sind die US Supporter, mit dennen konnte ich alles schnell lösen.