Category Archives: Rechenzentrum

Review of Open-E DSS v7

Initially, we were looking for a 10GE-iSCSI storage solution that would do synchronous or at least memory-synchronous mirroring of data to a second system and automatic failover. We planned to use the system as storage backend for a few dozens VMs, and wanted the storage to be highly available on a shared IP. Active-Active supported seemed pretty awesome too, and the system should allow seamless failover.

However, some vendors didn’t provide this option, others were too expensive and the project was an open bidding, so we had to be cheap. The only viable options seemed to be building something ourselves or buying Open-E from a distributor that would take care of the hardware part. They were also offering 24/7 support. It sounded pretty good, and our distributor was saying he did a couple of installations with it, so we went for it. The system consisted of:

Continue reading

DELL about SSH key authentication on PowerConnect M6220

A coworker asked DELL about SSH public key auth on PowerConnect M6220, because we wanted to automate something. Anyways, it’s pretty common to automate stuff nowadays, right?

Here is DELL’s reply on using a ssh with keys:

Von: xxx@Dell.com [mailto:xxx@Dell.com]
An: xxx
Cc: xxx@Dell.com
Betreff: Dell SR:xxxxxxxxx

Good morning Mr. xxx,

I’m the engineer to whom Mr. xxx has escalated the above mentioned
service request. I’m writing to you today to inform you that unfortunately
public key authentication isn’t supported in way  you’d like to on these
switches with version 5.x firmware. That is, you will either always end up
being asked for a password, in addition / even though you supply correct
private/public key during your ssh session negotiations, or you might end
up inadvertently opening a backdoor for unauthenticated users wishing to
use password authentication only.

Either way, it seems this will take quite some time to fix, so I suggest I
close this service request out now and will keep you informed about any
future developments on this issue.

Kind regards,
xxx, B.Sc. (Hons) in IS/IT

Enterprise Product Engineer, Networking & Linux & Security
Dell | Enterprise Support Services
Office Number (+353 1 xxx xxxx)
M-F (8:00 – 16:30 IST)
Certification: CCNP, RHCE & JNCIP-SEC
How am I doing? Email my manager xxx, xxx
<mailto:xxx@DELL.com>  with any feedback.

Uhm… what? Oo

Why Oracle (and Java) sucks

We were investigating random crashes of a webapp, and it turned out to be a JAVA CORBA bug, so we  reported it to Oracle. Today, I got the following reply in my mail:

I own your service request *-*********** about a Java Corba problem.
Please understand that Java is a free product. Hence support (and bugfixes) can only be provided for Java SE contract customers.

Unfortunately, the Customer Support Identifier CSI#******** you entered does not entitle you to Oracle Java SE support. To submit a Service Request (SR) for Java SE when running your own or third party Java applications, a Java SE Support contract is required.

The bug you are referring is in state accepted indeed. There are no plans right now to fix it. Of course, if a contract customer requests a fix, then it will be fixed. Otherwise it might take quite some time until a fix becomes available.

[…]

Oracle Java Support starts at $10,000. Thank you very much, so leave the bug open and let other customers stumble upon it. Just yesterday I heard that armed US drones are using CORBA for communication, too… oh my.

Our approach might be using Nagios Eventhandlers to restart crashed application servers, as we got more than N+1 of those. Oracle’s mentality pisses me off…

Defending against & having fun with WebLOIC

Lately, one of the websites under my protection was being DDoSed by a well-known trouble-making party whose name shall not be released and stay anonymous. Another party that is monitoring the web for threats against our websites notified me that a DDoS was currently being  started. It seemed that the attackers were spamming a link to an automatically starting WebLOIC via mail and tricked others with a variation of methods to open the URL so that they would automatically participate in the DDoS.

Let’s move to the technical side: it was a pretty small DDoS, with about 50MBit/s – we probably wouldn’t have noticed as it just looked like a normal traffic spike and did not endanger the availability of the website at all. We’ve handled much larger legitimate traffic spikes for that site already.

A quick investigation showed that WebLOIC was being used and was ‘hosted’ on a nopaste service. Requests looked like this:

GET /?id=1300380622178&msg=We%20Are%20Legion! HTTP/1.1
Host: XXXXXXXXX
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:8.0) Gecko/20100101 Firefox/8.0
Accept: image/png,image/*;q=0.8,*/*;q=0.5
Accept-Language: en-us,en;q=0.5
Accept-Encoding: gzip, deflate
Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7
Connection: keep-alive
Referer: http://xxxxxxxxx.html

Quickly checking out the referer gave me the sourcecode; the id consists of a timestamp and the request id. The ‘msg’ is a user-changable text, but ‘id’ is javascript-generated.

How to block it?
– Use a regular expression for the query string (very easy)
– Block users with a referer from that nopaste service (very easy, too)
– Block users that do more than X connections within a minute (easy if you have a decent firewall), side-effects might cause large NAT gateways from mobile providers to be blocked, but that’s better than being completely offline, right?

Where to block:
– block as early as possible: in your DPI firewall, web application firewall or loadbalancer
– *not* in every single webserver

Having Fun:
As the WebLOIC runs in the attacker’s browser, there are lots of possibilites:
– redirect attackers to a site known to be monitored by the FBI (explosives, terrorism etc.)
– CSRF, make them post something on a service like facebook or twitter (#iDDoS-site.tdl) and search for their posts. Kindly ask them to stop.
– redirect the attackers to do lots of google searches – they will quickly be blocked by google services
– send a gzip-encoded stream that consumes lots of cpu time and memory on their side, this might even crash the browser
– ‘reflect’ the DDoS to somewhere else (sending 301/302 redirects is pretty low-bandwidth for you)

So in total, WebLOIC was a good idea, but right now rather inefficient and its usage might have unwanted sideeffects… 😉

byebye, SLES 9

Heute kam wenig überraschend eine Mail von Novell, dass der SLES 9 Support nun nach 7 Jahren zu Ende ist – außer man ist LTS Kunde – was wohl ziemlich teuer wird und nur für Großunternehmen Sinn macht. SLES9 ist so dermaßen angegraut, beim Blick auf die Paketversionen wird man schon leicht Nostalgisch… ;D

 

Interessant sind die Statistiken:

Gesamte Updates: 1565
Security: 767
Empfohlen: 718
Optional: 72
YaST: 8

Das CVE-Tracking war im ersten Jahr nicht vollständig, daher sind die Nummer eigentlich etwas höher.

Insgesamte CVE-Einträge: 1447

Top CVE-Kanidaten:

127 IBM Java 5
125 IBM Java 1.4.2
84 kernel
66 SUN Java 1.4.2
76 clamav
68 ethereal
58 mozilla
44 cups
32 mysql
31 libpng
28 freetype2
24 ruby
21 XFree86
20 libexif
18 mailman
18 horde
16 quagga
17 tomcat
16 apache2
15 openssh
15 gd
14 python
13 yast2-packagemanager-devel
13 km_nss
12 openssl
12 samba
11 tk
10 pcre

Interessant, dass der Kernel sogar schlimmer als Mozilla war…ich dachte das wäre unmöglich… :/ Clamav mit 76 Löchern finde ich aber auch eine ziemliche Katastrophe, zumal das ja auch oftmals auf Mailgateways läuft…

FreeDOS usb boot

Jeder kennt das Problem: immer wieder mal benötigt man einen USB Stick mit einem DOS wegen BIOS Updates, muss einen zickigen RAID-Controller flashen oder sonstiges… hier eine Kurzanleitung dazu, wie man einen bootbaren USB-Stick mit FreeDOS erstellt.

Vorbereiten der Umgebung

mkdir -p /tmp/usbboot/exec/file-system
mkdir -p /tmp/usbboot/build
cd /tmp/usbboot/build

Tool zum Schreiben der Daten auf USB-Stick herunterladen, entpacken, compilieren

wget "http://prdownloads.sourceforge.net/advancemame/makebootfat-1.4.tar.gz?download" -O - | tar -xvzf -
cd makebootfat*
./configure
make
cp makebootfat ../../exec
cd ..

Nötige FreeDOS-Binaries und MBR herunterladen

wget http://www.ibiblio.org/pub/micro/pc-stuff/freedos/files/distributions/1.0/pkgs/kernels.zip
wget http://www.ibiblio.org/pub/micro/pc-stuff/freedos/files/distributions/1.0/pkgs/commandx.zip
wget http://www.ibiblio.org/pub/micro/pc-stuff/freedos/files/distributions/1.0/pkgs/unstablx.zip
wget https://www.kernel.org/pub/linux/utils/boot/syslinux/syslinux-4.04.tar.gz
tar -xzvf syslinux-4.04.tar.gz syslinux-4.04/mbr/mbr.bin
mv syslinux-4.04/mbr/mbr.bin ../exec

Nun extrahiert man die relevanten Dateien aus den .zips und kopiert sie an die passende Stelle:

unzip kernels.zip
unzip commandx.zip 
unzip unstablx.zip
find ./ -name command.com -exec mv '{}' ../exec/file-system ';'
find ./ -name kernel.sys -exec mv '{}' ../exec/file-system ';'
 
for file in fat12.bin fat16.bin fat32lba.bin
do
	find ./ -name $file -exec mv '{}' ../exec ';'
done
 
cd ..
rm -fr build
cd exec

An dieser Stelle kopiert man alle Dateien, die nachher auf dem Stick liegen sollen nach ./file-system. Daraufhin befüllt man den USB-Stick per makebootfat und räumt auf.

./makebootfat -o /dev/sdb -E 255 -1 fat12.bin -2 fat16.bin -3 fat32lba.bin -m mbr.bin ./file-system
cd ../../
rm -fr usbboot

Getestet zuletzt am 17.09.2017 – sofern es nicht mehr funktioniert, bitte kurz kommentiere, dann aktualisiere ich ggfs. den Blogeintrag.

Paketverlust mit D-Link DGS-1210-48

Beim Testen von zwei brandneuen D-Link DGS-1210-48 fiel auf, dass auf dem Gerät Paketverlust auftritt. Je größer die Pakete, desto höher war auch der Verlust – bis zu 7%. Das ist natürlich nicht akzeptabel. Nach einigem Überlegen hatte ich eine Idee woran es lag:


Tatsächlich behob das Umstellen dann auch das Problem. Offensichtlich gibt es hier ein Problem damit, dass der Switch die Sendeleistung bei kurzen Kabeln (wie sie im Rechenzentrum im Access-Layer üblich sind) herunterschraubt. Eigentlich schade, denn grade die Stromsparfeatures machen den Switch interessant. Ansonsten machen die Geräte bisher aber einen ganz soliden Eindruck. 🙂

EDIT 21.08.2016: ich habe alle DGS nun gegen etwas solideres ausgetauscht – die DGS hatten über 3 Jahre eine Ausfallrate von ~75%! Sample size: 20 Switches.