fb: Useless people

April 22, 2005

Useless people

Recently I started noticing how useless some brits are. I'm not sure If I'm just unlucky or what, but when I make a suggestion I really get irritated if I get ignored.

Below is a quick episode of 4 days of pure hell and irritation to me of a client app on one side of a firewall gets NAT'd and tries to reach the other side of a firewall where the server app is running, sadly this is all between certain contracting companies who believes in throwing in many over eager over escalating-hungry project managers at "problems" that's quite simple to solve in the first place.

I've left quite a few bits out, apologies in advance...

Monday:

them> You got some access through some firewall for us and it aint working, can you join our conference call?
fb> ok
them> I'll send you some e-mail detailing what we think
fb> ok
*receives mail*
Thread about how a guy tarred up a box and put the app on another box which is now allowed to talk but it "aint working"
fb> *conference call dialin*
them> hello, here, meet the 15 people on the call and what they do.
them> your fw is fucked, it must be your fw because this machine is a mirror of the other one
fb> what is the error?
them> "No common encryption algorithm with client"
fb> How is that a firewall problem?
them> It works on the other machine!
fb> It sounds to me like the client cannot establish a compatible cypher to the server, is this SSL?
them> yes, we think it's because it is NAT'd, NAT's break everything
fb> Then why would it complain about a Cipher?
them> The firewall is broken! It does not happen on other machine!
fb> *knowing this is going nowhere*, Give me contact details of engineer with access to the client machine
fb> and somebody on the server side.
them> Here you go, it's really important, had to work last week, I'm going to escalate to 30 more people if
them> this does not get fixed.
fb> righty.
fb> *calls dude on client side*
fb> What you using?
them> Solaris, with perl and some ssl libs
fb> Great, try openssl ss_client --connect :
fb> do you see the SSL connect and public keys being thrown about?
them> hold on I cannot find openssl
fb> /usr/local/bin ?
them> ahh, yes.
them> *tries*
them> It says something about Ciphers and gives error 500?
fb> It's a webserver then? 500? Internal Server Error?
them> yes
fb> Righty, so its obviously talking to the server.
them> No, the firewall is broken, I setup many firewalls at home, I'm sure its NAT that is broken!
fb> No, a firewall cannot produce a http error, its not clever.
them> No, its the firewall! (insert nonsense here)
fb> Go to your working machine, try the same
them> SSL connect works!
fb> Right, something is wrong with your SSL on your client machine
them> It's a tarball! Everything is the same! All patches etc!
fb> It is talking to the webserver and cannot establish a cipher to handshake with.
them> It's the NAT, does the packets know how to get back from the server to the client?
fb> How will you get a error 500 from the webserver if the packets cannot come back?
them> It's the NAT!
fb> *hangs up after giving up with this guy*
them> Mail thread about how it's NAT because the return path "obviously" does not work.

Tuesday:

fb> *comes into work to find a e-mail with tcpdumps, thinking that they can obviously see what is going on now*
fb> *wrong* *gives them a call*
fb> You showed me tcpdumps showing a wonderfull textbook 3way handshake, then you tell me the return leg does not work?
them> Yes, you can see the data pausing for a while and then the connection closing
fb> Yes, when a application does not work it generally spits out something and closes the connection, it's the content of this conversation that is important, ie. the Error 500 the client is getting?
them> No, the return leg is not working, it is working fine on the no-NAT box!
fb> It's obviously a SSL handshake issue on the client side since a connection has been established.
them> No, it's nat *goes on for 5mins about his home firewalls*
fb> *2 hours later*
them> *via email* We analyzed the data of the tcpdump's/snoop's and we are convinced the IP connection from our server to your firewall is not being returned to the client.
them> *project manager chips in mail thread* Yes, and I cannot connect from the server to the client machine on the same port, surely this means the data is not going back!
fb> Errrr, you asked for client->server on tcp port xyz. Why on earth would server->client same tcp port work?
them> que?
them> We tested again from our working machine with no NAT or firewall, it is working fine!
fb> *ignores*
them> *"clever" user on the server side* That PM had something clever to say about the connection not going back. notice the syn sent when I do a telnet on the port!
fb> Err, you'll get a SYN_SENT when you try and reach something on that firewall and it aint allowing the traffic through
them> So that means the data cannot go to the client!
fb> *ARGH*
fb> No, it means the firewall does not allow a connection on that port frmo your machine, as specified in the rules
them> Ahhh
fb> *hangs up after giving up*
fb> *types 5 page e-mail detailing with examples of how TCP traffic works, with the syn's and synack's and all.
them> Please call server dude!
fb> *calls*
them> Please explain what you mailed? I am a programmer actually and trying to explain your firewall problem
fb> "firewall problem?"
fb> Client makes connection to you, it tries to handshake with server, it does not, it dies?
them> But surely when it handshakes it talks back to the client?
fb> Err yes?
them> On what port does it talk back? Surely you need to open both ends?
fb> No, it is not a seperate connection backwards, the connection is alread established *shows source ports*
them> Err oh, I thought everything in life has a forwards and backwards seperate connection
fb> Righty

Wednesday:

*luckily little time for them* *phones up client dude after a while*
fb> Any news?
them> Yes, we're sure it is the NAT that is breaking it, it works elsewhere
fb> *sigh* What about the error you're getting?
them> Must be client side!
fb> Right, make a telnet to the server on that port, lets converence in server side guy to make a netstat
them> okay
fb> Right, server side dude, you should see a session if you do a netstat
them> Yup, I see ESTABLISHED
fb> that means you are talking to the server and that communucations are fine between both sides of this wall
them> But it is working fine on the other machine!
fb> *sigh*

Thursday:
*get e-mail*
them> Oh, you do not have to worry, it is not a firewall problem, it was trying to handshake with SSL1
them> we made the webserver talk SSL2 and its all working now.
fb> !

I do not want to start to imagine how many £'s was lost in trying to "solve" such a problem. I sometimes wonder how the economy can be so strong. Absolutely no wonder South Africans are socalled good workers.

Posted by fbotha at April 22, 2005 03:48 PM