I feel like a dinosaur. What ever happened to the good old days of parallel and serial based devices that were easily used and controlled from simple applications? I mean, what happened to the days when I could plug in a serial modem into a serial port and send commands to it using a simple echo command at the console?

Now everything is USB and the ability to just talk to a device is completely obfuscated. First you have to interpret the language to talk to the device, then you can send your commands. Gone are the days when I could send a “echo ‘ATH0′ >> /dev/ttyS0″. Now I have to have the drive drivers installed first so that the kernel knows to put the device somewhere in the UDEV system and then I have to figure out which of the million USB devices is the one I want. It doesn’t help that I have 3 powered USB hubs hanging off my computer, each with 8 ports. So theres another 24 ports on top of the 8 built into the machine. Have you ever tried to crawl through your devices in this way?

The old days of pure serial communication are dying away to a forgotten and distant memory :-( Worse still, there are so many kids around now that have no clue what it was like. My first modem that I personally owned was a 1200bps rocket. I’d used a 300bps prior to that at school. I remember my 2400bps and 9600bps modems and by the time I graduated to 14400bps modems, I thought the speed was just amazing. Of course, my 33600bps modem blew me away. By the time we got to 56700bps I was an old hand at these things. And this was all before the Internet was a big thing.

I know that I exaggerate it slightly. There will be a fair few of you that say things are not so different now to what they were before. Its just a different method to achieve the same end.

But consider that I only ever had to plug any modem in to a serial port, regardless of brand or model, and I could start issuing commands directly to it. I didn’t have to think about it and I rarely had to change my commands.

“But you don’t NEED an old style modem anymore. We’ve got broadband now.”

Actually, I can think of many uses for an old style modem that are still valid. The main use I’m looking at doing shortly being one of them. I want to build a fax server. Thats not so hard and in fact, with my old 56k modem, its drop dead simple.

But I also want to do the same thing for my brothers workshop. And that is a totally different story. You see, modern computers don’t have serial ports any more. And its nearly impossible to locate a new serial modem anyway even if you plug a serial card into a PCI slot on the mainboard. So you have to rely on a USB device instead. Most internal modems are the truly crappy “winmodems” in which the majority of the work is passed off to the computer itself instead of the modem device doing it. This in itself is even more annoying, completely outside of driver issues. I also consider it unacceptable. I don’t want to waste CPU cycles doing work the modem should be doing on the ASIC.

But then we get to another point. Why are we so quick to drop the old serial ports? I mean, I know they’re slow and they slow the bus of the PC down some. But every network device worth its salt (such as routers and switches) still offer a serial based terminal/console port. A lot of the guys I work with currently probably haven’t had to use them because most of our clients are small businesses. In fact, I can recall a conversation not too long ago when I mentioned a serial terminal server and got funny looks.

“A what?”

“A terminal server. A machine with multiple serial ports on it that you plugged in to the console ports of your switches and routers. You’d telnet to the terminal server and then that would allow you to access the console port of the network devices. Or if the network was offline, you could plug one serial cable into the terminal server and access all the devices without needing to unplug from a device all the time.”

“Really? Never heard of it. I always thought a terminal server is a Windows server you use RDP to connect to.”

I’m a freaking dinosaur.

Mind you, networking diagnosis is very much greek to a lot of guys too. Actually, a lot of things that would make our lives easier to support our clients are things that people newer to the IT industry than me have never even heard of. I blame Microsoft for this. Main reason being for a lot of people, a network is a bunch of Windows hosts connected to a full 24bit Class C network and everything is done on the Windows hosts itself. Layers 1-5 on the OSI model are greek.

Why is this? Why is it that so many people seem to lack a lot of knowledge about such a broad range of options?

One client was having network connectivity issues. Their wireless link between two offices several kilometres apart was dropping off all the time and having all sorts of issues. No one could figure it out. I stuck an SNMP monitor in there and we could watch the key pieces of info about the devices involved. You could literally watch the signal strength rise and fall in real time. You could see when one side of the link was having more trouble than the other. It made sense.

Another client had a VPN connection going across a DSL line that kept dropping off. Often for days at a time. This was unacceptable. Instead of finding out what the modems at either side were saying, the first port of call was the ISP who said it wasn’t them and blamed the modems. But the modems had just been replaced to rule them out. So the ISP blamed the line company. Of course they said it wasn’t them.

They eventually managed to get it sorted out. Then one day the line dropped off again and I got the call. First thing I did was reboot the modem at once side of the link. Did it come back up? Yes. Sweet! Next thing to do, stick a syslog daemon on one of the servers and send everything from the modem to it. Find out exactly what the modem sees at the time it drops offline. That way you have a better idea of where the problem is.

If this had been done in the first place, the problem would’ve probably been a lot easier to resolve. But its just not thought of anymore.

Information is so important. If you don’t have all the facts, how can you resolve the problem? If there is an issue, use the resources available to you to resolve the problem. Logging is by far one of the most important aspects of problem solving, yet so few people make use of it anymore.

I partially hold the Microsoft Event Logs responsible for this mess. They are so difficult to tweak properly into something that is useful. They also fill up really rapidly. Often, when I look in the event logs, they’re either misconfigured to log every little detail, or they’re not logging enough detail. Worse yet, a lot of what gets put in the event logs is cryptic to the point where you have to go spend ages on the web trying to track down the scenario that fits your problem.

Part of the problem is the “Microsoft Method” as I call it. There is only so much you can learn from a book. People seem to think that once you have a piece of paper from Microsoft, you’re some uber admin. Unfortunately they over estimate what they can do. If its not a Windows issue, or its outside the Windows scope, the problem can’t exist or its faulty “something.” Also, the Microsoft certifications really only provide you with the skills to configure a Microsoft application. If theres a problem, you’ll find Microsoft taught admins resort back to comparing the configuration with the way they’ve learned to configure that application. This isn’t incorrect by any means, but it leaves out so many other methods.

A good example of this is from when I was at Datacom. One of the other techs on the Microsoft server queue was very much indoctrinated to the point where if it wasn’t defined in one of the Microsoft Press books, it wasn’t valid or couldn’t be done. He was taught to see things as if he was inside the box himself. This kind of thinking limited the effectiveness of resolving the problems.

Let me be very clear about this. MCP and MCSE certification makes you a proficient systems administrator for Microsoft platforms and technologies. You can know everything there is to know about Windows and Exchange and MSSQL and IIS, but if you hit a networking error or you’re dealing with something that doesn’t have a Windows interface, you’re going to get lost.

The web has made a big change in that now you get network devices coming with simplistic web interfaces that allow you to configure your devices without having to know how the configuration actually works. For example, we use SnapGear firewalls at work because they’re small business friendly (read “cheap”) and they have a very straight forward web interface that makes managing them easy. But underneath them they’re just Linux boxes. They run FreeSWAN, PoPToP, Netfilter/iptables and so on. All these things I manually installed on the proxy/firewall thats running at my brothers workshop.

But what amazes me is that the guys I work with can’t read the logs of these devices too well. Well, let me put it another way. They know when to filter for certain key markers that will tell them what they want to know based on past experience. But they can’t just sit there and read through the logs to see whats happening on the device before and after the event they’re looking for. Interpretting messages put in Syslog seems to be a dying art.

One example of this is using the syslog messages from FreeSWAN or OpenSWAN to trace an IPSEC tunnel being built from the initial handshake, through the IKE keying and then finally with the tunnel being brought online. I mean, sure its cryptic for the most part. But they are readable and they do make sense if you take the time to actually look at what you’re seeing there. If you understand HOW IPSEC works they’re not actually that difficult to follow. Microsoft teaches you how their KDC works for Kerberos, and they’ll show you how to configure IPSEC for their platforms, but they provide very little background as to WHY it works that way and whats actually going on.

This is really starting to sound like I’m on major diatribe against Microsoft and the MCSE process, which really isn’t the case. What bothers me is more the fact that people are fencing themselves in by learning how to do something without understanding why they’re doing it whats actually going on.

Unfortunately, this isn’t limited to just the Microsoft and Windows environments. This is true of nearly all platforms that operate at such a high level. I mean, just because I know how to build a computer does not mean that I have any understanding of how the parts of a computer communicate with each other. I honestly couldn’t tell you anything at all about the communication between any devices on a mainboard. I know the fundamentals, but thats it. My only saving grace in this area is that if theres a problem with a card in my computer, its likely to be only one of two things. The card or the mainboard. And I know enough to be able to figure out which.

The downside to that is that as we move forward, more and more is being added to the mainboard so you don’t need as many cards. I still remember when your disk controllers were a separate card you plugged into a ISA or EISA port on your mainboard. I remember when USB was so new you had to buy a card to be able to use it. If one part failed, you replaced the card. Now, if something fails, chances are you replace the mainboard entirely.

But if we stay away from hardware and concentrate on Layer 2 and above of the OSI model (which covers ALL software) then its becoming extremely frightening how few people understand those lower layers any more. In fact, they know the Application (layer 7) and Presenation (layer 6) layers very well, and may have a pretty good understanding of the session (layer 5) but they might only have a very basic understanding of how the lower layers work. Enough to make it work, but not really why what they did works.

I called an ISP today to get a PTR record changed in their DNS. The guy on the other end of the line tells me that I’ll need to contact the domain registrar or make the changes myself because they don’t host the domain.

Me : “Excuse me? A PTR record maps an IP address to a hostname, not the other way around. The hostnames are already pointing to the correct IP address. I just want you to change the PTR record so that when someone does a lookup on that address, it points to the correct hostname.”

ISP: “I’m sorry, but you’ll need to do that with your registrar. We don’t have the authority to make changes to your domain.”

Me : “I’m not asking you to change my domain. You own the IP address and therefore its your DNS servers that get queried when someone does a reverse lookup on that IP address. My registrar has no authority to manage PTR records for IPs you own. Only you can do that.”

ISP: “Oh. I see. I’ll speak to one of the DNS admins and get back to you.”

Me : “Thanks, you do that.”

How can someone work for an ISP and not have an understanding of the technologies required to run and manage an ISP? The guy on the other end of the line sounded young, but in the IT industry that doesn’t mean a lot. But even 10 years ago I made it my business to know as much as I could about the technology I had to work with, support or manage.

I’m a freaking Dinosaur!