Category Archives: Blog

Building a network monitor

As so many of us I’m currently working from home. My internet at home is most of the time pretty ok, but it can get very flaky, as in, speed drops, it can’t connect to certain hosts etc.

However, every time I report this to my ISP, I get the same routine: restart, unplug, hold for 30 seconds, pray to Ubuntu and other ridiculous demands that make no sense because I didn’t change anything in the hardware (I didn’t even move it).

So to prove my point, I decided to refurbish one of my Raspberry Pis as a network monitor using InfluxDB to collect the results and Grafana to spice up the charts

So to do this you obviously need

  • A raspberry PI
  • An internet connection (doh!)
  • A time series database InfluxDB
  • Some python3 scripts to collect the data.
  • A graphing tool Grafana

Setting up the PI

Following one of the many tutorials, I managed to setup a Pi in headless mode. Aka, it’s plugged into power and ethernet and using my router it’s configured to always be at a certain IP address. It has SSH running so I can connect to it from my laptop. This is the basic setup and requires the minimal amount of wires.

I didn’t set it up wireless as I want to be as close as possible to the router so it literally sits next to the router plugged in in one of the available ethernet ports of the router. This should give me the optimal network so they can’t blame my wifi.

First thing you do obviously is to change the password for the pi user to something other than raspberry. The thing will be hooked to the internet and you can never be safe enough.

Once I log into it using ssh, I first upgrade it to the latest versions, so I need to run a few commands: 

sudo apt update
sudp apt upgrade

Go grab a cup of coffee as this might take a while if you haven’t been updating that Pi regularly. You have to be on at least Stretch or Buster for InfluxDB. You can’t go from Jesse to Buster in one go, so if you want to do that, you have some time to waste.

If you want to be quick about this and you don’t care about your existing image, it’s faster to just grab the latest image using the imager from https://www.raspberrypi.org/downloads/

If you want a graphical UI, you can enable VNC simply by typing sudo raspi-config and select the Interfacing Options, VNC, enable. This should install the vncserver and configure it. When it’s done, it should be running on port 5900. Obviously, I have ssh enabled as that’s the main way to connect to it.

Installing InfluxDB

InfluxDB is not a standard install so we will need to get it from another source than the default apt repositories. I took my cue from https://pimylifeup.com/raspberry-pi-influxdb/

wget -qO- https://repos.influxdata.com/influxdb.key | sudo apt-key add -

All on one line. This will download the key and pass it into apt-key for further use. As I am on buster, I can use:

echo "deb https://repos.influxdata.com/debian buster stable" | sudo tee /etc/apt/sources.list.d/influxdb.list

Obviously, replace buster with stretch for stretch. Now we’ve added a new repo, we can again do ‘sudo apt-get update’ and then ‘sudo apt install influxdb

And to make sure influxdb runs on boot:

sudo systemctl unmask influxdb
sudo systemctl enable influxdb
sudo systemctl start influxdb

our next step here is to setup a database in influx, so we must be able to login, for that, we need the client.

sudo apt install influxdb-client

And then you can just type `influx`. First we’ll be setting up authentication (replace your password with what you want)

CREATE USER admin WITH PASSWORD '<password>' WITH ALL PRIVILEGES

And then we’re going to make this user the only admin by setting up authentication by editing the influxdb.conf file:

so exit out of the client and back on console, type ‘sudo nano /etc/influxdb/influxdb.conf‘ and Find the [http] section and under there add:

auth-enabled = true
pprof-enabled = true
pprof-auth-enabled = true
ping-auth-enabled = true

Then we need to restart the database by typing ‘sudo systemctl restart influxdb

Now, if we want to connect we need to type:

Influx -username admin -password ‘’

By passing in the empty string influx will prompt for the password. This keeps it out of your history file. Though, I doubt that your pi will be setup to host many users so you might just not give much for this.

Ok, so now we’re going to setup another user for it’s own database, I called mine netmonitor and the database the same.

create database netmonitor
create user “netmonitor” with password ‘<again your choice>’
grant all on “netmonitor” to “netmonitor”

Take note that the password requires single quotes!

Great, InfluxDB is installed. Now it’s time to collect the data

 

The Python bits

At first I started out with two scripts, one that used speedtest-cli (I based this one on prior art from https://pimylifeup.com/raspberry-pi-internet-speed-monitor/) and another one that simply connects to Google’s DNS server at 8.8.8.8 with a 3 seconds timeout to monitor if the internet is up or down. Later I added a 3rd that captures the output of pings to various sites I was interested in (at some point I will probably make that configurable)

Obviously it’s fun to paste a lot of Python code here, but you can also just check out the repository here: https://github.com/siriousje/speedtest because over time I will probably add some more scripts.

So, the easiest way to get the code is to login to your Pi, and if for some reason you don’t want to type sudo before each command, do some root magic like 'sudo su -‘ and become root.

Clone this repository under /opt so that it creates /opt/speedtest Then make sure to install all requirements: sudo pip3 install -r requirements.txt\

Copy config.yml.sample to config.yml and update the values for your installation, then add the scripts to crontab using sudo crontab -e

it should look something like

*/5 * * * * python3 /opt/speedtest/speedtest-influx.py
*/1 * * * * python3 /opt/speedtest/internet-influx.py
*/1 * * * * python3 /opt/speedtest/latency-influx.py

Change the numbers to what you want, I run the speedtest only once per 5 minutes.

You can also use environment variables to override your config, but I wouldn’t really bother 🙂

Anyway, this is it, it should now be adding data to your InfluxDB. You can check this by logging into Influx as the netmonitor user by typing

influx -database "netmonitor" -username "netmonitor" -password "what you did"

and then as a normal database, `select * from internet_latency` to see the data flowing in.

Setting up Grafana

Congratulations! You made it to the sexy bits, graphs! This is what we came here for! Let’s go!

Depending on your PI, you may need to find the correct version of Grafana for your architecture. There is a difference between the Pi v1 and Pi Zero v.s. the Pi 2 and up. I use a pi3, so I go with 

wget https://dl.grafana.com/oss/release/grafana_6.6.1_armhf.deb
sudo dpkg -i grafana_6.6.1_armhf.deb

If you were using a pi-zero you should use this:

wget https://dl.grafana.com/oss/release/grafana-rpi_6.6.1_armhf.deb
sudo dpkg -i grafana-rpi_6.6.1_armhf.deb

The small difference there is the -rpi. And again we want this up and running after boot, so here we go:

sudo /bin/systemctl daemon-reload
sudo /bin/systemctl enable grafana-server
sudo /bin/systemctl start grafana-server

And then we go to the webinterface (as my pi is fixed to 192.168.1.199, the service is running on port 3000) so for me it is at http://192.168.1.199:3000/login and change the password to something you fancy

Next step is to configure the datasources!

Grafana already suggest the defaults for InfluxDB, but you still have to type it in the field (it’s a placeholder…) You want to keep localhost because it’s all on that same Pi.

Then obviously, fill out the login details for the datasource. So you need to setup the basic auth and the InfluxDB details:

And then you can test and save your datasource. If it’s green it’s good to go. Great, now we’re going to create a dashboard and add our statistics. So click on the plus, then add dashboard and then we choose add Query

Then we can basically select from (I started with internet_speed), selected field pin, changed mean() to distinct() by clicking on the plus behind it and selecting Aggregate/distinct() and then we can give it an alias whatever you want:

Now if you are using tags, like for the internet_latency stat where we use the tags to collect the hosts, you can add tag(host) to the group by setting and it will plot all hosts over each other

Anyway, configure your dashboard anyway you want. Mine ended up looking like this:

Which gives me a clear view that my internet is capped on the top, but also has a lot of drops in speed and even regular timeouts getting connected to the nameserver.

Anyway, with this data at least I have something to tell my ISP that it’s not just me 🙂

 

Writing your own Google Apps Plugins

So I am currently between jobs and I like to use that time to learn new programming languages as I believe that you should try to think outside your day to day experiences. Especially in this world of microservices, it makes sense to broaden your horizon to see if you’re not always applying your favorite lawnmower to a problem that requires a hammer.

Given that I have about 4 weeks, I figure I can do more than one so I started with Erlang and I used Google docs to keep track of the notes. I quickly ran into a huge issue: there is no ‘format as code’ block functionality in standard Google Docs! Being used to Atlassian’s Confluence and even WordPress I was a bit undone. What next? Obviously I looked for plugins but they always want all the rights (like, run while you’re not around, access to your bookmarks and browser history, credit card details, you name it)

I also didn’t feel much for applying the same styles over and over again. So, what is a good procrastinator to do? Right! take a small break from his endeavours into Erlang and write a plugin for Google Docs! The API is pretty well documented and there are some samples online. But basically, if you go to Tools/Script Editor you’re already halfway there.

In the editor that popped up I wrote the following script (probably still buggy enough but it does the trick):

function onOpen() {
  DocumentApp.getUi()
  .createMenu('Custom')
  .addItem('Format As Code', 'formatAsCode')
  .addToUi();
}
 
// Define code styling
var style = {};
style[DocumentApp.Attribute.FONT_FAMILY] = "Courier New";
style[DocumentApp.Attribute.FONT_SIZE] = 10;
style[DocumentApp.Attribute.BACKGROUND_COLOR] = "#000000";
style[DocumentApp.Attribute.FOREGROUND_COLOR] = "#FFFFFF";
style[DocumentApp.Attribute.BOLD] = false;
 
// Apply code formatting
function formatAsCode() {
  var selection = DocumentApp.getActiveDocument().getSelection();
 
  if (selection) {
    var body = DocumentApp.getActiveDocument().getBody();
 
    var insert = null;
    var table = null;
    var cell = null;
 
    var elements = selection.getRangeElements();
    for (var i = 0; i < elements.length; i++) {
      var element = elements[i];
      if (element.getElement().editAsText) {
        if (insert == null) {
          insert = element.getElement().getParent().getChildIndex(element.getElement());
          var detached = element.getElement().removeFromParent();
          var table = body.insertTable(insert);
          var cell = table.appendTableRow().appendTableCell();
          cell.setAttributes(style);
          cell.appendParagraph(detached.asText().getText()).setAttributes(style);
          // for some reason there is an empty text element now
          cell.getChild(0).removeFromParent();
        }
        else {
          cell.appendParagraph(element.getElement().removeFromParent().asText().getText()).setAttributes(style);
        }
 
      }
    }
  }
}

And it allows you to save and immediately try the effects.

So, now I can finally get back to what I was doing, learning Erlang 🙂

 

Your arguments are invalid.

Because it’s enough to have just one. See, thanks to scope and something called currying, any multi-argument function can be rewritten as a unary chained function. A simple example is this Javascript function to add two arguments:

  1. function add(x, y) {
  2.    return x + y;
  3. }

it’s easy to see what that function would do, you pass in two arguments and they get added:

  1. var sum = add(2, 3); // 5

now, in a functional world, you want to have unary functions that take only one argument. How can you achieve a simple add function if you’re not allowed to take more than one argument? This can be done by currying (and it requires scope)

  1. function add(x) {
  2.    return function (y) {
  3.       return x + y;
  4.    }
  5. }

now the call to this method would be different because it just takes one argument. Thanks to the scope, the inner function can see the x, so you call this method like so:

  1. var sum = add(2)(3); // 5

And that’s how you can decompose a multi-argument function into many, nested unary functions.

You are more likely to be killed by an asteroid than by a terrorist.

Though what happened in Berlin today is still ‘most likely’ an attack (by a lone wolf, not by an army of asylum seekers) we can’t generalise and blame this on asylum seekers in general. That’s just what our nationalistic leaders want, what the news papers want because it sells more papers.
 
By these rules of generalisation, all men are rapists, all fathers are incestuous, all mothers are whores, all white people trade slaves and basically all human beings are monsters. You cannot blame anything like this on entire groups of people. The fact is that only one person was driving that truck. Not a nation, not a group, not anyone else but that guy.
 
The chances that you are going to be killed in the next ‘terror attack’ are much smaller than the chances that you are getting killed by a drunk driver, by crossing a road, by actually driving a car and for some reason, in the USA, by having your toddler grab your gun from your purse.
 
The media loves to make these things bigger than they are, because this kind of fear makes their sales go up. The chances of you dying in a terrorist attack are pretty slim if you are living in a western society like most of Europe, the USA and here in Australia. It’s practically non-existent.
 
You are about 10 times more likely to die from SUICIDE than from murder. You are about 5 times more likely to die from DIABETES than from a traffic accident. You stand a better chance at winning the lottery than getting killed in a terror attack.
 
Let’s look at a quote from one of the sources below:
 
“The chances of being killed in a terrorist attack are about 1 in 20 million. A person is as likely to be killed by his or her own furniture, and more likely to die in a car accident, drown in a bathtub, or in a building fire than from a terrorist attack.”
 
ok, 1:20,000,000, should be able to handle that. Let’s compare it to something else:
 
“The chances a person will be killed by an asteroid are 1 in 200,000, which is much higher than the odds of being killed by hail, which is 1 in 734,400,000.”
 
Let that sink in people. You run a 100x (a hundred times) bigger chance to be hit by an asteroid than be killed in a terrorist attack!!! Why the fuck aren’t we walking around with titanium umbrellas!?
 
for crying out loud, let’s stop giving the media what they want: attention. Let’s stop giving away our privacy over this bloated sense of insecurity. And the great thing with ‘terror’ is, if you show a complete disregard of the effects, it’s not terrifying at all. They’re just what they really are: sorry excuses for living beings.
  
just some of the sources below.
 
 
 

Security and safety

Recently people who use Mac computers have asked me about security. Good question. I’d say they are pretty secure. They didn’t ask me about safety. That is a different question. I will try to explain why:

The only safe operating system is the one that is turned off, disconnected and preferably locked in a safe. And even then, it only takes one human to breach that security. As long as it is not turned on, or plugged into a network, it is still secure, but not safe.

Now, if you understand those rules, you can maneuver pretty safe and secure on the internet.

To do so, there are some simple rules you can follow:

Choose an operating system that is released as open source (why? because everything is transparent, every single bit of your operating system is out there, people know about it, good hackers and bad hackers the like, history has proven this, you are more vulnerable with a Microsoft (closed source) OS than with any obscure linux variant out there). Apple is turning towards the Microsoft side, but is at least based on a BSD unix variant, so, as long as you don’t give out your password to install the next .dmg you download, you are pretty safe (even if history has proven that Mac users are the least computer savvy users out there, you just got lucky that it’s in essence a unix machine)

Choose an operating system that is maintained (what does that mean? simply, the more people that are working on it, the more people who know a lot more than you do about operating systems are working on it) If more people are working on it, it does not mean it is not ready yet, it actually most often means that those good hackers are quite fond of it (pick any popular linux or freebsd version, it’s free and people are actively working on it to make it the most secure system they can think of)

Third, be a user when you can, only change into an administrator if you need to be. The way the unix (and alike) operating systems were designed was about giving users power to do things, but not to disrupt other users. I personally change into my root clothes when I think it’s time to delete a user account that has been compromised, or at least leans towards it or when I’m simply done with it. And since I have the power to become root, I can always create a new user.

Finally, and absolutely not the least, don’t be lazy, don’t reuse that password on website nr1 you used for website nr2, unless you don’t care about either of these (this is a rule I break often, I start out with using a weak password until I think I really want to keep that account).

Don’t use that service name in your password. Cryptology is only as smart as you are. You can have all your keys encoded in (who cares) 8 gigabyte keys, but if you use “-facebook” for facebook and “-twitter” for twitter, the other party already has 50% of your password decoded.

No matter how many bits you use to encrypt it, if they know half the message, it’s pretty useless because knowing what you are looking for makes it easy. (Really don’t want to go into details here, but you can understand that looking for “Hi Mom!” in a decrypt is easier if you are sure it contains “Hi Mom!” then if you don’t know that it should contain that)

A password like ‘IchHateMeinFuckingFriendDieZesBlocksDownWoont’ (normal words, but 3 different languages) is far more effective than the not so random ‘qwerty123’, especially if you, as a hacker, already know that the password is going to end in the plaintext ‘-facebook’. (and it is easy to remember). As with anything, but especially in cryptoanalysis, if you already know what you are looking for, it is easier to find it.

Coming back to the 3 language 10 words password, it is 46 characters, let’s assume we only use lowercase, that would be 26^42 combinations. Being able to use uppercase makes it 52^42, include numbers, 62^42, giving a resolution of 1.9074403212938070052188251342723e+75 possible combinations.

With this I’d also advice to look at how the password is used. Yes, the iphone number password is actually quite safe, BUT ONLY if you turned on the feature to wipe the phone after 10 misses. If you don’t, it’s just 10000 tries in which case brute force works. And quickly too. Even the hardware of the iphone you are trying to crack is much faster in 0-10000 than you are. Don’t forget, the computing power in your pocket supersedes in many magnitudes the computing power that was used to put a man on the moon.

Don’t underestimate brute force, as a rule, it takes only 1/2 to crack it. Either they get it right at ‘A’ or at the end at ‘ZZZZZZZZZZZ’, on average it is half and brute force is actually one of the ‘cracking algorithms’ that is 100% sure to crack any code, unless it is accompanied by a total wipeout, as the iphone offers. Use it. Objects that completely lock after X tries are much safer than object that use whatever amount of gigabytes for keys. Imagine what luck you need to crack a safe if you have the previously amount of options (1.9074403212938070052188251342723e+75) with only 10 tries. Brute force is completely useless against systems that lock.

But, it’s not completely useless. If it’s a digital safe? it can be copied. You may lock after 10 tries, but if I can make 1.9074403212938070052188251342723e+75 copies, I only need to try 1 combination per copy. So what do you learn from this? If someone wants to crack you, and if they have enough resources, they can and will. The safest place to keep really private information is still locked in your brain. You will never be secure if your brain is not safe.

As per today, 25/04/2012, they haven’t found a way to secure anything that was saved in your brain and as long as you keep it there, it can’t be protected by security

T is for Telecom, where communication goes wrong…

It’s a wonderful day in the world of telecommunications… First, the 3G service on my phone went dead (it still showed wonderful connection, but as soon as you tried anything it said I didn’t have cellular data). Since I recently moved from Vodafone to Virgin Mobile who uses Optus, I thought, how bad can it be. But hey, after a few tests with luckily another phone at hand, it obviously showed that there is probably one flag not set correctly at their end. We can only hope that the morning crew is able to set the flag.

So that one is still on-going. The other problem I have is with Telstra (the fourth telco to mention in this post, however, the biggest here). Anyway, I updated my subscription with them, because, well, hey, after 2 years, times change, you can probably get a better deal. So I first started to use the DIY way via the website.

Since that quickly enough proved not viable, due to the exact combination of switches I wanted, I followed the advice and started a chat session. I really had some fun with a great guy on the other hand and after about 30 minutes we arrived at a new package for the bundle (phone, internet, tv). I would be called the next morning to arrange for the installation guy to do the physical part of the deal (installing the HD box and moving the non-HD box to the second room).

So I waited about a week. Obviously, nobody called. But since there is a store for anything, there are also stores for Telstra. The next week (last week) I go to the T-store and apparently, my previous order could not be found. I think, well, while I’m here, lets do it again. This time however, I also get a T-hub (Telstra’s take on a tablet, connected to your base station etc) thrown in the deal. Who am I to say no to a better deal, so I go home, thinking to have the same order, just this time with some new gadgets for the home.

After installing the T-hub, the base-station and 2 handsets, I start to play around with the new toys and decide to enable sms. Why not. However, to enable that, I first had to enable something else, which I couldn’t enable since it was already enabled (let’s call it voicemail). So, as the device instructed me, I called the helpdesk. Again, easy-peasy, no problemo and the issue was quickly resolved. Except for the fact that the activation code was expired. But, there was a link in the activation email that would allow me to generate a new activation code at any time.

Today I tried to generate that activation code (as my evening was already ruined because of the trouble with the 3G on the mobile phone), I click that link to find that the website is not exactly working as it should. Actually, somewhere halfway, the process advised me to call the corresponding helpdesk (Telstra has many).

Calling the T-Hub helpdesk is not so bad, apparently, not many people do that, so I got connected quickly enough (around 8.45pm). During the conversation, I was asked to install LogMeIn and even though I do know better in most of the cases, I was easy going and a nice customer, so I let them control my Mac for a while, until they discovered that a certain part of the website was not working. (Yes, the same link to get a new activation code)

That kind of brought the nice lady at the other end of the line to the end of her flowchart and the incident was logged and over the next few days, someone will contact me to activate the sms service on the T-Hub (I hope, but I should know better). However, to log the incident, she needed the serial number on the T-hub, which is conveniently stored inside the battery compartment. (note to reader: store this in mind, it will come back later…). So I remove the batter, read the serial number (quite long, but hey, we got through it) and that was about it.

“Anything else I can help you with”, she asks. Sure, why not, while we’re here (9.15pm), I kind of ordered something with you guys about 2 weeks ago, and again a week ago and I was to be contacted within 24 hours, but hey, they kind of didn’t. But, since she was the T-hub helpdesk, no can do. I get put back in hold, this time for the Billing helpdesk.

Billing is a bigger problem as it took a tad bit longer to finally get put through. Yay! I explain my quest (2 weeks ago via chat, last week via the store, no confirmation, no contact, no email, no nothing). She starts up the system after I’ve identified myself (again) and she can see both orders. However, she sees differences in the packages created by both colleagues (who both came to the same final price though, so how that happens, you tell me).

Since I now clearly know what I want and what it will cost me, we can go quickly through the changes and she decides that it is probably best to cancel both previous orders (sure, but, wouldn’t it be nice to just let one go through and keep me, well, informed?) and we create a new one (again, exact same price, but totally different bundle with the same components, really…). Only one thing, she can’t finalise it as that is just something the Bundle Team can do. Incident report number 2, reference number whatever, I’ll get a call on Friday.

“Anything else I can do for you?”. (9.45pm) Well, apparently, when I removed the battery from the T-hub, as your colleague asked me to do, the touch screen on the T-hub has become unresponsive (I noticed this because, well, you have to do something during the time you are put on hold). “Let me just connect you to our T-hub helpdesk”

And so, an hour later, I was back full circle with the T-hub helpdesk. Again, the personal details, my phone number, name, age, first 4 digits of my pin and the ccv code, the works, I am identified (ok, that about the pin and ccv is not true, but come on, 3 times during 1 phone call from a landline supplied by Telstra themselves? really?)

Anyway, after first chewing away the legendary question: “Have you tried turning it off and on again” we again get to the point where I have to remove the battery and read out the serial number. (again, remember this!) Reinsert the battery, try again, the works. At some point during this conversation her flowchart also ended and the T-hub is going to be replaced.

To do so, they also need the serial number that is on the base station as well as on the handset that came with it. So I start with the number on the base station and after 3 digits I get this strange feeling that I know this number. With only one exception. Where the T-Hub had a T, the base station has a convenient B. Where the base station had the B, the handset had a C.

Curious as I am, I ask the nice lady: “Wouldn’t it have been easier for me, instead of removing the battery from the T-hub, simply to lift up the base station and read that serial number?”. “Yes, they are the same, except that T stands for T-hub, B for base station and C for Communicator”…

“So, basically, strictly hypothetical, since it is too late anyway, I could have told the first person the serial number, which is printed on the outside of the base station and she could have concluded the serial number of the T-hub without me having to remove the battery and consequently ruining the device’s touchscreen?”

“Yes, when we replace them, we have to replace all 3”. Anyway, I now have a 3rd incident number and they will send me a replacement set. (it’s now 10.15pm). But, since I got here, I’m not done yet…

“How about the contact list I have stored in the device? I just had all my contacts in there?” (this is not entirely true, I just had my mobile number in there just to test stuff, we didn’t get around to actually transfer any of our other contacts in the system, but hey, at this point, you got to ask).

“I can give you the number of our technical support team who will help you with that”. I thanked the lady. I’ll go a long way playing innocent customer, but I sure won’t spend another 30 minutes on the phone with the next person to explain to me that I can use the extra handset to store the numbers and transmit them to the new base station, as I had already read the manual.

Let’s just wait and see what Friday brings. It can only get better from here I guess…

 

Jack of all trades

One of the first things I noticed when applying for a job Down Under was that where in Europe they recruit specialists, down here, they primarily recruit the ‘jacks of all trade’

Probably because there is a huge lack of really good software engineers (they are here, but there’s not enough of them). Where in Europe, the roles ‘analyst’, ‘developer’ and ‘tester’ are strictly different roles.

Think of it like  the ‘Trias Politica’, where you differ between the power that defines law (politicians), the power that upholds law (police) and the power that checks if the other two are operating within the law (judges).

The same actually does go for software development projects. You’ve got (business) analyst (they who describe what is to be made), the developers (they who make what is described) and testers (they who check if what has been described is actually made).

As a developer in Europe, I’ve fought many battles with the analysts. Mostly about the ‘why’. I’ve fought many battles with the testers, mostly about the ‘how’. Although, we all agreed on the ‘what’, in my opinion, it’s great to work with a team that separates these powers.

Down here the rules seem to be completely different, if you are a ‘software specialist’, you are analyst, developer and tester in one. I can see the market value of a jack of all trades, but let me give you some examples why this isn’t working.

Analyst can go on forever. It’s in their job description to never stop wondering about the why, analyse it, break it down into measurable entities. These generally are people with a huge fear of failure. If they didn’t capture it, millions of lives will be dependant. That’s why analysts want to capture each and every aspect of anything, all the time.

Developers are inherently lazy, well, not all of them, but the better ones are. Developers love to cut corners, to do in one line what others do in 20, their laziness defines their efficiency. You do things once, maybe twice, but if you have to do them a third time? You write a program to do it for you. You never know when you have to do it a fourth time.

Testers are nitpicking bastards, developers are confronted with each and every corner they cut, but the testers caught them. It’s an ever ongoing war between the ‘can do’ mind of the developer and the ‘but you missed this’ mindset of the tester. Actually, there are no more roles so cut out for each other than testers and developers. The tester has to sign off what the developer made. Developers hate testers, but good developers hate not having testers even more. They are their conscience. I personally love testers who are capable of detecting my latest mischievous hack. And in the end, when developers age, with all their experience, they probably become the testers. But right now, they are probably too creative to even think about that fate.

As a developer, I hate analysts because they specify things I can’t build, I hate testers because they interpret the analysis in a different way. I love the analysts because they translate what the business needs, and I love the testers because they provide the security that I actually wrote what the users wants. And if it is a good team, we understand that we need each other. They probably hate me for seeing things too binary (‘it either is or isn’t damnit! there is no magic in computers!’)

Hence the Trias Politica, each working in a team, to obtain the same goals, but looking at the project with different angles. Where things start going wrong is where management thinks either of these roles can be performed by the other. Developers doing analysis or testing, or testers doing analysis. And beware of the day analysts start development.

Sadly enough, down here, it seems more common than I’ve been used to. There seems to be a common feeling that ‘everything with a computer’ can be done by ‘someone who does stuff with computers’. It’s like your parents asking you where the button for ‘bold’ is, because ‘you design chips, so you know everything about computers’

How do you tell them that knowing everything about bits and accumulators and registers doesn’t make you the same guy that knows how to underline or how to use italics in their word processor? Some of us may know how to do 2048 bits encryption, but we just may lack the knowledge of how to add 1 and 1 in an excel sheet.

There simply is no Jack of all Trades, and there never will be. In a computer science world, that has been a station passed. I can only hope that one day I will convince the people down here that you really need the 3 of us and that there simply isn’t one that is all 3.

The Meta-Question

I think I’ve been using Google as a search engine from the day Google emerged as a search engine. Which, of course, was there sole business when they did. It was the fastest, most accurate and the least gruesome of all search engines of it’s time. Just this textbox and two buttons (at least, I can’t remember the day the ‘I feel lucky’ button was introduced, to me, it seems like it’s always been there)

So there it was, this mostly white page, with nothing more than that colorful logo just shouting at you ‘let me find what you want!’. Back in those days, I didn’t even think of the Meta-Question. I just typed and found. Even back then, I wondered what good a Search Engine would be if it didn’t Find anything. To me, it was the ultimate Find Engine.

Many years have passed and still today, it’s my Find Engine of choice. If I can’t find in on Google, I’m sure as hell not going to Find it on Yahoo, Bing, Just Jeeves or Ask.com. If Google can’t find it, it simply becomes a question to which there is no answer. I know it told me the meaning of life.

Without Google, my IQ drops at least 20 points. I’ve always said that knowing where to find the answer is as valuable as actually knowing the answer. It no longer falls back on the answer, but how you state the Question. Just asking ‘I need pussy’, doesn’t really get you laid. It however, might get you a kitten though.

But I’ve never asked the engine the Meta-Question: ‘Where can I find it’. Today I asked the mother of all search engines that question. The number one pick as about a celebrity style website. Nah, that wasn’t what I was looking for. So I asked Yahoo. Guess you can find it at Totally Toddler. So I asked Bing. I seem to find it at Find. Except from advertising, I can’t enter what I’d like to find at all.

Concluding, none of the mayor ‘search engines’ actually find themselves. A simple question remains unanswered. Where can I find it leads to dead links and outskirts of the internet. There is simply no answer to that simple Meta-Question.

And if you ask ‘What can I find’, the biggest hit is the wayback machine. So, if you ask the internet what it is you can find, it will point you to the biggest archive of all stuff ever found but never searched for. The question where you can actually find stuff remains unanswered.

I guess that’s why they still call them ‘Search Engines’. It’s about time someone invented a ‘Find Engine’. I know I’m done Searching.

Blog

I’ve been thinking about this a long time. Years ago, I started a blog to document and sometimes contemplate my and my families move to Australia. You can find it at http://www.kemna.eu. In the beginning I’d just put up some new post about where in the process we were, how the kids were doing etc. It was a family blog.

However, more and more colleagues asked me to write about my work life as well. I resisted every attempt by pointing out that most of the people blogging do not really care about what they are saying, they care about how many people actually take notice of their exhibitionism.

I can remember, years ago, years before wordpress, blogger, facebook and others made it easy to blog, there was a guy who wrote a piece called ‘why I hate weblogs’. The original post is long gone but I tried to find it today and I’m pretty sure I found the correct content, but it’s just put into… a blog…

Actually, people have copied it to their blogs, threw it on facebook, it’s mentioned in a weblog about blogging, oh wait, this post is becoming just that. Stop!

Anyway, thankfully the internet has more memory than I have and the original can still be found at the Wayback Machine, the original post at October 13, 2002 up until it’s deletion in June 2008, and even the funny test can be taken, but the outcome will forever be gone…

I guess I’ll never know, and who’s going to read it anyway 🙂

Cheers!