Wednesday, December 20, 2006

The real problem is it isn't actually "sensible"

One of the big news story of the day is John Reid's u-turn on the ID card database. Essentially, he's decided that instead of starting with a fresh schema the ID cards will be based on multiple databases already in existence. According to John Reid, this is "lower risk, more efficient and faster". Now, I don't support the ID Card policy on principle, however, the argument that using already existing systems rather than starting afresh will be "lower risk, more efficient and faster" is, frankly, bollocks.

Starting with "lower risk", what Reid is really saying is less risk of overrun in costs due to poor estimates at the outset. However, from a data integrity point of view, using already existing systems, which are known to hold inaccurate data, and are accessed and updated by separate groups under different protocols, is actually an increase in risk.

There is more potential for information leakage, failures, and, most fundamentally, a wider net of loosely controlled human beings affecting data that could significantly impact on people's live if mistakes are made. Let's not be under any illusions here, mistakes will be made. I don't say this for political reasons I say it in operational terms.

Large-scale databases, especially those that carry out masses of transactional queries and updates will always have problems. "Fixing" data is a necessary fact of operational life. Place that reality in the context of multiple databases under multiple theatres of control, and you have a very risky situation indeed. Especially when it is about a card that, if introduced, will apparently become the de facto point for all manner of access to services, and other general day-to-day living.

The idea that such a system will also be more "efficient" is, to say the very least fanciful. I imagine that the argument is based on the notion of efficiency when related to data gathering. Why gather lots of data you already have in other databases that are already able to be easily queried? However, the problem of efficiency, again from an operational understand is highly questionable.

Again, the disconnected management of these discreet systems to be used, means that multiple layers of bureaucracy will stifle operational administration. In Reid's proposal, when a mistake occurs which effect data integrity, the process for rectifying it will be wholly inefficient and laborious. The consequential impact on those who mistakes impact could potentially be massive.

Take for example if the ID card becomes a requirement for receiving medical treatment. What happens when data integrity is lost for someone who can then no longer be managed through the pervasive all seeing system of the state? What happens, when due to a failure in one system, a person becomes effectively a non-person for a period whilst the bureaucracy grinds on between the different stakeholders to rectify that situation?

Finally, one has to assume that when Reid claims the system will be "faster" he is only referring to the idea that they can get it all up and running within their given deadlines. However, there cannot be a serious argument that the system itself, once running will be faster when it will make multiple access and query requests to multiple databases in multiple locations across saturated bandwidth networks? Throw on top an overhead for encryption (which one assumes must be planned), and it's pretty clear that this system will be anything but faster than the original proposal.

There is of course a little bit of political trickery and triangulation going on here as well though. After all, opposing the Reid U-turn proposals on the grounds that it will not be lower risk, efficient or faster than the original plan, suggests, consequentially, that one is supporting the original plan - this is not necessarily so. There is also something else we should note in Reid's comment though. He claimed that "[d]oing something sensible is not necessarily a U-turn".

Putting aside the absurdity of this argument regarding the U-turn not being a U-turn, these proposals are not - it seems - actually sensible at all. The system that will be produced now, will - if it ever manages to become operational - be high risk, inefficient and slower. There is nothing sensible about Reid's statement, but then that doesn't surprise me in the slightest. He's a politician trying to talk about IT without having a clue about what the real implications are.

Hat Tip: The Spine for the image


Dr Dan H. said...
20 Dec 2006 13:07:00  

There's actually a technical reason why this is a daft idea. When you plan a database system, you should know how much data is going to get put in it, so when you spec out the disks you say "Aha, this system is to hold X data, so I shall cunningly spec thee disks to hold 2X of data to cope with mission creep".

Now Mr Reid is saying "I want the databases designed for X amount of data to hold 100X data. Make it so".

Problem is, this isn't easy to arrange. Doubling the size of a database is usually fairly easy; more disks in more arrays and you add some more datafiles to them.

Increasing by a few orders of magnitude is not nearly so easy. First, you need the extra space. Second, you need to know how active the data will be; is it going to be pretty much static, or will it chop and change a lot?

If the former, you optimise for a data warehouse; fast read, slower write. If the latter, you spread the data out more taking up more room in the datafiles, and allow faster writes. This ties into the disk arrays you need; can you get away with NetApps, or does it have to be fast SCSI all the way?

Then, how are you going to back it up? A few terabytes, no problem; existing systems can do that, though retrieval is a bugger. Go bigger than that per unit, and backup gets to be very difficult; you really need a remote secure site to mirror to. Oh, and gigabit networking or better to connect to it.

This is a "solution" though up by an ill-advised luddite who doesn't know what he's talking about. Building a set of new databases from scratch would actually be cheaper than modifying already existing units, but dumbo Reid and his moronic boss Blair apparently cannot see that.

Frankly, I pity the poor bugger who has to try to make it work; I know I'd run like hell if I saw those two ignoramuses coming my way.

dizzy said...
20 Dec 2006 13:28:00  

I would've said all that but I feared my readers would start glazing over if I strarted talking about things like NetApps. Good response though. is a participant in the Amazon Europe S.à.r.l. Associates Programme, an affiliate advertising programme designed to provide a means for sites to earn advertising fees by advertising and linking to