Sorry - 2TB of data is sooooooooo much disk space!

2 TB, or 2 terabytes, or 2,000,000,000,000 bytes is sooooooooo much disk space, of course you can’t have a clone of of the production database for your development, UAT, or pre-production/staging environments!  After all, 2TB of disk space is ….. well that is a whopping £74.99 (if you choose to purchase the physical hardware, or a few clicks and a few dollars/month if you are in a commercial third-party cloud).

The clone provided for development and other purposes will be a cut down version of the database; a table that contains 0.5billion rows in production will contain 3000 rows in your development, UAT, and so on databases. That is enough!

If the development etc. database

• isn’t on comparable hardware to that as observed on a production system, all bets on performance or throughput metrics are off
• doesn’t contain a snapshot of representative data before coding changes are propagated from UAT to pre-production and then production systems, all bets are off as to whether it will even work

To put things another way, rolling out a database code change without developing against, and testing against, an intact production database clone is effectively saying …. the implications of the code changes are going to be tested on the production system and can bed-in there.

To put things a third way, all developer(s) can honestly report is that the deployment package compiles, builds, and so on, and that he/she/they have dip-tested the changes on system that is nothing like the production box.

To put things a fourth way, on roll-out day, it should not be necessary to wish someone “Good Luck”, to make a prayer, or to be on tenterhooks as you know the phone is going to ring during the rollout because of some major snafu, as it always does!

The only word I can use to describe this scenario is recklessness. There is no excuse. Not one. I am not feigning annoyance either – this blog post has been prompted by recent experience subcontracting on a ‘big data’ project. The constraint was placed on the primary contractor (a large organisation) by the client (a huge organisation) infrastructure teams. Given what is at stake should something go wrong, I just do not understand it.

— Published by Mike, 10:41:22 24 July 2017 (CET)