MongoDB - A very humbling first experience

I need a MongoDB database populated with some test data – a handful of small thumbnail images and a few other fields for development (so just junk/test data for the time being and weeks down the line I can populate with real business data). I have JIRA task with only one day allocated. On top of all of this, I know little, next to nothing, about MongoDB. I had better stop mucking around and get my head down. Now!

Step 1. Install MongoDB. Done. Well that was straight forward, and a download size just shy of 100MB to boot. I do not know what my expectations were but the binaries to install MySQL and Oracle are ca. 300MB and 3GB respectively.

Step 2. Watch the Pluralsight MongoDB database course. Wow! That was clear and concise and 2.5h of my time well invested.

Step 3. Generate a few thousand placeholder thumbnail images to populate into my MongoDB database that I will use solely for development purposes. For development, I will store the image as base64 (and the front-end browser or app can decode to display on-the-fly if/as necessary). Although leveraging GridFS is the obvious choice for larger (binary) file content, according to the MongoDB documentation, for small thumbnail images I intend to persist the base64 content directly within the database for now. Anything else at the moment is a bit heavyweight for my requirement. Storing directly within the database also greatly simplifies my database backup, replication, and distribution strategy, and these things are important when you do not know what you are doing!

For the thumbnails, although some random image would do, I want some identifying feature in the image file to enable me to visually confirm that I have the correct thumbnail image from any database query I might choose to execute. The obvious solution would be to have the image to contain a piece of text, such as a number. A quick Google search reveals lots of somebodies that have implemented this type of functionality online. I certainly do not wish to make a meal of this part of the task, so I intend to leverage their efforts. One somebody is Russell Heimlich. Using a few Unix shell commands, I can script up the rendering of a handful of base64 images containing a number using the offering exposed on Russell’s website.

Michaels-MacBook-Pro:imgfiles mdo$ 
Michaels-MacBook-Pro:imgfiles mdo$ 
Michaels-MacBook-Pro:imgfiles mdo$ ls -la
total 0
drwxr-xr-x 3 mdo staff 96 19 Okt 10:39 .
drwx------+ 23 mdo staff 736 19 Okt 10:39 ..
Michaels-MacBook-Pro:imgfiles mdo$ 
Michaels-MacBook-Pro:imgfiles mdo$ 
Michaels-MacBook-Pro:imgfiles mdo$ 
Michaels-MacBook-Pro:imgfiles mdo$ 
Michaels-MacBook-Pro:imgfiles mdo$ for i in $(seq -f %04g 8000 8009)
> do
> curl "https://dummyimage.com/80x80/000/fff.jpg&text=${i}" | base64 > "${i}.jpg.b64"
> done
 % Total % Received % Xferd Average Speed Time Time Time Current
 Dload Upload Total Spent Left Speed
100 1804 100 1804 0 0 3648 0 --:--:-- --:--:-- --:--:-- 3644
 % Total % Received % Xferd Average Speed Time Time Time Current
 Dload Upload Total Spent Left Speed
100 1678 100 1678 0 0 3355 0 --:--:-- --:--:-- --:--:-- 3356
 % Total % Received % Xferd Average Speed Time Time Time Current
 Dload Upload Total Spent Left Speed
100 1778 100 1778 0 0 3757 0 --:--:-- --:--:-- --:--:-- 3751
 % Total % Received % Xferd Average Speed Time Time Time Current
 Dload Upload Total Spent Left Speed
100 1782 100 1782 0 0 3776 0 --:--:-- --:--:-- --:--:-- 3783
 % Total % Received % Xferd Average Speed Time Time Time Current
 Dload Upload Total Spent Left Speed
100 1743 100 1743 0 0 3428 0 --:--:-- --:--:-- --:--:-- 3424
 % Total % Received % Xferd Average Speed Time Time Time Current
 Dload Upload Total Spent Left Speed
100 1775 100 1775 0 0 3811 0 --:--:-- --:--:-- --:--:-- 3817
 % Total % Received % Xferd Average Speed Time Time Time Current
 Dload Upload Total Spent Left Speed
100 1805 100 1805 0 0 3643 0 --:--:-- --:--:-- --:--:-- 3639
 % Total % Received % Xferd Average Speed Time Time Time Current
 Dload Upload Total Spent Left Speed
100 1699 100 1699 0 0 3584 0 --:--:-- --:--:-- --:--:-- 3591
 % Total % Received % Xferd Average Speed Time Time Time Current
 Dload Upload Total Spent Left Speed
100 1835 100 1835 0 0 3777 0 --:--:-- --:--:-- --:--:-- 3775
 % Total % Received % Xferd Average Speed Time Time Time Current
 Dload Upload Total Spent Left Speed
100 1809 100 1809 0 0 3662 0 --:--:-- --:--:-- --:--:-- 3669
Michaels-MacBook-Pro:imgfiles mdo$ 
Michaels-MacBook-Pro:imgfiles mdo$ base64 -D 8007.jpg.b64 > 8007.jpg
Michaels-MacBook-Pro:imgfiles mdo$ ls -la
total 96
drwxr-xr-x 14 mdo staff 448 19 Okt 10:39 .
drwx------+ 23 mdo staff 736 19 Okt 10:39 ..
-rw-r--r-- 1 mdo staff 2409 19 Okt 11:26 8000.jpg.b64
-rw-r--r-- 1 mdo staff 2241 19 Okt 11:26 8001.jpg.b64
-rw-r--r-- 1 mdo staff 2373 19 Okt 11:26 8002.jpg.b64
-rw-r--r-- 1 mdo staff 2377 19 Okt 11:26 8003.jpg.b64
-rw-r--r-- 1 mdo staff 2325 19 Okt 11:26 8004.jpg.b64
-rw-r--r-- 1 mdo staff 2369 19 Okt 11:26 8005.jpg.b64
-rw-r--r-- 1 mdo staff 2409 19 Okt 11:26 8006.jpg.b64
-rw-r--r-- 1 mdo staff 1699 19 Okt 11:27 8007.jpg
-rw-r--r-- 1 mdo staff 2269 19 Okt 11:26 8007.jpg.b64
-rw-r--r-- 1 mdo staff 2449 19 Okt 11:26 8008.jpg.b64
-rw-r--r-- 1 mdo staff 2413 19 Okt 11:26 8009.jpg.b64
Michaels-MacBook-Pro:imgfiles mdo$ 
Michaels-MacBook-Pro:imgfiles mdo$ 
Michaels-MacBook-Pro:imgfiles mdo$ 
Michaels-MacBook-Pro:imgfiles mdo$ 
Michaels-MacBook-Pro:imgfiles mdo$ 
Michaels-MacBook-Pro:imgfiles mdo$ 
Michaels-MacBook-Pro:imgfiles mdo$ cat 8007.jpg.b64 | fold -w60
/9j/4AAQSkZJRgABAQEAYABgAAD//gA+Q1JFQVRPUjogZ2QtanBlZyB2MS4w
ICh1c2luZyBJSkcgSlBFRyB2ODApLCBkZWZhdWx0IHF1YWxpdHkK/9sAQwAI
BgYHBgUIBwcHCQkICgwUDQwLCwwZEhMPFB0aHx4dGhwcICQuJyAiLCMcHCg3
KSwwMTQ0NB8nOT04MjwuMzQy/9sAQwEJCQkMCwwYDQ0YMiEcITIyMjIyMjIy
MjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIy/8AA
EQgAUABQAwEiAAIRAQMRAf/EAB8AAAEFAQEBAQEBAAAAAAAAAAABAgMEBQYH
CAkKC//EALUQAAIBAwMCBAMFBQQEAAABfQECAwAEEQUSITFBBhNRYQcicRQy
gZGhCCNCscEVUtHwJDNicoIJChYXGBkaJSYnKCkqNDU2Nzg5OkNERUZHSElK
U1RVVldYWVpjZGVmZ2hpanN0dXZ3eHl6g4SFhoeIiYqSk5SVlpeYmZqio6Sl
pqeoqaqys7S1tre4ubrCw8TFxsfIycrS09TV1tfY2drh4uPk5ebn6Onq8fLz
9PX29/j5+v/EAB8BAAMBAQEBAQEBAQEAAAAAAAABAgMEBQYHCAkKC//EALUR
AAIBAgQEAwQHBQQEAAECdwABAgMRBAUhMQYSQVEHYXETIjKBCBRCkaGxwQkj
M1LwFWJy0QoWJDThJfEXGBkaJicoKSo1Njc4OTpDREVGR0hJSlNUVVZXWFla
Y2RlZmdoaWpzdHV2d3h5eoKDhIWGh4iJipKTlJWWl5iZmqKjpKWmp6ipqrKz
tLW2t7i5usLDxMXGx8jJytLT1NXW19jZ2uLj5OXm5+jp6vLz9PX29/j5+v/a
AAwDAQACEQMRAD8A+f6KKKACiiigAooooAKKKKACiiigAorX0fwtrevxSzab
p8k0EJxJOSqRIeuC7EKD+NS6p4P1/RrL7be6c4s87Tcwuk0QPoXQlQfYmgDD
oq1p2m32rX0dlp1pNdXUhwkUKFmP4Ctu98AeKLC0nuZdKZ4oAWmNvNHMYgOp
YIxKgepFAHNUUV0tp8P/ABReWcN1HpTRxTjMP2iaOFpR6qrsCw9MDmgDmqKs
6hp17pN9LZahazWt1EcPFMhVl/A1t2PgHxPqVjDe2elPLbzLujcSoNw/FqAO
borqz8NPGAVm/sSUhVLHEsZOAMno3pXKUAdz4kiu9R8CeGbrTBJLo1latBdR
x8i3u/MdnZwOm5ShDHqBjtSfD5LqxGqareeZF4d+wXEF4XyI7hmjYRxD+8+8
qQByME8CrN3rE3w0lh0zQUSLWXs4pL/UJFEjBpYw4jjU5VVCuATgknPIHFS6
P4om+IGp2HhvxaEu/tJFrZ6gkYSe1duE+6AHTdgEEdCSCKAM6zmfw/8AC2W8
tW8q+1y+ezaVThhaxIrMgPYM7rn1C1zGjaxe6Dq1tqenTGK5t3DqR0PqpHdT
0I7iuo1uFv8AhVmhrwWsNWvbabDZw7LGw/RT+VcTQB6g/h3S0+M0n+iodFWA
6z5B+75XkeeEx6bvlx6V55q+rXuu6tc6nqM7T3Vw5d3Y/oPQDoB2Ar1Yjb8T
rjRsYuf+EW/s4At/y1+wg4+vb6145QB29/M/iD4V22oXTebf6Lfiy85uWa2k
Qsisep2srAegbFcRXY2C/Z/hDrUzj5bvV7WCM56lI5Xb8gw/MVzWkaZca1rF
npdou64u5khjz0yxxk+3PNAHTacf+EY8Cz6wMLqWtl7KzOcNFbLxNIB/tHEY
PoHrja6Xx3qUF94mltbF92m6Yi2Fl6eVF8u7/gTbn/4FXNUAdjLrfh/xLY2a
+IjqFnqlrAlsL+ziSdZ40GE8yNmU7gMDcG5wMjvUum6x4X8IzjUtFfUdV1mM
MLaa8t0t4LdiMeZsDuXYZ4BIAPPNcTRQB0nh7xJBZWV/o+s28t5o+oFXmWJw
JYpVztljJ43DJBB4IODWhZy+BNEu11GObVtamiYPb2VxapbRbhyPNYOxYA9g
Bn6VxdFAGsviXVE8Vf8ACSLcf8TP7V9q83GRvznp6dsenFbt3L4F1q7bUZZ9
W0aaZzJPZwWqXEQY8ny2LqVGegIOOnNcZRQB0XiXxBaaja2OkaPaS2mjWG8w
pM4aWaRsbpZCONxwBgcADApvhHXLbw5qV1qUqTNdpZTR2JjAISd12B2yeAoZ
jxk5Arn6KACiiigAooooAKKKKACiiigAooooAKKKKAP/2Q==
Michaels-MacBook-Pro:imgfiles mdo$ 
Michaels-MacBook-Pro:imgfiles mdo$ 
Michaels-MacBook-Pro:imgfiles mdo$ 
Michaels-MacBook-Pro:imgfiles mdo$ 
Michaels-MacBook-Pro:imgfiles mdo$ 

Here is image 8007.jpg generated prior to base64 encoding above.

So far no rocket science!

Step 4. Load these base64 encoded thumbnail images, and a couple of other fields, into MongoDB. This is the hard part.

Here is the core code I wrote.

for (i=8000; i<=8009; i++)
{
 var fb64 = cat(i + '.jpg.b64').trim();
 var b = BinData(0, fb64);
 db.foo.save({_id:NumberInt(i), fileNumber: NumberInt(i-8000), someOtherField0: 0, someOtherField1: 1, img: b});
}

Here is the unadulterated audit log of what I executed using within the Mongo shell.

Michaels-MacBook-Pro:imgfiles mdo$ 
Michaels-MacBook-Pro:imgfiles mdo$ 
Michaels-MacBook-Pro:imgfiles mdo$ 
Michaels-MacBook-Pro:imgfiles mdo$ mongo
MongoDB shell version v3.4.9
connecting to: mongodb://127.0.0.1:27017
MongoDB server version: 3.4.9
Server has startup warnings: 
2017-10-19T11:27:52.319+0100 I CONTROL [initandlisten] 
2017-10-19T11:27:52.319+0100 I CONTROL [initandlisten] ** WARNING: Access control is not enabled for the database.
2017-10-19T11:27:52.319+0100 I CONTROL [initandlisten] ** Read and write access to data and configuration is unrestricted.
2017-10-19T11:27:52.319+0100 I CONTROL [initandlisten] 
> show databases
admin 0.000GB
local 0.000GB
> 
> 
> use mikedb
switched to db mikedb
> show databases
admin 0.000GB
local 0.000GB
> 
> 
> for (i=8000; i<=8009; i++) {var fb64 = cat('i + '.jpg.b64').trim(); var b = BinData(0, fb64); db.foo.save({_id:NumberInt(i), fileNumber: NumberInt(i-8000), someOtherField0: 0, someOtherField1: 1, img: b}); }
WriteResult({ "nMatched" : 0, "nUpserted" : 1, "nModified" : 0, "_id" : 8009 })
> 
> 

What! I thought what I had done above was supposed to be the hard part! This cannot be right, can it? I have created a new database, somehow (I switched to a new database named mikedb, that didn’t exist), I created a new collection (object foo, by adding to a collection that also didn’t exist). WTF! Let’s give this all a quick smoke test in step 5!

Step 5: Was it really all that painless?

Michaels-MacBook-Pro:imgfiles mdo$ 
Michaels-MacBook-Pro:imgfiles mdo$ 
Michaels-MacBook-Pro:imgfiles mdo$ 
Michaels-MacBook-Pro:imgfiles mdo$ 
Michaels-MacBook-Pro:imgfiles mdo$ mongo
MongoDB shell version v3.4.9
connecting to: mongodb://127.0.0.1:27017
MongoDB server version: 3.4.9
Server has startup warnings: 
2017-10-19T11:38:52.319+0100 I CONTROL [initandlisten] 
2017-10-19T11:38:52.319+0100 I CONTROL [initandlisten] ** WARNING: Access control is not enabled for the database.
2017-10-19T11:38:52.319+0100 I CONTROL [initandlisten] ** Read and write access to data and configuration is unrestricted.
2017-10-19T11:38:52.319+0100 I CONTROL [initandlisten] 
> 
> 
> show databases
admin 0.000GB
local 0.000GB
mikedb 0.000GB
> db.getCollectionNames();
[ "foo" ]
> 
> 
> 
> 
> db.foo.count()
10
> 
> 
> 
> db.foo.find({_id: 8007}).pretty()
{
"_id" : 8007,
"fileNumber" : 7,
"someOtherField0" : 0,
"someOtherField1" : 1,
"img" : BinData(0,"/9j/4AAQSkZJRgABAQEAYABgAAD//gA+Q1JFQVRPUjogZ2QtanBlZ
yB2MS4wICh1c2luZyBJSkcgSlBFRyB2ODApLCBkZWZhdWx0IHF1YWxpdHkK/9sAQwAIBgYHB
gUIBwcHCQkICgwUDQwLCwwZEhMPFB0aHx4dGhwcICQuJyAiLCMcHCg3KSwwMTQ0NB8nOT04M
jwuMzQy/9sAQwEJCQkMCwwYDQ0YMiEcITIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyM
jIyMjIyMjIyMjIyMjIyMjIyMjIy/8AAEQgAUABQAwEiAAIRAQMRAf/EAB8AAAEFAQEBAQEBA
AAAAAAAAAABAgMEBQYHCAkKC//EALUQAAIBAwMCBAMFBQQEAAABfQECAwAEEQUSITFBBhNRY
QcicRQygZGhCCNCscEVUtHwJDNicoIJChYXGBkaJSYnKCkqNDU2Nzg5OkNERUZHSElKU1RVV
ldYWVpjZGVmZ2hpanN0dXZ3eHl6g4SFhoeIiYqSk5SVlpeYmZqio6Slpqeoqaqys7S1tre4u
brCw8TFxsfIycrS09TV1tfY2drh4uPk5ebn6Onq8fLz9PX29/j5+v/EAB8BAAMBAQEBAQEBA
QEAAAAAAAABAgMEBQYHCAkKC//EALURAAIBAgQEAwQHBQQEAAECdwABAgMRBAUhMQYSQVEHY
XETIjKBCBRCkaGxwQkjM1LwFWJy0QoWJDThJfEXGBkaJicoKSo1Njc4OTpDREVGR0hJSlNUV
VZXWFlaY2RlZmdoaWpzdHV2d3h5eoKDhIWGh4iJipKTlJWWl5iZmqKjpKWmp6ipqrKztLW2t
7i5usLDxMXGx8jJytLT1NXW19jZ2uLj5OXm5+jp6vLz9PX29/j5+v/aAAwDAQACEQMRAD8A+
f6KKKACiiigAooooAKKKKACiiigAorX0fwtrevxSzabp8k0EJxJOSqRIeuC7EKD+NS6p4P1/
RrL7be6c4s87Tcwuk0QPoXQlQfYmgDDoq1p2m32rX0dlp1pNdXUhwkUKFmP4Ctu98AeKLC0n
uZdKZ4oAWmNvNHMYgOpYIxKgepFAHNUUV0tp8P/ABReWcN1HpTRxTjMP2iaOFpR6qrsCw9MD
mgDmqKs6hp17pN9LZahazWt1EcPFMhVl/A1t2PgHxPqVjDe2elPLbzLujcSoNw/FqAOborqz
8NPGAVm/sSUhVLHEsZOAMno3pXKUAdz4kiu9R8CeGbrTBJLo1latBdRx8i3u/MdnZwOm5ShD
HqBjtSfD5LqxGqareeZF4d+wXEF4XyI7hmjYRxD+8+8qQByME8CrN3rE3w0lh0zQUSLWXs4p
L/UJFEjBpYw4jjU5VVCuATgknPIHFS6P4om+IGp2HhvxaEu/tJFrZ6gkYSe1duE+6AHTdgEE
dCSCKAM6zmfw/8AC2W8tW8q+1y+ezaVThhaxIrMgPYM7rn1C1zGjaxe6Dq1tqenTGK5t3DqR
0PqpHdT0I7iuo1uFv8AhVmhrwWsNWvbabDZw7LGw/RT+VcTQB6g/h3S0+M0n+iodFWA6z5B+
75XkeeEx6bvlx6V55q+rXuu6tc6nqM7T3Vw5d3Y/oPQDoB2Ar1Yjb8TrjRsYuf+EW/s4At/y
1+wg4+vb6145QB29/M/iD4V22oXTebf6Lfiy85uWa2kQsisep2srAegbFcRXY2C/Z/hDrUzj
5bvV7WCM56lI5Xb8gw/MVzWkaZca1rFnpdou64u5khjz0yxxk+3PNAHTacf+EY8Cz6wMLqWt
l7KzOcNFbLxNIB/tHEYPoHrja6Xx3qUF94mltbF92m6Yi2Fl6eVF8u7/gTbn/4FXNUAdjLrf
h/xLY2a+IjqFnqlrAlsL+ziSdZ40GE8yNmU7gMDcG5wMjvUum6x4X8IzjUtFfUdV1mMMLaa8
t0t4LdiMeZsDuXYZ4BIAPPNcTRQB0nh7xJBZWV/o+s28t5o+oFXmWJwJYpVztljJ43DJBB4I
ODWhZy+BNEu11GObVtamiYPb2VxapbRbhyPNYOxYA9gBn6VxdFAGsviXVE8Vf8ACSLcf8TP7
V9q83GRvznp6dsenFbt3L4F1q7bUZZ9W0aaZzJPZwWqXEQY8ny2LqVGegIOOnNcZRQB0XiXx
Baaja2OkaPaS2mjWG8wpM4aWaRsbpZCONxwBgcADApvhHXLbw5qV1qUqTNdpZTR2JjAISd12
B2yeAoZjxk5Arn6KACiiigAooooAKKKKACiiigAooooAKKKKAP/2Q==")
}

>
> 
> 
> db.foo.find({}, {_id:0, img:0})
{ "fileNumber" : 0, "someOtherField0" : 0, "someOtherField1" : 1 }
{ "fileNumber" : 1, "someOtherField0" : 0, "someOtherField1" : 1 }
{ "fileNumber" : 2, "someOtherField0" : 0, "someOtherField1" : 1 }
{ "fileNumber" : 3, "someOtherField0" : 0, "someOtherField1" : 1 }
{ "fileNumber" : 4, "someOtherField0" : 0, "someOtherField1" : 1 }
{ "fileNumber" : 5, "someOtherField0" : 0, "someOtherField1" : 1 }
{ "fileNumber" : 6, "someOtherField0" : 0, "someOtherField1" : 1 }
{ "fileNumber" : 7, "someOtherField0" : 0, "someOtherField1" : 1 }
{ "fileNumber" : 8, "someOtherField0" : 0, "someOtherField1" : 1 }
{ "fileNumber" : 9, "someOtherField0" : 0, "someOtherField1" : 1 }
> 
> 
> 

Yes, it seems it was that painless.

I have spent a considerable portion of my IT career designing and maintaining performant normalized relation database systems. Spinning up a development system containing a bit of test data for development purposes, on a development machine, always takes time, sometimes a lot of time. There is the database installation (taking hundreds of MB if not GB of disk space), privs, schema creation, table creation and FK relationships, data constraints etc. On top of this, there is creating a data layer mapping the database rows into objects. None of this here – the database is created when the first object is added, the collection is created when the first object is added, the result set structure is simple JSON, and so on (and mapping from JSON to a C# class for example is … trivial!). Furthermore in one execution of the Mongo find(….) command shown above, did you notice that I specified content I wanted excluded from the database query? Compare that to SQL where … well that is just not possible; in SQL you specify the columns you want returned in the result set (included), or all columns, not the columnar content you wish to be excluded.

This is all quite humbling, especially as I am surrounded by others with demonstrable competency in this technology. To a seasoned NoSQL developer, nothing I have written above is a big deal or even surprising. I certainly need to learn more about MongoDB – this little experience has certainly whet my appetite for more. Furthermore I have completed my 1 day JIRA task, and it has only just passed lunchtime. What shall I do for the rest of the day? Perhaps the time to learn more about MongoDB is right now. I know I have only scratched the surface thus far!

— Published by Mike, 12:12:12 19 October 2017 (GMT)