{"id":215,"date":"2021-03-16T20:29:07","date_gmt":"2021-03-16T19:29:07","guid":{"rendered":"http:\/\/www.fabidouille.com\/?p=215"},"modified":"2021-03-16T20:35:19","modified_gmt":"2021-03-16T19:35:19","slug":"deep-learning-palms-on","status":"publish","type":"post","link":"https:\/\/www.fabidouille.com\/?p=215","title":{"rendered":"Deep learning palms-on"},"content":{"rendered":"

And Now for Something Completely Different, let\u2019s move away from concrete stuff and bite a bit of Machine Learning abstraction (which \u2013 who knows \u2013 we may plug in a \u201creal life\u201d apparatus) in the good company of indisputably the best story tellers of the 20th century, Carl Barks and Don Rosa.<\/p>\n

The Masters of a rich palmed world<\/h2>\n

 <\/p>\n

\u00a0Carl Barks, the creator\u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0\u00a0\"\"<\/a><\/p>\n

Don Rosa, the innovator\u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0\u00a0\"\"<\/a><\/p>\n

\n
\n

For the unlucky ones who don\u2019t know them, or rather the lucky ones who don\u2019t know them (who can\u2019t worship them (yet)), Carl Barks and Don Rosa are respectively the creator of the wide Duck world and the guy who refreshed the \u201cfranchise\u201d.<\/p>\n<\/div>\n<\/div>\n

They both share a tender respect for the both mild and deep universe they created.<\/p>\n

A dive into the awe of an anthropomorphic universe, where adventures always mean discoveries, where gold is often but not always at stake, sometimes Goldie is to be found.<\/p>\n

\"\"<\/a>
Goldie, an ambiguous acquaintance from Scrooge\u2019s youth<\/figcaption><\/figure>\n

 <\/p>\n

As for Goldie, characters are often not what they appear, Donald Duck despite his uncontrollable anger happens to be the best uncle for his three nephews, most cherished treasures of the stingly Scrooge McDuck are clearly not worth a lot of dollars or the lucky-lookingly Gladstone Glander does not live such a wonderful life, really.<\/p>\n

Clearly, that\u2019s an invitation to see characters behind appearances.<\/p>\n

Ok, then what?<\/h2>\n

Now you may wonder, why such hommage in this blog?<\/p>\n

Well, your devoted blog author (myself) being (as you guessed) an idolatrous of this universe, I thought this could be an entertaining opportunity to hands-on Tensorflow, a very accessible neural network framework.<\/p>\n

Long term goal is to apply training\/inference to real-world devices with sensors, and actuators.<\/p>\n

Short term goal, depicted in this article, will be one of the simplest activity offered out-of-the-box by such frameworks (many<\/a>\u00a0coexist) : classification.<\/p>\n

How? Well, Carl Barks and Don Rosa have very different drawing styles, obvious to the passionate reader.<\/p>\n

So\u00a0I was wondering whether a cold, (a priori) unpassionate convolutional neural net could figure that out.<\/strong><\/p>\n

Why Tensorflow?<\/h2>\n

First, let me restate that this blog is about experiments done out of curiosity, and clearly not written by an expert in the fields and techniques involved.<\/p>\n

That being said, your servant (myself) happened to briefly look at the state-of-the-art of neural networks\u2026 at the end of last century (the nineties).<\/p>\n

A time when perceptrons and Hopfield topologies were ruling keywords under the then-more-restricted neural network realm. A time those under 20 years old cannot know, a more primitive time where sigmo\u00efd were the only viable activation functions, where you had to write your C++ code for your gradient descent (for the learning phase) with your bare p\u0336a\u0336l\u0336m\u0336s\u0336 hands.<\/p>\n

Good that time is now gone!<\/p>\n

So, TensorFlow is a miracle of simplicity exposed to the user compared to that. Plus it has good community, good support among hardware vendors (GPU acceleration via Nvidia Cuda and also A\u0336T\u0336I\u0336 AMD ROCm) but also Google Cloud services (such as the excellent Google Colaboratory, a free environment where you can enjoy dedicated 12 GB with Nvidia Volta GPU or Google\u2019s own TPU).<\/p>\n

Will that be enough hardware muscles for only a cold, maybe unpassionate network to perceive the specifics of the two creators? Let\u2019s find out!<\/p>\n

By the way, what is a neural network, and how could it classify drawings?<\/h2>\n

Hmm, vast subject.<\/p>\n

Your minion (the author of these lines) wrote some lines on that subject back in 2001 in a dedicated website (iacom.fr.st, now redirecting to some unclear Japanese contents).<\/p>\n

I would have bet without question that this site has completely vanished into the ashes of the past eternity of the web, but to my great surprise the web archive project\u00a0kept\u00a0one<\/strong>\u00a0copy back in 2013<\/a>. Only the front page was saved\u2026<\/p>\n

Anyway, one of the best short introductions to contemporary n\u0336e\u0336u\u0336r\u0336a\u0336l\u0336 \u0336n\u0336e\u0336t\u0336w\u0336o\u0336r\u0336k\u0336s\u0336 Deep Learning I can recommend is here:\u00a0https:\/\/codelabs.developers.google.com\/codelabs\/cloud-tensorflow-mnist<\/a><\/p>\n

If you can afford more time, I\u2019d recommend the Stanford Coursera course (https:\/\/www.coursera.org\/learn\/machine-learning<\/a>).<\/p>\n

\"\"<\/a><\/p>\n

Coursera course will reward your studying labours with a certificate, see this as a way to say thank you to Coursera for their otherwise free (& excellent) courses.<\/figcaption><\/figure>\n

This ellipsis being made, let\u2019s jump into the dataset preparation!<\/p>\n

Dataset preparation<\/h2>\n

\u201cAll quality and no quantity would make our model a dull model.\u201d<\/p>\n

Nobody in particular<\/cite><\/p><\/blockquote>\n

\"\"<\/a><\/p>\n

\"\"<\/a><\/p>\n

We have plenty of input data available. Btw, check out the superb volumes edited by Glenat (in France) with lots of annotation\/anecdotes, some by Don Rosa himself for the collection dedicated to his work!<\/p>\n

In particular if you sometimes fail to discover the hidden D.U.C.K. (Dedicated to Unca Carl from Keno<\/em>) dedications in the first panels of Don Rosa stories.<\/p>\n

Digitization<\/h3>\n

I have no doubt that your imagination makes this step unnecessary.<\/p>\n

At the end of this stage, we had:<\/p>\n

\n\n\n\n\n\n\n\n
<\/td>\nStats<\/td>\n<\/tr>\n
Carl Barks, in English<\/td>\n1158 files
\n1.35 GB
\n1.17 MB per file in average<\/td>\n<\/tr>\n
Carl Barks, in French<\/td>\n1544 files
\n2.8 GB
\n1.8 MB per file in average<\/td>\n<\/tr>\n
Don Rosa, in English<\/td>\n689 files
\n0.484 GB
\n0.702 MB per file in average<\/td>\n<\/tr>\n
Don Rosa, in French<\/td>\n464 files
\n0.313 GB
\n0.676 MB per file in average<\/td>\n<\/tr>\n<\/tbody>\n<\/table>
Heterogeneous sources, but nearly even FR\/EN ratio (to limit possible learning bias)<\/figcaption><\/figure>\n

Filtering<\/h3>\n

Whether we use a stack of only-dense layers (aka old school perceptron) or have mostly convolutional layers, input layer will always have a fixed size, exactly mapping pixel resolution of input images \u2026<\/p>\n

This means we have to feed our network during training phase with images of same pixel resolution.<\/p>\n

\"\"<\/a>
Terminology: A is a panel, B is a borderless panel, green area is a tier, rest is called gutters<\/figcaption><\/figure>\n
\"\"<\/a>
A typical Don Rosa page<\/figcaption><\/figure>\n

 <\/p>\n

\"\"<\/a>
A slightly less usual, but not so unusual, Don Rosa page<\/figcaption><\/figure>\n
\n

 <\/p>\n<\/div>\n

\"\"<\/a>
A typical Carl Barks page, pretty regular 8 panels page<\/figcaption><\/figure>\n

 <\/p>\n

As shown above, terminology related to comics had to be fixed beforehand.<\/p>\n

Also, regarding their respective styles, while Carl Barks has a purer, \u201cclassic\u201d stroke of pen, Don Rosa, the engineer by training, has definitely profound sense of detail.<\/p>\n

While Carl Barks will often sketch everyday life scenes, Don Rosa tends to lean towards epic setups. Naturally, there are many shades of grey in between, these are only tendencies felt by the human reader which I am.<\/p>\n

\"\"<\/a>
Carl Barks sometimes deviates from 8 panels layout for dynamic scenes<\/figcaption><\/figure>\n

 <\/p>\n

Ideally, in order to compare drawing styles of the two authors, we would feed images with same resolutions.<\/p>\n

\"\"<\/a>
One of the many famous Barks lithographies<\/figcaption><\/figure>\n

 <\/p>\n

\"\"<\/a>
Don Rosa also drawed full size images (sometimes for covers)<\/figcaption><\/figure>\n

 <\/p>\n

Both authors created full size images, which could be good samples. However, in case of Carl Barks the intention is more esthetic arts than support for story telling. His drawing style for lithographies is not similar to the one used in his comics. And we want to analyse the style used in the stories.<\/p>\n

Also, due to the vast amount of work and time to create such images, there are much less samples falling in these categories.<\/p>\n

So we really want to use for our dataset extracts from stories, not these full size images.<\/p>\n

Page panel extraction : Kumiko<\/h4>\n

Kumiko<\/a>\u00a0is an open source tool is a set of tools using OpenCV\u2019s contour detection algorithm to compute meta information on comic pages, such as panels.<\/p>\n

A first attempt was done to extract all panels using this tool, but sometimes the algorithm went wrong (borderless panels are difficult to identify). But normalisation would have implied image scaling, and panel shapes are too different to have non destructive shape uniformisation.<\/p>\n

So I decided to use images as input of neural networks (knowing that even reduced, such images may not be easy to digest \u2026).<\/p>\n

I went for a 1007\u00d7648 format, as a compromise between keeping details and not overwhelming our net.<\/p>\n

Many pages are descriptive, fully-fletched with text and various layouts.<\/p>\n

\"\"<\/a><\/p>\n

Also because of Don Rosa\u2019s old engineer habits some pages tend to be \u2026 singular ;), such as these very precious ones, depicting Uncle Scrooge\u2019s money bin secret plans:<\/p>\n

\"\"<\/a><\/p>\n

Useful for the Beagle boys, but clearly not for us.<\/p>\n

In order not to interfere with story telling pages, we use Kumiko to count the number of panels.<\/p>\n

Kumiko json output<\/p>\n

\"\"<\/a><\/p>\n

Some Python code helps us to automate the reading-kumiko-panel-meta-information-then-resizing part of input images.<\/p>\n

Note, alternatively a rescaling layer can be added at the beginning of our network topology, but that would mean more data to be sent to Colaboratory (as original images are always larger than target resolution).<\/p>\n

\"\"<\/a><\/p>\n

As can be seen, we only select images for which we detect more than three panels.<\/p>\n

Results look good, only meaningful images are left.<\/p>\n

However, some visually corrupted images were still present.<\/p>\n

\"\"<\/a><\/p>\n

Which were easily moved away using, again, simple python logic:<\/p>\n

\"\"<\/a><\/p>\n

At the end, 1025 clean files remained for Barks\/FR, 692 for Barks\/EN, 598 for Rosa\/EN and 377 for Rosa\/FR.<\/p>\n

Now, let\u2019s get into Tensorflow!<\/h2>\n

As suggested previously, Google Colaboratory is well integrated into Google\u2019s world. Notably Google Drive.<\/p>\n

So after having pushed all files filtered and scaled by our previous python logic (~2.3 GB of .png files compressed in .tar.bz2) to a Google Drive folder, it could be mounted on a Colaboratory session.<\/p>\n

\"\"<\/a><\/p>\n

After some manual (re)exploration of the dataset, it is time to convert images to Keras input dataset format, and at the same time we want to specify the training\/validation ratio.<\/p>\n

As we have enough images, we reserve 20% of our sample for the validation:<\/p>\n

 <\/p>\n

\"\"<\/a><\/p>\n

As we want to train the network to distinguish between Rosa and Barks images, we have two classes, one for each.<\/p>\n

The two classes are implicitly detected after the names of the subdirectories from main dataset directories (one class for subdirectory barks\/ and the other for rosa\/).<\/p>\n

Then comes the network topology.<\/p>\n

Convulational networks tend to be better then multi dense-layers networks because their cubes-2-cubes internal transformations are somehow better at matching shapes.<\/p>\n

Also, that smells newer than our retro perceptrons, so let\u2019s go for it!<\/p>\n

\"\"<\/a><\/p>\n

Chosen topology<\/figcaption><\/figure>\n

Another disclaimer: this experiment was done in a brief amount of time, so no time at all was spend to experiment on various topology variants.<\/p>\n

Only the number of neurons in last dense layer was changed a bit.<\/p>\n

The rescaling layer at the beginning is just to normalize inputs from 0->255 ranges (RGB colors) to 0->1.<\/p>\n

An interesting yet simple explanation for choice of relu activation functions can be read\u00a0here<\/a>.<\/p>\n

\"\"<\/a>
A summary of model as seen by Tensorflow<\/figcaption><\/figure>\n

 <\/p>\n

An important thing is to NOT prefetch images in training or validation sets – as it is often the case by default -, otherwise training will quickly cause OOMs (not enough RAM) the session, whatever the batch size is. Because our input images are too big.<\/p>\n

After 15 epochs (aka training steps) with batches of 16 images each, we converged to a surprisingly good 99.44% accuracy!\u00a0\ud83d\ude42<\/p>\n

It took ~22 min to train with a GPU enabled session.<\/p>\n

Be careful, by default session starts with no hardware accelerator\u2026 so make sure to select it at begining of your session.<\/p>\n

\"\"<\/a><\/p>\n

\"\"<\/a><\/p>\n

Training and accuracy validation plots seem to show 15 epochs should be enough.<\/p>\n

However, after attempting some predictions, it appeared another run of 15 epochs did improve results for tested samples.<\/p>\n

Despite convergence towards lower accuracy overall:<\/p>\n

 <\/p>\n

\"\"<\/a><\/p>\n

Interesting is that during 2nd run validation accuracy fell a bit before reincreasing, and redecreasing (could be interesting to test with more epochs). This might be related to overfitting.<\/p>\n

First more natural steps would be to stop learning as soon as validation accuracy drops, and training accuracy stops growing (slightly after epoch 15).<\/p>\n

As we shuffled both training and validation sets, plus we dedicated a good part to validation (20%), it is not sure massively increasing the amount of training data would help a lot. Decreasing number of model parameters could possibly be key. Or alternatively, we could augment training\/validation sets by performing image transformations such as scaling (could help identify full size drawings).<\/p>\n

Maybe dropout could be used to reset some neurons at each training iteration. Also, playing with learning rate could be worth testing in a future session.<\/p>\n

Funny enough, while reaching a lower accuracy (99.26%), results were much better on samples we used for prediction.<\/p>\n

And now, let\u2019s predict our Duck stories author<\/h2>\n

Let\u2019s have a look together at some samples not part of training nor validation set found randomly on the net, and see whether prediction was correct or not (and in parenthesis the confidence factor), and my comments for each picture in legend.<\/p>\n

\"\"<\/a>
Correct! (72% confidence) This one from Barks is interesting, this is an unfavorable case, with introductive format. But the model gets the correct answer \ud83d\ude42<\/figcaption><\/figure>\n
\"\"<\/a>
Correct (100%)! Easy one, very \u201ctypical\u201d from Barks, nothing fancy here.<\/figcaption><\/figure>\n
\"\"<\/a>
Barks, correct(100%)! Nothing fancy either.<\/figcaption><\/figure>\n
\"\"<\/a>
Barks, correct (100%)!<\/figcaption><\/figure>\n
\"\"<\/a>
Incorrect (99.86%) \ud83d\ude41 A bit of disappointment for this one. This is an \u201cepic\u201d theme from Carl Barks, not so typical and more Don Rosa like. Even a human would have hesitated! Maybe an improvement of the training could change the result \u2026<\/figcaption><\/figure>\n
\"\"<\/a>
This was Barks, but identified as Rosa. Incorrect (100%) \ud83d\ude41<\/figcaption><\/figure>\n
\"\"<\/a>
Incorrect again (100%) \ud83d\ude41 Looks like black&white comics don\u2019t fit in learned Barks\u2019 pattern \u2026 maybe such samples could be added to the training set (but there are few of them)<\/figcaption><\/figure>\n
\"\"<\/a>
Incorrect (99.86%) \ud83d\ude41 Disappointment. Clearly the colorisation is aged, but the trait is specific to Barks (and note this is the same story as in the 3rd picture)<\/figcaption><\/figure>\n
\"\"<\/a>
Correct (88%) \ud83d\ude42 A full page drawing from Don Rosa, correctly found. Nice!<\/figcaption><\/figure>\n
\"\"<\/a>
Correct (100%) \ud83d\ude42 A really newish style for this one, dynamic traits, vivid colors. Clearly on Rosa side, but could have probably been misidentified as other recent Duck artists.<\/figcaption><\/figure>\n
\"\"<\/a>
Correct (58%). Another full page drawing from Don Rosa, correctly found, yet with less confidence. Obvious to the enthusiast however.<\/figcaption><\/figure>\n
\"\"<\/a>
Correct (100%) \ud83d\ude42 An easy win again for the model. Very dynamic Rosa style.<\/figcaption><\/figure>\n
\"\"<\/a>
Correct (100%) This one from Rosa is too easy for our model!<\/figcaption><\/figure>\n
\"\"<\/a>
Correct (100%). A full page image, with looots of details, typical for Rosa. \ud83d\ude42 \ud83d\ude42<\/figcaption><\/figure>\n
\"\"<\/a>
Correct (96%). \u201cI know Don Rosa\u201d something murmured \ud83d\ude09<\/figcaption><\/figure>\n
\"\"<\/a>
Correct (100%), Rosa naturally<\/figcaption><\/figure>\n
\"\"<\/a>
Correct (100%). Rosa in Danish, another big homeland for Duck stories.<\/figcaption><\/figure>\n
\"\"<\/a>
Rosa at 100%: unknown artist. But I agree with the model.<\/figcaption><\/figure>\n
\"\"<\/a>
Rosa at 99%. This time, I disagree! \ud83d\ude09<\/figcaption><\/figure>\n

To conclude<\/h2>\n

This hands-on on \u201cDeep Learning\u201d was a very fun experiment for me, I really appreciated digging into that.<\/p>\n

I\u2019d say results are beyond my initial expectations, however I felt a bit disappointed when the learned model fell in some of the set traps (the \u201cepic\u201d Bark panel for example).<\/p>\n

Now, to which extent the model learned the artistic style (whatever this means) or just the layout, the vividness of colorisation or drawing techniques (half a century separates the two!), difficult to say!<\/p>\n

However the capability of the model to still have good predictions even for images slightly more exotic than the sometimes rigid 8-panels format used for lots of Carl Barks artworks is appreciable.<\/p>\n

Now a next step could be to try other esteemed, modern artists such as Romano Scarpa. Would it still be distinctive enough for a simple model such as the one we trained here?<\/p>\n

\"From
Ducks as drawn by Romano Scarpa<\/figcaption><\/figure>\n

And who knows, maybe this kind of approach could pave way for artificially generated Duck stories in a not-so-distant future?<\/p>\n","protected":false},"excerpt":{"rendered":"

And Now for Something Completely Different, let\u2019s move away from concrete stuff and bite a bit of Machine Learning abstraction (which \u2013 who knows \u2013 we may plug in a \u201creal life\u201d apparatus) in the good company of indisputably the best story tellers of the 20th century, Carl Barks and Don Rosa. The Masters of…<\/p>\n

Continue reading →<\/span><\/a><\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[1],"tags":[],"class_list":["post-215","post","type-post","status-publish","format-standard","hentry","category-non-classe"],"_links":{"self":[{"href":"https:\/\/www.fabidouille.com\/index.php?rest_route=\/wp\/v2\/posts\/215","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.fabidouille.com\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.fabidouille.com\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.fabidouille.com\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/www.fabidouille.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=215"}],"version-history":[{"count":8,"href":"https:\/\/www.fabidouille.com\/index.php?rest_route=\/wp\/v2\/posts\/215\/revisions"}],"predecessor-version":[{"id":283,"href":"https:\/\/www.fabidouille.com\/index.php?rest_route=\/wp\/v2\/posts\/215\/revisions\/283"}],"wp:attachment":[{"href":"https:\/\/www.fabidouille.com\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=215"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.fabidouille.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=215"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.fabidouille.com\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=215"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}