Importing Models into CoreML

By Bolot Kerimbaev on May 06, 2018


About

How do I import a trained model into CoreML?

Apple released Core ML framework at WWDC 2017, which allowed developers to integrate machine learning into their iOS applications. In this screencast, we’ll show you how to import Keras image classification models into CoreML.

Basic knowledge or experience building iOS applications is recommended.

For the More Curious...

Apple Developer - Core ML
Keras Documentation
BNR Guide to iOS
BNR Guide to Swift
iOS & Swift Courses

Transcript

Importing Keras Models into CoreML

My name is Bolot Kerimbaev. I’m an iOS and Android developer at Big Nerd Ranch. Today I will show you how to import Keras image classification models into CoreML. For the purpose of demonstration, I will use a simple convolutional neural network that classifies chess pieces in the printed chessboard diagrams.

There are 6 types of chess pieces, King, Queen, Rook, Knight, Bishop and Pawn, in one of two colors. Thus, there are 12 classes of items, so this problem is very similar to MNIST.

In this screencast, we’ll show you how to import Keras image classification models into CoreML.

CoreMLTools

CoreMLTools is a Python utility that converts machine learning models from Keras, Caffe, and scikit-learn to CoreML format. If you don’t already have coremltools, you can install them using PIP. For example, using Python3, you can run this in the terminal:

sudo python3 -m pip install -U coremltools

You will also need to install h5py utility, which is needed to import the models that Keras exports.

sudo python3 -m pip install -U h5py

Conversion

The default conversion method is very simple:

import coremltools
coreml_model = coremltools.converters.keras.convert(model)
coreml_model.author = 'Bolot Kerimbaev'
coreml_model.short_description = 'Predict something, basic form'
coreml_model.save('model_v01.mlmodel')

Here, the model is a Keras model, either an object in memory or saved to disk in h5 format.

By default, the input and output types are MLMultiArray, CoreML's counterpart to numpy ND arrays. Default input and output names are input1, input2, output1, etc.

To make the model more convenient to use in an iOS project, it is better to override the defaults.

We can make the model even easier to use in our project if we override input name and indicate that it's an image, as well as specify class labels and change the output name:

import coremltools
coreml_model = coremltools.converters.keras.convert(model, input_names=['image'], image_input_names=['image'], class_labels=['K', 'Q', 'R', 'B', 'N', 'P', 'k', 'q', 'r', 'b', 'n', 'p'], output_names=['probabilities'])
coreml_model.author = 'Bolot Kerimbaev'
coreml_model.short_description = 'Predict something, with class labels'
coreml_model.save('model_v02.mlmodel')

Adding Models to Xcode

Drag and drop the generated CoreML model files into Xcode. Select the files and compare the basic/default conversion versus the one where we specified input/output names, indicated that the input is an image, and provided class labels.

In the basic/default conversion case, the input is called input1 and has the type of MLMultiArray (Double 1 x 22 x 22), which corresponds to grayscale images 22x22 pixels. The output is called output1, with the type MLMultiArray (Double 12), which corresponds to the array of probabilities for the 12 classes (chess pieces).

In the second version of the model, the input and output are better labeled and will be easier to use in code. The input is called image and is of type Image (Grayscale 22x22). The output is actually split into two, one called classLabel and the other probabilities, which is now a Dictionary which maps the class labels to their corresponding probabilities.

Let's take a look at the Swift model class that Xcode generated.

This means that we can load the model and start using it in our project by instantiating the Swift model class:

let chess = ChessKerasCNN_v02()

Add the new predict(imageName:) function as an extension to the generated ChessKerasCNN_v02 class. We will use the CoreMLHelpers project to make it easier to pass image files as input.

extension ChessKerasCNN_v02 {
    internal func predict(image: UIImage, expectedLabel: String) {
        guard let pixelBuffer = image.pixelBufferGray(width: 22, height: 22) else {
            print("couldn't convert the image \(expectedLabel)")
            return
        }
        guard let prediction = try? prediction(image: pixelBuffer) else {
            print("couldn't make a prediction for \(expectedLabel)")
            return
        }
        print("Expected: \(expectedLabel), prediction: \(prediction.classLabel), probabilities: \(prediction.probabilities)")
    }
}

Now we can classify the images that are bundled with the project or even the ones taken with the camera:

// load images...
chess.predict(image: blackKnight, expectedLabel: "n")
chess.predict(image: whitePawn, expectedLabel: "P")

The output will look like this:

Expected: p, prediction: p, probabilities: ["b": 0.29357486963272095, "r": 0.0034754548687487841, "N": 0.0011770286364480853, "p": 0.56082522869110107, "n": 0.0039671962149441242, "R": 0.081584960222244263, "B": 0.0075710322707891464, "K": 0.00050108536379411817, "q": 0.039405584335327148, "k": 0.00060429668519645929, "P": 0.0022438699379563332, "Q": 0.0050693545490503311]

If we didn't specify the class labels, the output would be a 12-element array, containing only the probabilities:

[0.00050108536379411817, 0.0050693545490503311, 0.081584960222244263, 0.0075710322707891464, 0.0011770286364480853, 0.0022438699379563332, 0.00060429668519645929, 0.039405584335327148, 0.0034754548687487841, 0.29357486963272095, 0.0039671962149441242, 0.56082522869110107]

We would have to find the highest probability in the array, then lookup which class it corresponds to. But now the CoreML model contains both the dictionary mapping the labels to probabilities and the class label field that is set to the label with the highest probability.

End

In this screencast, we looked at how we can convert a Keras model to the format understood by CoreML and Xcode. We made our model easier to use by specifying the appropriate conversion parameters.

Thank you for watching. My name is Bolot Kerimbaev, I am a sensei at Big Nerd Ranch.

Downloads

Project files are only available for subscribers. Create an account today to access our collection of screencasts, skill packs, and more.

Comments