Using Kotlin for Inference with custom model

[jesse-schallenberger]

Hello,

I've been encountering an issue trying to utilize DJL for inference on a custom PyTorch model.

I recently trained a PyTorch model on the 17 flowers dataset. From there I updated to a JIT model via tracing, and verified the inference output to match on the 17 flowers images.

I attempted to leverage the notebook here. I converted to Kotlin and had no expectation that would cause any real issues.

Here is a sanitized version of what I have:

internal fun main(args: Array<String>) {

    val modelPath = "file:///path"  // .pt file path

    val artifactId = "ai.djl.localmodelzoo:torchscript_17flowers"

    val pipeline = Pipeline()

    pipeline.add(CenterCrop(224, 224))
        .add(Resize(224, 224))
        .add(ToTensor())
        .add(Normalize(floatArrayOf(0.485f, 0.456f, 0.406f), floatArrayOf(0.229f, 0.224f, 0.225f)))

    val translator = ImageClassificationTranslator.builder()
        .setPipeline(pipeline)
        .optSynsetArtifactName("synset.txt")
        .optApplySoftmax(true)
        .build();

    System.setProperty("ai.djl.repository.zoo.location","build/pytorch_models/torchscript_17flowers")

    val criteria = Criteria.builder()
        .setTypes(Image::class.java, Classifications::class.java) // defines input and output data type
        .optTranslator(translator)
        .optArtifactId(artifactId) // defines which model to load
        .optProgress(ProgressBar())
        .build()

    val model = ModelZoo.loadModel(criteria)

    // single image test
    var img = ImageFactory.getInstance().fromUrl("https://image.jpg");
    img.getWrappedImage()

    val predictor: Predictor<Image, Classifications> = model.newPredictor()
    val classifications: Classifications = predictor.predict(img)
    val best = classifications.best<Classifications.Classification>()
}

I attempted to verify the inference output was matching, and I saw a great number of mismatches. I did not write code to determine the exact percentage, but my expectation was 100% match.

I was curious if there were any ideas of where to look to see why the mismatch might be occurring or if there is something obvious that would cause this to happen?

Thanks

[zachgk]

One reason you could see the accuracy drop is if your pre-processing differs between the Java and Python. I would recommend checking this first. I see two potential issues with it:

1. You are using both CenterCrop and Resize, but you should only need one. Doing both will just have the second one do nothing
2. ToTensor should be the last operation in your transformation pipeline. It actually converts between the NHWC which is used for pre-processing and NCHW which is used for models.

[jesse-schallenberger]

The processing was different.