How to load pytorch model file directly?

[csxiang18]

Question

I've already export a pytorch model into a pt file. For example: model/my_model.pt, this directory is under the resources directory.

When I want to load this model, I use this way:

String modelPath = "model";
URI modelUri = classLoader.getResource(modelPath).toURI();
Path modelDir = Paths.get(modelUri);
Model ptModel = Model.newInstance("my_model", Device.cpu());
ptModel.load(modelDir);

Is there any other way that I could use like this, load the pt file directly: ptModel.load("model/my_model.pt")?

[lanking520]

Starting from 0.9.0, you can load from modelDir from ModelZoo:

https://github.com/awslabs/djl/blob/master/api/src/main/java/ai/djl/repository/zoo/Criteria.java#L343

You can refer to our doc here: http://docs.djl.ai/master/docs/load_model.html. There are many ways to load: like http, hdfs, s3 or simply local file paths are all supported.

If you are looking for loading the model directly, you can do the followings:

ptModel.load(Paths.get("model"), "my_model");

Which is very similar when you create new instance of the model. There seemed to be no way to bypass setting model name from the Model API. But doable if you try to load from the ZooModel api

[frankfliu]

Can you try examples in DJL repository directly?

cd djl/pytorch/pytorch-model-zoo
gradlew test

[dtneverdie]

Can you try examples in DJL repository directly?

cd djl/pytorch/pytorch-model-zoo
gradlew test

There is no gradlew.bat in this directory so i used ..\..\ gradlew test command instead and it showed that BUILD SUCCESSFUL in 12s 9 actionable tasks: 9 executed.
Well i ran my project again and exceptions still exist. Failed to load Pytorch native library problem still the same but now it is caused by
Caused by: java.lang.UnsatisfiedLinkError: C:\Users\57495\.djl.ai\pytorch\1.8.1-cu102-win-x86_64\0.13.0-SNAPSHOT-cu102-djl_torch.dll: **找不到指定的程序**。 at java.lang.ClassLoader$NativeLibrary.load(Native Method) at java.lang.ClassLoader.loadLibrary0(ClassLoader.java:1941) at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1824) at java.lang.Runtime.load0(Runtime.java:809) at java.lang.System.load(System.java:1086) at ai.djl.pytorch.jni.LibUtils.loadNativeLibrary(LibUtils.java:351) at ai.djl.pytorch.jni.LibUtils.loadLibrary(LibUtils.java:77) at ai.djl.pytorch.engine.PtEngine.newInstance(PtEngine.java:50) ... 4 more。 The Chinese words mean 'cant find appointed application'. I dont know why this error came in local language.

[frankfliu]

DJL 0.13.0-SNAPSHOT should use pytorch-native-auto:1.9.0-SNAPSHOT

You can use DJL 0.12.0 if use pytorch-native-auto:1.8.1

[dtneverdie]

DJL 0.13.0-SNAPSHOT should use pytorch-native-auto:1.9.0-SNAPSHOT

You can use DJL 0.12.0 if use pytorch-native-auto:1.8.1

now it shows ai.djl.engine.EngineException: torch::jit::load()received a file fromtorch.save(), but torch::jit::load()can only load files produced bytorch.jit.save() at ai.djl.pytorch.jni.PyTorchLibrary.moduleLoad(Native Method) at ai.djl.pytorch.jni.JniUtils.loadModule(JniUtils.java:1359) at ai.djl.pytorch.engine.PtModel.load(PtModel.java:85) at ai.djl.repository.zoo.BaseModelLoader.loadModel(BaseModelLoader.java:152) at ai.djl.repository.zoo.Criteria.loadModel(Criteria.java:166) at ai.djl.repository.zoo.ModelZoo.loadModel(ModelZoo.java:123) at mxnet.inference.TestingModel.loadDefaultDnnModel(TestingModel.java:52) at mxnet.inference.TestingModel.main(TestingModel.java:62)

[frankfliu]

@dtneverdie
As the error message state, your model is not jit traced model. see: http://docs.djl.ai/master/docs/pytorch/how_to_convert_your_model_to_torchscript.html

[frankfliu]

@csxiang18 If you put model in resource file, most likely you will pack the model in a jar file when deploy it.

I this case, you have to use jar:// url to load the model: http://docs.djl.ai/docs/load_model.html#load-model-from-a-url

Criteria<Image, Classifications> criteria = Criteria.builder()
        .setTypes(Image.class, Classifications.class) // defines input and output data type
        .optTranslator(ImageClassificationTranslator.builder().setSynsetArtifactName("synset.txt").build())
        .optModelUrls("jar:///model")
        .optEngine("PyTorch")
        .optModelName("my_model")
        .build();

ZooModel<Image, Classifications> model = criteria.loadModel();

[csxiang18]

Thanks for answering this question, as it's my first time to try DJL, so may some API I do not use very correctly. In fact I want to pack the model in the jar file as a lib to deploy so that others could use this. Maybe I could state the problem more clearly.
@frankfliu my model is a self-defined model, for-example, a 3 layer dnn, no existing model in ModelZoo. I follow this doc: Export Pytorch Model to export the model as a my_model.pt file. Then I put the model file under the resource directory model/my_model.pt so that I could use getResouce to load the model.

ClassLoader classLoader = LoadModelTest.class.getClassLoader();
String modelPath = "model";
URI modelUri = classLoader.getResource(modelPath).toURI();

Then I want to provide a lib, and write like this, so others could use the function like this: LoadModelTest.predict(features)

package com.test;

public class LoadModelTest {
    public static Model model;
    static {
        try {
            loadModel();
        } catch (Exception e) {
            e.printStackTrace();
        }
    }
    public static void main(String[] args) {
        float[] features = new float[199];
        float[] output = predict(features);
        System.out.println(output[0]);
    }

    public static float[] predict(float[] features) {
        try {
            Predictor<float[], float[]> predictor = model.newPredictor(new FloatToFloatTranslator());
            return predictor.predict(features);

        } catch (Exception e) {
            e.printStackTrace();
        }
        return null;
    }

    public static void loadModel() throws Exception {
        ClassLoader classLoader = LoadModelTest.class.getClassLoader();
        String modelPath = "model";
        URI modelUri = classLoader.getResource(modelPath).toURI();
        Path modelDir = Paths.get(modelUri);
        model = Model.newInstance("my_model", Device.cpu());
        model.load(modelDir);
    }

    private static final class FloatToFloatTranslator implements Translator<float[], float[]> {

        @Override
        public Batchifier getBatchifier() {
            return Batchifier.STACK;
        }

        @Override
        public float[] processOutput(TranslatorContext translatorContext, NDList ndList) throws Exception {
            NDArray tempArray = ndList.get(0);
            return tempArray.toFloatArray();
        }

        @Override
        public NDList processInput(TranslatorContext translatorContext, float[] features) throws Exception {
            NDManager manager = translatorContext.getNDManager();
            return new NDList(manager.create(features));
        }
    }
}

The above script works fine when I wrote the lib. However, when I package this lib, it could not load the model correctly. As my_model.pt is in the Jar file and Paths could not parse the model directory correctly. So how could I solve this?

Another question is that in the doc, it said that the predictor is not thread-safe. So the above way is the correct way to predict features?

[frankfliu]

@csxiang18
The PyTorch engine can only load model from file system. So the loadModel() can only take Path

If you package your classes in a jar file, the URL returned from classLoader.getResource is a special URL that only supported by JVM something like: "jar://your.jar!/model". Path modelDir = Paths.get(modelUri); won't give you a correct file system path, so load model will fail since Path is not valid.

To resolve this issue, you can use Criteria API to load your model as I posted earlier. The api can take care both jar and unjared case.

Criteria<Float[], Float[]> criteria = Criteria.builder()
        .setTypes(Float[].class, Float[].class) // defines input and output data type
        .optTranslator(new FloatToFloatTranslator())
        .optModelUrls("jar:///model")
        .optModelName("my_model")
        .build();

ZooModel<Float[], Float[]> model = ModelZoo.loadModel(criteria);