Blackbird module and MrBean module do not work together in JDK17

[davidconnard]

We have a project that uses both Blackbird and MrBean modules. We are trying to upgrade this project to JDK17, but are hitting a number of problems. We seem to have encountered a problem where Blackbird module is incompatible with MrBean module, when abstract classes exist and when run under JDK17.

The following code works fine in JDK11:

    public static void main(String[] args) {
        try {
            JsonMapper jsonMapper = JsonMapper.builder()
                    .disable(MapperFeature.USE_GETTERS_AS_SETTERS)
                    .build();
            jsonMapper
                    .registerModule(new ParameterNamesModule())
                    .registerModule(new MrBeanModule())
                    .registerModule(new BlackbirdModule())
                    .disable(DeserializationFeature.FAIL_ON_UNKNOWN_PROPERTIES)
                    .disable(SerializationFeature.FAIL_ON_EMPTY_BEANS);

            String json = jsonMapper.writeValueAsString(new Wrapper(new Square(1.5d)));
            System.out.println("json = " + json);
            Shape shape = jsonMapper.readValue(json, Wrapper.class).shape;
            System.out.println("shape = " + shape);
        } catch (Exception e) {
            e.printStackTrace();
        }
    }

With associated classes that look like:

public class Wrapper {
    @JsonTypeInfo(use = JsonTypeInfo.Id.CLASS)
    public final Shape shape;

    @JsonCreator
    Wrapper(Shape shape) {
        this.shape = shape;
    }
}

public abstract class Shape {
    @Override
    public String toString() {
        return ToStringBuilder.reflectionToString(this); // engage commons-lang3 reflective toString for the purposes of this test
    }
}

public class Square extends Shape {
    public final double edgeLength;

    @JsonCreator
    public Square(double edgeLength) {
        this.edgeLength = edgeLength;
    }
}

When run under JDK11, it produces the following output:

json = {"shape":{"@class":"foo.Square","edgeLength":1.5}}
shape = foo.Square@574b560f[edgeLength=1.5]

However, when run under JDK17, it fails:

json = {"shape":{"@class":"foo.Square","edgeLength":1.5}}
java.lang.invoke.LambdaConversionException: Invalid caller: com.fasterxml.jackson.module.mrbean.generated.foo.Shape
        at java.base/java.lang.invoke.AbstractValidatingLambdaMetafactory.<init>(AbstractValidatingLambdaMetafactory.java:125)
        at java.base/java.lang.invoke.InnerClassLambdaMetafactory.<init>(InnerClassLambdaMetafactory.java:175)
        at java.base/java.lang.invoke.LambdaMetafactory.metafactory(LambdaMetafactory.java:336)
        at com.fasterxml.jackson.module.blackbird.deser.CreatorOptimizer.lambda$createOptimized$1(CreatorOptimizer.java:90)
        at com.fasterxml.jackson.module.blackbird.util.Unchecked.lambda$supplier$1(Unchecked.java:41)
        at com.fasterxml.jackson.module.blackbird.deser.CreatorOptimizer.createOptimized(CreatorOptimizer.java:98)
        at com.fasterxml.jackson.module.blackbird.deser.BBDeserializerModifier.updateBuilder(BBDeserializerModifier.java:96)
        at com.fasterxml.jackson.databind.deser.BeanDeserializerFactory.buildBeanDeserializer(BeanDeserializerFactory.java:286)
        at com.fasterxml.jackson.databind.deser.BeanDeserializerFactory.createBeanDeserializer(BeanDeserializerFactory.java:126)
        at com.fasterxml.jackson.databind.deser.DeserializerCache._createDeserializer2(DeserializerCache.java:415)
        at com.fasterxml.jackson.databind.deser.DeserializerCache._createDeserializer(DeserializerCache.java:350)
        at com.fasterxml.jackson.databind.deser.DeserializerCache._createAndCache2(DeserializerCache.java:264)
        at com.fasterxml.jackson.databind.deser.DeserializerCache._createAndCacheValueDeserializer(DeserializerCache.java:244)
        at com.fasterxml.jackson.databind.deser.DeserializerCache.findValueDeserializer(DeserializerCache.java:142)
        at com.fasterxml.jackson.databind.DeserializationContext.findNonContextualValueDeserializer(DeserializationContext.java:632)
        at com.fasterxml.jackson.databind.deser.BeanDeserializerBase.resolve(BeanDeserializerBase.java:539)
        at com.fasterxml.jackson.module.blackbird.deser.SuperSonicBeanDeserializer.resolve(SuperSonicBeanDeserializer.java:79)
        at com.fasterxml.jackson.databind.deser.DeserializerCache._createAndCache2(DeserializerCache.java:294)
        at com.fasterxml.jackson.databind.deser.DeserializerCache._createAndCacheValueDeserializer(DeserializerCache.java:244)
        at com.fasterxml.jackson.databind.deser.DeserializerCache.findValueDeserializer(DeserializerCache.java:142)
        at com.fasterxml.jackson.databind.DeserializationContext.findRootValueDeserializer(DeserializationContext.java:642)
        at com.fasterxml.jackson.databind.ObjectMapper._findRootDeserializer(ObjectMapper.java:4805)
        at com.fasterxml.jackson.databind.ObjectMapper._readMapAndClose(ObjectMapper.java:4675)
        at com.fasterxml.jackson.databind.ObjectMapper.readValue(ObjectMapper.java:3629)
        at com.fasterxml.jackson.databind.ObjectMapper.readValue(ObjectMapper.java:3597)
        at io.atlassian.cloudprovisioner.Main.main(Main.java:65)

Note that: you can see in the above stacktrace that MrBean is engaged, and has created the generated class for the abstract com.fasterxml.jackson.module.mrbean.generated.foo.Shape. Under JDK17, Blackbird has failed to perform some operation on that generated class (which is not entirely unsurprising, as JDK17 is much stricter about class access). Also note that the Blackbird CreatorOptimizer.createOptimized method (from the above stacktrace) has the following code comment in it:

            // The LambdaMetafactory requires a specifying interface, which is not possible to provide
            // for methods with arbitrary parameter lists.  So we have to use a spread invoker instead,
            // which is not a valid target for the metafactory.  Instead, we wrap it in a trampoline to
            // avoid creating megamorphic code paths and hope that code inlining covers up our reflective sins.

Note the "reflective sins" part ... I don't really grok the entire comment, but I'd guess that it refers to something that has not really worked and is causing this to fail under JDK17.

In our specific case, we found a work-around, which was simply to drop the abstract from our super-class (ie. the Shape class in the above example) - ie. thereby avoiding MrBean from engaging for this specific class. While this seems to avoid the problem for this one class we found out about, and obviously leaves it lurking elsewhere in our system.

Another possible approach we looked at was to avoid MrBean for all abstract types, eg:

                    .registerModule(new MrBeanModule(new AbstractTypeMaterializer() {
                        @Override
                        protected boolean _suitableType(JavaType type) {
                            return !Modifier.isAbstract(type.getRawClass().getModifiers()) && super._suitableType(type);
                        }
                    }))

but, this seems to de-value the MrBean module somewhat..?

MrBean documentation states that it is not compatible with polymorphic types, so, in one sense, this is not entirely surprising. However, I can't see anywhere where is parses classes for a @JsonTypeInfo annotation..?

Note that #99 has some more commentary around the compatibility issues with polymorphic types

[GedMarc]

This is probably more related to the usage of proxy considering it only impacts abstract classes,
I'll take a look this evening (gmt+2), it probably needs to move across to the new jdk17 mechanics,
I'll try to keep it backwards compatible, but there are obviously restrictions, otherwise a multirelease jar may be required

[stevenschlansker]

Hi @davidconnard,

there's recently been a fix around Proxy access and Blackbird (#210)

With this fix applied, and running Java 20, I re-ran your (slightly modified) test case and it seems to work now:

java = 20.0.1
json = {"shape":{"@class":"com.fasterxml.jackson.module.blackbird.misc.MrBeanTest$Square","edgeLength":1.5}}
shape = Square [edgeLength=1.5]

Could you please verify if this is still a problem with the fix applied? Currently it is only in the 2.16 branch which is not released yet, but if this fixes your errors and you want the fix sooner, @cowtowncoder might consider backporting the fix to 2.15 (or I can upon request)

[cowtowncoder]

I am ok with retrofit for 2.15; if anyone wants to submit PR that'd be cleanest (happy to merge).

[davidconnard]

Thanks for the fix!!

I haven't had a chance to test your 2.16 branch, however, I'm confident that the example I supplied was a good replication for the problem we were seeing.

A backport to 2.15 would be nice, but not totally necessary, we did find an alternate way of working around it.

Thanks!!

[cowtowncoder]

In that case let's not yet backport. 2.16 (rc1) may be out not later than the next 2.15.x patch anyway.