Mask layer using CustomLayerInterface causes rendering issues on layers above - help needed to understand stencil buffer

[bameyrick]

I have created a mask layer using CustomLayerInterface to mask a raster layer behind it. The mask layer works correctly in hiding the raster behind, however it is causing issues with a line layer above which comes from a vector tile source.

The custom mask layer uses a WebGL stencil buffer to punch out polygons from a buffer that covers the whole screen, and I think this is where the issue lies as according to this answer Mapbox (which MapLibre was forked from):

uses a stencil buffer to track which layer goes under/over which one even if they are not drawn in order.

---

Issue Observed

Here is an example of the rendering issues observed where only some of the line layer is rendered:

---

Code example

The custom MaskLayer:

import { isEqual } from '@qntm-code/utils';
import earcut, { flatten } from 'earcut';
import { Feature, Polygon } from 'geojson';
import maplibregl from 'maplibre-gl';

export class MaskLayer implements maplibregl.CustomLayerInterface {
  public type = 'custom' as const;

  private readonly shaderMap = new Map();

  private buffers: WebGLBuffer[] = [];
  private vertexCounts: number[] = [];
  private aPos: GLint;

  private fullScreenQuadBuffer: WebGLBuffer;

  private map?: maplibregl.Map;

  constructor(public readonly id: string) {}

  private polygons: Feature<Polygon>[] = [];

  public setPolygons(polygons: Feature<Polygon>[]) {
    if (isEqual(polygons, this.polygons)) {
      return;
    }

    this.polygons = polygons;

    this.map?.triggerRepaint();
  }

  // Helper method for creating a shader based on current map projection - globe will automatically switch to mercator when some condition is fulfilled.
  private getShader(gl: WebGLRenderingContext, shaderDescription: maplibregl.CustomRenderMethodInput['shaderData']): WebGLProgram {
    // Pick a shader based on the current projection, defined by `variantName`.
    if (this.shaderMap.has(shaderDescription.variantName)) {
      return this.shaderMap.get(shaderDescription.variantName);
    }

    const vertexSource = `#version 300 es
      // Inject MapLibre projection code
      ${shaderDescription.vertexShaderPrelude}
      ${shaderDescription.define}

      in vec2 a_pos;

      void main() {
          gl_Position = projectTile(a_pos);
      }`;

    // create GLSL source for fragment shader
    const fragmentSource = `#version 300 es

      precision highp float;
      out highp vec4 fragColor;
      uniform vec4 color;

      void main() {
          fragColor = color;
      }`;

    // create a vertex shader
    const vertexShader = gl.createShader(gl.VERTEX_SHADER)!;
    gl.shaderSource(vertexShader, vertexSource);
    gl.compileShader(vertexShader);

    // create a fragment shader
    const fragmentShader = gl.createShader(gl.FRAGMENT_SHADER)!;
    gl.shaderSource(fragmentShader, fragmentSource);
    gl.compileShader(fragmentShader);

    // link the two shaders into a WebGL program
    const program = gl.createProgram();
    gl.attachShader(program, vertexShader);
    gl.attachShader(program, fragmentShader);
    gl.linkProgram(program);

    this.aPos = gl.getAttribLocation(program, 'a_pos');

    this.shaderMap.set(shaderDescription.variantName, program);

    return program;
  }

  public onAdd(map: maplibregl.Map, gl: WebGLRenderingContext): void {
    this.map = map;

    this.fullScreenQuadBuffer = gl.createBuffer();
    gl.bindBuffer(gl.ARRAY_BUFFER, this.fullScreenQuadBuffer);
    gl.bufferData(
      gl.ARRAY_BUFFER,
      new Float32Array([
        -1,
        -1, // bottom left
        1,
        -1, // bottom right
        -1,
        1, // top left
        1,
        1, // top right
      ]),
      gl.STATIC_DRAW,
    );
  }

  public render(gl: WebGLRenderingContext, args: maplibregl.CustomRenderMethodInput): void {
    this.createBuffers(gl);

    const program = this.getShader(gl, args.shaderData);
    gl.useProgram(program);
    gl.uniformMatrix4fv(
      gl.getUniformLocation(program, 'u_projection_fallback_matrix'),
      false,
      args.defaultProjectionData.fallbackMatrix, // convert mat4 from gl-matrix to a plain array
    );
    gl.uniformMatrix4fv(
      gl.getUniformLocation(program, 'u_projection_matrix'),
      false,
      args.defaultProjectionData.mainMatrix, // convert mat4 from gl-matrix to a plain array
    );
    gl.uniform4f(gl.getUniformLocation(program, 'u_projection_tile_mercator_coords'), ...args.defaultProjectionData.tileMercatorCoords);
    gl.uniform4f(gl.getUniformLocation(program, 'u_projection_clipping_plane'), ...args.defaultProjectionData.clippingPlane);
    gl.uniform1f(gl.getUniformLocation(program, 'u_projection_transition'), args.defaultProjectionData.projectionTransition);

    // Enable stencil testing
    gl.enable(gl.STENCIL_TEST);
    gl.stencilFunc(gl.ALWAYS, 1, 0xff);
    gl.stencilOp(gl.KEEP, gl.KEEP, gl.REPLACE);
    gl.stencilMask(0xff);
    gl.clear(gl.STENCIL_BUFFER_BIT);

    // First pass: Draw polygons into stencil buffer
    gl.colorMask(false, false, false, false);
    // Draw each polygon separately
    this.buffers.forEach((buffer, index) => {
      gl.bindBuffer(gl.ARRAY_BUFFER, buffer);
      gl.enableVertexAttribArray(this.aPos);
      gl.vertexAttribPointer(this.aPos, 2, gl.FLOAT, false, 0, 0);
      gl.drawArrays(gl.TRIANGLES, 0, this.vertexCounts[index]);
    });

    // Second pass: Draw fullscreen quad with stencil test
    gl.colorMask(true, true, true, true);
    gl.stencilFunc(gl.NOTEQUAL, 1, 0xff);
    gl.stencilMask(0x00);

    // Draw full screen quad
    const colorLoc = gl.getUniformLocation(program, 'color');
    gl.uniform4f(colorLoc, 1, 1, 1, 1); // White overlay
    gl.bindBuffer(gl.ARRAY_BUFFER, this.fullScreenQuadBuffer);
    gl.enableVertexAttribArray(this.aPos);
    gl.vertexAttribPointer(this.aPos, 2, gl.FLOAT, false, 0, 0);
    gl.drawArrays(gl.TRIANGLE_STRIP, 0, 4);

    // Clean up
    gl.disable(gl.STENCIL_TEST);
  }

  private createBuffers(gl: WebGLRenderingContext): void {
    this.buffers = [];
    this.vertexCounts = [];

    this.polygons.forEach(({ geometry }) =>
      geometry.coordinates.forEach((coordinates) => {
        const { buffer, vertexCount } = this.processPolygon(coordinates as [number, number][], gl);

        this.buffers.push(buffer);
        this.vertexCounts.push(vertexCount);
      }),
    );
  }

  private processPolygon(coordinates: [number, number][], gl: WebGLRenderingContext): { buffer: WebGLBuffer; vertexCount: number } {
    const flatCoords: number[] = [];
    const resultVertices: number[] = [];

    coordinates.forEach(([lng, lat]: [number, number]) => {
      const mercatorCoord = maplibregl.MercatorCoordinate.fromLngLat({ lng, lat });
      flatCoords.push(mercatorCoord.x, mercatorCoord.y);
    });

    const { vertices, holes, dimensions } = flatten([coordinates]);
    const triangles = earcut(vertices, holes, dimensions);

    triangles.forEach((index) => resultVertices.push(flatCoords[index * 2], flatCoords[index * 2 + 1]));

    const buffer = gl.createBuffer();
    gl.bindBuffer(gl.ARRAY_BUFFER, buffer);
    gl.bufferData(gl.ARRAY_BUFFER, new Float32Array(resultVertices), gl.STATIC_DRAW);

    return { buffer, vertexCount: resultVertices.length / 2 };
  }
}

Example of usage:

import * as maplibregl from 'maplibre-gl';
import { cogProtocol } from '@geomatico/maplibre-cog-protocol';
import { Feature, Polygon } from 'geojson';

import { MaskLayer } from './mask-layer';

// Add a cog protocol to load geotiff tiles
maplibregl.addProtocol('cog', cogProtocol);

const map = new maplibregl.Map({
  container: 'map',
  style: {
    version: 8,
    sources: {
      boundaries: {
        type: 'vector',
        tiles: [`some-tile-url`]
      },
    },
    layers: [
       {
          id: 'admin_boundaries',
          type: 'line',
          source: 'boundaries',
          'source-layer': 'admin_boundaries',
          paint: {
            'line-color': '#000',
          },
       }
    ],
  },
});

const maskLayer = new MaskLayer('mask');

// Add the mask layer behind the admin_boundaries line layer
map.addLayer(maskLayer, 'admin_boundaries');

map.addSource('geotiff', {
  type: 'raster',
  url: `cog://${tiffUrl}`,
  tileSize: 256,
});

// Add the geotiff raster layer behind the mask layer
map.addLayer({
  id: 'geotiff',
  type: 'raster',
  source: 'geotiff',
}, 'mask');

const polygons: Feature<Polygon>[] = [
  // Some polygons
];

maskLayer.setPolygons(polygons);

The layer order ends up as :

• Geotiff raster layer
• Custom MaskLayer
• Line vector layer

I have attempted to save the current stencil buffer state before drawing the mask and then restoring it afterwards but this doesn't seem to resolve the issue.

Example of attempt to restore stencil buffer:

public render(gl: WebGLRenderingContext, args: maplibregl.CustomRenderMethodInput): void {
  // Existing projection setup code...

  // Save current stencil buffer state
  const savedStencilMask = gl.getParameter(gl.STENCIL_WRITEMASK);
  const savedStencilFunc = gl.getParameter(gl.STENCIL_FUNC);
  const savedStencilRef = gl.getParameter(gl.STENCIL_REF);
  const savedStencilValueMask = gl.getParameter(gl.STENCIL_VALUE_MASK);
  const savedStencilFail = gl.getParameter(gl.STENCIL_FAIL);
  const savedStencilPassDepthFail = gl.getParameter(gl.STENCIL_PASS_DEPTH_FAIL);
  const savedStencilPassDepthPass = gl.getParameter(gl.STENCIL_PASS_DEPTH_PASS);

  // Existing code to enable stencil testing and draw mask...

  // Restore previous stencil buffer state
  gl.stencilMask(savedStencilMask);
  gl.stencilFunc(savedStencilFunc, savedStencilRef, savedStencilValueMask);
  gl.stencilOp(savedStencilFail, savedStencilPassDepthFail, savedStencilPassDepthPass);

  // Clean up
  gl.disable(gl.STENCIL_TEST);
}

---

How do I make sure the stencil buffer used in the render function does not conflict with MapLibre's own stencil buffer that is being used to draw subsequent layers? Is this even the correct approach to take?

[bameyrick]

I think I might have solved this (by sheer luck) 🥳! I added this line to the end of my render function and that seems to have solved my rendering issues on the line layer above:

this.map?.painter.clearStencil();

I don't know if my MaskLayer implementation would be any help, but wondering if it could be a basis for a native MapLibre mask layer as suggested in #3237