GraphDB crashes

[ddeboer]

GraphDB is crashing a lot, so let’s investigate /cc @coret.

https://triplestore.netwerkdigitaalerfgoed.nl/monitor/system#resource doesn't show much; it’s not possible to look further back in time?!

The last crash, from the GraphDB logs:

2023-11-10 21:10:35.770 | [INFO ] 2023-11-10 20:10:35,770 [repositories/registry \| o.e.r.h.s.r.n.NamespacesController] GET namespace list |  
-- | -- | --
  |   | 2023-11-10 21:10:35.674 | ... 112 common frames omitted |  
  |   | 2023-11-10 21:10:35.674 | at org.apache.catalina.connector.OutputBuffer.realWriteBytes(OutputBuffer.java:335) |  
  |   | 2023-11-10 21:10:35.674 | at org.apache.coyote.Response.doWrite(Response.java:606) |  
  |   | 2023-11-10 21:10:35.674 | at org.apache.coyote.http11.Http11OutputBuffer.doWrite(Http11OutputBuffer.java:193) |  
  |   | 2023-11-10 21:10:35.674 | at org.apache.coyote.http11.filters.ChunkedOutputFilter.doWrite(ChunkedOutputFilter.java:110) |  
  |   | 2023-11-10 21:10:35.674 | at org.apache.coyote.http11.Http11OutputBuffer$SocketOutputBuffer.doWrite(Http11OutputBuffer.java:540) |  
  |   | 2023-11-10 21:10:35.674 | at org.apache.tomcat.util.net.SocketWrapperBase.write(SocketWrapperBase.java:544) |  
  |   | 2023-11-10 21:10:35.674 | at org.apache.tomcat.util.net.SocketWrapperBase.writeBlocking(SocketWrapperBase.java:600) |  
  |   | 2023-11-10 21:10:35.674 | at org.apache.tomcat.util.net.SocketWrapperBase.doWrite(SocketWrapperBase.java:775) |  
  |   | 2023-11-10 21:10:35.674 | at org.apache.tomcat.util.net.NioEndpoint$NioSocketWrapper.doWrite(NioEndpoint.java:1431) |  
  |   | 2023-11-10 21:10:35.674 | at org.apache.tomcat.util.net.NioChannel.write(NioChannel.java:136) |  
  |   | 2023-11-10 21:10:35.674 | at java.base/sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:462) |  
  |   | 2023-11-10 21:10:35.674 | at java.base/sun.nio.ch.IOUtil.write(IOUtil.java:50) |  
  |   | 2023-11-10 21:10:35.674 | at java.base/sun.nio.ch.IOUtil.write(IOUtil.java:79) |  
  |   | 2023-11-10 21:10:35.674 | at java.base/sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:113) |  
  |   | 2023-11-10 21:10:35.674 | at java.base/sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:47) |  
  |   | 2023-11-10 21:10:35.674 | at java.base/sun.nio.ch.FileDispatcherImpl.write0(Native Method) |  
  |   | 2023-11-10 21:10:35.674 | Caused by: java.io.IOException: Broken pipe |  
  |   | 2023-11-10 21:10:35.674 | at java.base/java.lang.Thread.run(Thread.java:829) |  
  |   | 2023-11-10 21:10:35.674 | at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61) |  
  |   | 2023-11-10 21:10:35.674 | at org.apache.tomcat.util.threads.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:659) |  
  |   | 2023-11-10 21:10:35.674 | at org.apache.tomcat.util.threads.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1191) |  
  |   | 2023-11-10 21:10:35.674 | at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:52) |  
  |   | 2023-11-10 21:10:35.674 | at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1794) |  
  |   | 2023-11-10 21:10:35.674 | at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:928) |  
  |   | 2023-11-10 21:10:35.674 | at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:63) |  
  |   | 2023-11-10 21:10:35.674 | at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:390) |  
  |   | 2023-11-10 21:10:35.674 | at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:342) |  
  |   | 2023-11-10 21:10:35.674 | at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:74) |  
  |   | 2023-11-10 21:10:35.674 | at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:93) |  
  |   | 2023-11-10 21:10:35.674 | at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:130) |  
  |   | 2023-11-10 21:10:35.674 | at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:481) |  
  |   | 2023-11-10 21:10:35.674 | at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:90) |  
  |   | 2023-11-10 21:10:35.674 | at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:168) |  
  |   | 2023-11-10 21:10:35.674 | at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:153) |  
  |   | 2023-11-10 21:10:35.674 | at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:178) |  
  |   | 2023-11-10 21:10:35.674 | at com.github.ziplet.filter.compression.CompressingFilter.doFilter(CompressingFilter.java:263) |  
  |   | 2023-11-10 21:10:35.674 | at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:153) |  
  |   | 2023-11-10 21:10:35.674 | at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:178) |  
  |   | 2023-11-10 21:10:35.674 | at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117) |  
  |   | 2023-11-10 21:10:35.674 | at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:201) |  
  |   | 2023-11-10 21:10:35.674 | at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:153) |  
  |   | 2023-11-10 21:10:35.674 | at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:178) |  
  |   | 2023-11-10 21:10:35.674 | at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117) |  
  |   | 2023-11-10 21:10:35.674 | at com.ontotext.forest.core.request.RequestFilter.doFilterInternal(RequestFilter.java:47) |  
  |   | 2023-11-10 21:10:35.674 | at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:153) |  
  |   | 2023-11-10 21:10:35.674 | at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:178) |  
  |   | 2023-11-10 21:10:35.674 | at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117) |  
  |   | 2023-11-10 21:10:35.674 | at com.ontotext.forest.clusterproxy.ClusterLoadBalancerFilter.doFilterInternal(ClusterLoadBalancerFilter.java:25) |  
  |   | 2023-11-10 21:10:35.674 | at com.ontotext.forest.clusterproxy.ClusterFilterBean.doFilter(ClusterFilterBean.java:73) |  
  |   | 2023-11-10 21:10:35.674 | at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:153) |  
  |   | 2023-11-10 21:10:35.674 | at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:178) |  
  |   | 2023-11-10 21:10:35.674 | at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:267) |  
  |   | 2023-11-10 21:10:35.674 | at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:354) |  
  |   | 2023-11-10 21:10:35.674 | at com.ontotext.forest.security.AdminDelegatingFilterProxy.doFilter(AdminDelegatingFilterProxy.java:80) |  
  |   | 2023-11-10 21:10:35.674 | at org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:190) |  
  |   | 2023-11-10 21:10:35.674 | at org.springframework.security.web.FilterChainProxy.doFilterInternal(FilterChainProxy.java:225) |  
  |   | 2023-11-10 21:10:35.674 | at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:361) |  
  |   | 2023-11-10 21:10:35.674 | at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117) |  
  |   | 2023-11-10 21:10:35.674 | at com.ontotext.forest.security.audit.AuditLoggingFilter.doFilterInternal(AuditLoggingFilter.java:22) |  
  |   | 2023-11-10 21:10:35.674 | at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:361) |  
  |   | 2023-11-10 21:10:35.674 | at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117) |  
  |   | 2023-11-10 21:10:35.674 | at org.springframework.security.web.session.DisableEncodeUrlFilter.doFilterInternal(DisableEncodeUrlFilter.java:42) |  
  |   | 2023-11-10 21:10:35.674 | at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:361) |  
  |   | 2023-11-10 21:10:35.674 | at org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:87) |  
  |   | 2023-11-10 21:10:35.674 | at org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:117) |  
  |   | 2023-11-10 21:10:35.674 | at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:361) |  
  |   | 2023-11-10 21:10:35.674 | at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117) |  
  |   | 2023-11-10 21:10:35.674 | at org.springframework.security.web.context.request.async.WebAsyncManagerIntegrationFilter.doFilterInternal(WebAsyncManagerIntegrationFilter.java:62) |  
  |   | 2023-11-10 21:10:35.674 | at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:361) |  
  |   | 2023-11-10 21:10:35.674 | at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117) |  
  |   | 2023-11-10 21:10:35.674 | at org.springframework.security.web.header.HeaderWriterFilter.doFilterInternal(HeaderWriterFilter.java:75) |  
  |   | 2023-11-10 21:10:35.674 | at org.springframework.security.web.header.HeaderWriterFilter.doHeadersAfter(HeaderWriterFilter.java:90) |  
  |   | 2023-11-10 21:10:35.674 | at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:361) |  
  |   | 2023-11-10 21:10:35.674 | at org.springframework.web.filter.CompositeFilter.doFilter(CompositeFilter.java:74) |  
  |   | 2023-11-10 21:10:35.674 | at org.springframework.web.filter.CompositeFilter$VirtualFilterChain.doFilter(CompositeFilter.java:113) |  
  |   | 2023-11-10 21:10:35.674 | at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117) |  
  |   | 2023-11-10 21:10:35.674 | at org.springframework.security.web.authentication.www.BasicAuthenticationFilter.doFilterInternal(BasicAuthenticationFilter.java:168) |  
  |   | 2023-11-10 21:10:35.674 | at org.springframework.web.filter.CompositeFilter$VirtualFilterChain.doFilter(CompositeFilter.java:113) |  
  |   | 2023-11-10 21:10:35.674 | at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117) |  
  |   | 2023-11-10 21:10:35.674 | at com.ontotext.forest.security.stateless.GDBSignatureAuthenticationFilter.doFilterInternal(GDBSignatureAuthenticationFilter.java:33) |  
  |   | 2023-11-10 21:10:35.674 | at org.springframework.web.filter.CompositeFilter$VirtualFilterChain.doFilter(CompositeFilter.java:113) |  
  |   | 2023-11-10 21:10:35.674 | at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117) |  
  |   | 2023-11-10 21:10:35.674 | at com.ontotext.forest.security.stateless.GDBTokenAuthenticationFilter.doFilterInternal(GDBTokenAuthenticationFilter.java:41) |  
  |   | 2023-11-10 21:10:35.674 | at org.springframework.web.filter.CompositeFilter$VirtualFilterChain.doFilter(CompositeFilter.java:108) |  
  |   | 2023-11-10 21:10:35.674 | at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:361) |  
  |   | 2023-11-10 21:10:35.674 | at org.springframework.security.web.servletapi.SecurityContextHolderAwareRequestFilter.doFilter(SecurityContextHolderAwareRequestFilter.java:164) |  
  |   | 2023-11-10 21:10:35.674 | at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:361) |  
  |   | 2023-11-10 21:10:35.674 | at com.ontotext.forest.security.ExtendedAnonymousFilter.doFilter(ExtendedAnonymousFilter.java:43) |  
  |   | 2023-11-10 21:10:35.674 | at org.springframework.security.web.authentication.AnonymousAuthenticationFilter.doFilter(AnonymousAuthenticationFilter.java:100) |  
  |   | 2023-11-10 21:10:35.674 | at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:361) |  
  |   | 2023-11-10 21:10:35.674 | at org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:120) |  
  |   | 2023-11-10 21:10:35.674 | at org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:126) |  
  |   | 2023-11-10 21:10:35.674 | at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:361) |  
  |   | 2023-11-10 21:10:35.674 | at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.doFilter(FilterSecurityInterceptor.java:83) |  
  |   | 2023-11-10 21:10:35.674 | at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.invoke(FilterSecurityInterceptor.java:117) |  
  |   | 2023-11-10 21:10:35.674 | at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:361) |  
  |   | 2023-11-10 21:10:35.674 | at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117) |  
  |   | 2023-11-10 21:10:35.674 | at com.ontotext.forest.security.audit.AuditRequestCacheFilter.doFilterInternal(AuditRequestCacheFilter.java:26) |  
  |   | 2023-11-10 21:10:35.674 | at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:352) |  
  |   | 2023-11-10 21:10:35.674 | at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:153) |  
  |   | 2023-11-10 21:10:35.674 | at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:209) |  
  |   | 2023-11-10 21:10:35.674 | at javax.servlet.http.HttpServlet.service(HttpServlet.java:623) |  
  |   | 2023-11-10 21:10:35.674 | at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:883) |  
  |   | 2023-11-10 21:10:35.674 | at javax.servlet.http.HttpServlet.service(HttpServlet.java:529) |  
  |   | 2023-11-10 21:10:35.674 | at org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:898) |  
  |   | 2023-11-10 21:10:35.674 | at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:1006) |  
  |   | 2023-11-10 21:10:35.674 | at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:965) |  
  |   | 2023-11-10 21:10:35.674 | at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:1089) |  
  |   | 2023-11-10 21:10:35.674 | at org.springframework.web.servlet.DispatcherServlet.processDispatchResult(DispatcherServlet.java:1143) |  
  |   | 2023-11-10 21:10:35.674 | at org.springframework.web.servlet.DispatcherServlet.processHandlerException(DispatcherServlet.java:1332) |  
  |   | 2023-11-10 21:10:35.674 | at org.springframework.web.servlet.handler.AbstractHandlerExceptionResolver.resolveException(AbstractHandlerExceptionResolver.java:142) |  
  |   | 2023-11-10 21:10:35.674 | at org.springframework.web.servlet.handler.AbstractHandlerMethodExceptionResolver.doResolveException(AbstractHandlerMethodExceptionResolver.java:75) |  
  |   | 2023-11-10 21:10:35.674 | at org.springframework.web.servlet.mvc.method.annotation.ExceptionHandlerExceptionResolver.doResolveHandlerMethodException(ExceptionHandlerExceptionResolver.java:428) |  
  |   | 2023-11-10 21:10:35.674 | at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:135) |  
  |   | 2023-11-10 21:10:35.674 | at org.springframework.web.method.support.HandlerMethodReturnValueHandlerComposite.handleReturnValue(HandlerMethodReturnValueHandlerComposite.java:78) |  
  |   | 2023-11-10 21:10:35.674 | at org.springframework.web.servlet.mvc.method.annotation.HttpEntityMethodProcessor.handleReturnValue(HttpEntityMethodProcessor.java:219) |  
  |   | 2023-11-10 21:10:35.674 | at org.springframework.web.servlet.mvc.method.annotation.AbstractMessageConverterMethodProcessor.writeWithMessageConverters(AbstractMessageConverterMethodProcessor.java:290) |  
  |   | 2023-11-10 21:10:35.674 | at org.springframework.http.converter.AbstractGenericHttpMessageConverter.write(AbstractGenericHttpMessageConverter.java:104) |  
  |   | 2023-11-10 21:10:35.674 | at org.springframework.http.converter.json.AbstractJackson2HttpMessageConverter.writeInternal(AbstractJackson2HttpMessageConverter.java:456) |  
  |   | 2023-11-10 21:10:35.674 | at com.fasterxml.jackson.databind.ObjectWriter.writeValue(ObjectWriter.java:1063) |  
  |   | 2023-11-10 21:10:35.674 | at com.fasterxml.jackson.core.json.UTF8JsonGenerator.flush(UTF8JsonGenerator.java:1197) |  
  |   | 2023-11-10 21:10:35.674 | at com.fasterxml.jackson.core.json.UTF8JsonGenerator._flushBuffer(UTF8JsonGenerator.java:2203) |  
  |   | 2023-11-10 21:10:35.674 | at org.springframework.util.StreamUtils$NonClosingOutputStream.write(StreamUtils.java:287) |  
  |   | 2023-11-10 21:10:35.674 | at org.springframework.security.web.util.OnCommittedResponseWrapper$SaveContextServletOutputStream.write(OnCommittedResponseWrapper.java:638) |  
  |   | 2023-11-10 21:10:35.674 | at org.apache.catalina.connector.CoyoteOutputStream.write(CoyoteOutputStream.java:97) |  
  |   | 2023-11-10 21:10:35.674 | at org.apache.catalina.connector.OutputBuffer.write(OutputBuffer.java:361) |  
  |   | 2023-11-10 21:10:35.674 | at org.apache.catalina.connector.OutputBuffer.writeBytes(OutputBuffer.java:383) |  
  |   | 2023-11-10 21:10:35.674 | at org.apache.catalina.connector.OutputBuffer.append(OutputBuffer.java:680) |  
  |   | 2023-11-10 21:10:35.674 | at org.apache.catalina.connector.OutputBuffer.flushByteBuffer(OutputBuffer.java:777) |  
  |   | 2023-11-10 21:10:35.674 | at org.apache.catalina.connector.OutputBuffer.realWriteBytes(OutputBuffer.java:348) |  
  |   | 2023-11-10 21:10:35.674 | org.apache.catalina.connector.ClientAbortException: java.io.IOException: Broken pipe |  
  |   | 2023-11-10 21:10:35.673 | [WARN ] 2023-11-10 20:10:35,673 [http-nio-7200-exec-7 \| o.s.w.s.m.m.a.ExceptionHandlerExceptionResolver] Failure in @ExceptionHandler com.ontotext.forest.core.ForestErrorController#handleGeneric(Exception, HttpServletResponse) |  
  |   | 2023-11-10 21:10:33.346 | [INFO ] 2023-11-10 20:10:33,346 [repositories/registry \| c.o.g.s.TupleQueryResultView] Request for query -1496973462 is finished |  
  |   | 2023-11-10 21:10:33.326 | [INFO ] 2023-11-10 20:10:33,326 [repositories/registry \| c.o.g.s.TupleQueryResultView] Request for query 762259950 is finished

A crawler crash, from its logs (note different timestamp):

  |   | 2023-11-10 18:00:07.880 | HTTP error 500 for PUT http://graphdb/repositories/registry/rdf-graphs/service?graph=https%3A%2F%2Flod.uba.uva.nl%2FCinema-Context%2FCinema-Context |  
-- | -- | -- | -- | --
  |   | 2023-11-10 18:00:06.284 | {   "level": 30,   "time": 1699635606283,   "pid": 24,   "hostname": "registry-crawler-54f948cd59-2qh2j",   "msg": "Crawling registration URL https://lod.uba.uva.nl/Cinema-Context/Cinema-Context" } |  
  |   | 2023-11-10 18:00:06.091 | HTTP error 500 for PUT http://graphdb/repositories/registry/rdf-graphs/service?graph=https%3A%2F%2Flod.uba.uva.nl%2FCREATE%2FECARTICO |  
  |   | 2023-11-10 18:00:04.733 | {   "level": 30,   "time": 1699635604732,   "pid": 24,   "hostname": "registry-crawler-54f948cd59-2qh2j",   "msg": "Crawling registration URL https://lod.uba.uva.nl/CREATE/ECARTICO" } |  
  |   | 2023-11-10 18:00:04.431 | HTTP error 500 for PUT http://graphdb/repositories/registry/rdf-graphs/service?graph=https%3A%2F%2Flod.uba.uva.nl%2FCREATE%2FONSTAGE |  
  |   | 2023-11-10 18:00:00.037 | {   "level": 30,   "time": 1699635600036,   "pid": 24,   "hostname": "registry-crawler-54f948cd59-2qh2j",   "msg": "Crawling registration URL https://lod.uba.uva.nl/CREATE/ONSTAGE" }