asami #29

borkdude · 2020-11-12T15:35:26Z

I just tried out asami locally:

$ ./asami
(["Explorers"] ["Demolition Man"] ["Johnny Mnemonic"] ["Toy Story"])

Worked great, memory consumption during compilation is far more reasonable than datascript.
Hopefully once it gets durable storage it keeps working well like this.

The text was updated successfully, but these errors were encountered:

BrunoBonacci · 2020-11-13T11:23:47Z

Great, when you get chance could you please push an example?

quoll · 2021-06-05T16:07:47Z

Asami would be useful as a repl-driven app, so I tried this...

(ns asami.main
  (:require [clojure.main :as cm]
            [clojure.repl :as repl]
            [asami.core :as asami])
  (:gen-class))

(defn -main
  [& args]
  (require '[asami.core :as asami :refer [now instant instant? long-time create-database connect
                                          db as-of as-of-t since since-t graph as-connection
                                          transact entity q show-plan import-data export-data]])
  (apply cm/main args))

This works well with lein jar. Unfortunately, after building it gets into a loop of printing a repl prompt and an exception:

user=> Execution error (IllegalArgumentException) at asami.main/-main (main.clj:20).
No matching field found: read for class clojure.lang.LineNumberingPushbackReader

I thought that maybe the .read was being picked up via the Reader interface, or even through reflection, and hence the linker dropped the function... or something like that. But creating a LineNumberingPushbackReader around a StringReader and then reading from it works fine, though the repl code still complains.

Do you have any suggestions please?

borkdude · 2021-06-05T16:45:48Z

@quoll

With "after compilation" do you mean graalvm native-image compilation?

In this case don't require at runtime, rather pull the require to the top-level.

Can you give an example of how you would call your jar / native binary?

I think it's probably better to make the operations of your application explicit rather than going through clojure.main, but this depends on your example.

borkdude · 2021-06-05T16:59:37Z

@quoll Here is an example that demonstrates that asami works with native image compilation (well, this specific example):

borkdude/jayfu@7e6101c

I adapted the jayfu tutorial.

$ ./jayfu
(["Explorers"] ["Demolition Man"] ["Johnny Mnemonic"] ["Toy Story"])

borkdude · 2021-06-05T17:12:19Z

When using local storage ("asami:local://dbname") hits a reflection issue:

$ ./jayfu
Exception in thread "main" java.lang.IllegalArgumentException: No matching field found: getTime for class java.util.Date
	at clojure.lang.Reflector.getInstanceField(Reflector.java:397)
	at clojure.lang.Reflector.invokeNoArgInstanceMember(Reflector.java:440)
	at asami.internal$long_time.invokeStatic(internal.cljc:26)
	at asami.internal$long_time.invoke(internal.cljc:23)
	at asami.durable.store$fn__10772$new_db__10777$fn__10778.invoke(store.cljc:63)
	at asami.durable.store$fn__10772$new_db__10777.invoke(store.cljc:61)
	at asami.durable.store$fn__11150$create_database__11155$fn__11156.invoke(store.cljc:254)
	at asami.durable.store$fn__11150$create_database__11155.invoke(store.cljc:247)
	at asami.core$connection_for.invokeStatic(core.cljc:39)
	at asami.core$connection_for.invoke(core.cljc:32)
	at asami.core$fn__11800$create_database__11805$fn__11806.invoke(core.cljc:57)
	at asami.core$fn__11800$create_database__11805.invoke(core.cljc:51)
	at jayfu.main$_main.invokeStatic(main.clj:17)
	at jayfu.main$_main.doInvoke(main.clj:15)
	at clojure.lang.RestFn.invoke(RestFn.java:397)
	at clojure.lang.AFn.applyToHelper(AFn.java:152)
	at clojure.lang.RestFn.applyTo(RestFn.java:132)
	at jayfu.main.main(Unknown Source)

borkdude · 2021-06-05T17:20:01Z

The above issue should be fixed with threatgrid/asami#156

borkdude · 2021-06-05T17:22:08Z

New issue:

$ ./jayfu
Exception in thread "main" java.util.concurrent.ExecutionException: java.lang.IllegalArgumentException: No matching field found: close for class sun.nio.ch.FileLockImpl
	at java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:395)
	at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1999)
	at clojure.core$deref_future.invokeStatic(core.clj:2304)
	at clojure.core$deref.invokeStatic(core.clj:2324)
	at clojure.core$deref.invoke(core.clj:2310)
	at jayfu.main$_main.invokeStatic(main.clj:16)
	at jayfu.main$_main.doInvoke(main.clj:15)
	at clojure.lang.RestFn.invoke(RestFn.java:397)
	at clojure.lang.AFn.applyToHelper(AFn.java:152)
	at clojure.lang.RestFn.applyTo(RestFn.java:132)
	at jayfu.main.main(Unknown Source)
Caused by: java.lang.IllegalArgumentException: No matching field found: close for class sun.nio.ch.FileLockImpl
	at clojure.lang.Reflector.getInstanceField(Reflector.java:397)
	at clojure.lang.Reflector.invokeNoArgInstanceMember(Reflector.java:440)
	at asami.durable.store$fn__11015$transact_update_STAR___11020$fn__11024.invoke(store.cljc:187)
	at asami.durable.store$fn__11015$transact_update_STAR___11020.invoke(store.cljc:178)
	at asami.durable.store$fn__11050$transact_data_STAR___11059$fn__11064.invoke(store.cljc:217)
	at asami.durable.store$fn__11050$transact_data_STAR___11059.invoke(store.cljc:208)
	at asami.durable.store.DurableConnection.transact_data(store.cljc:232)
	at asami.core$fn__11938$transact__11943$fn__11950$fn__11958.invoke(core.cljc:187)
	at asami.core$fn__11938$transact__11943$fn$reify__11977.get(core.cljc:200)
	at java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1700)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
	at java.lang.Thread.run(Thread.java:829)
	at com.oracle.svm.core.thread.JavaThreads.threadStartRoutine(JavaThreads.java:553)
	at com.oracle.svm.core.posix.thread.PosixJavaThreads.pthreadStartRoutine(PosixJavaThreads.java:192)

borkdude · 2021-06-05T17:55:41Z

After fixing those reflection issues in asami and adding an additional reflection config for java.lang.Boolean (@quoll knows why, this was needed because booleans were serialized using some reflection I think), it works with local storage!

$ ./jayfu
(["Explorers"] ["Demolition Man"] ["Johnny Mnemonic"] ["Toy Story"])

See: https://github.com/borkdude/jayfu/tree/asami

borkdude · 2021-06-05T18:58:26Z

After multiple runs, a new reflection issue:

$ ./jayfu
Exception in thread "main" java.util.concurrent.ExecutionException: java.lang.IllegalArgumentException: No matching field found: id for class asami.graph.InternalNode
	at java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:395)
	at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1999)
	at clojure.core$deref_future.invokeStatic(core.clj:2304)
	at clojure.core$deref.invokeStatic(core.clj:2324)
	at clojure.core$deref.invoke(core.clj:2310)
	at jayfu.main$_main.invokeStatic(main.clj:16)
	at jayfu.main$_main.doInvoke(main.clj:15)
	at clojure.lang.RestFn.invoke(RestFn.java:397)
	at clojure.lang.AFn.applyToHelper(AFn.java:152)
	at clojure.lang.RestFn.applyTo(RestFn.java:132)
	at jayfu.main.main(Unknown Source)
Caused by: java.lang.IllegalArgumentException: No matching field found: id for class asami.graph.InternalNode
	at clojure.lang.Reflector.getInstanceField(Reflector.java:397)
	at clojure.lang.Reflector.invokeNoArgInstanceMember(Reflector.java:440)
	at asami.graph.InternalNode.equals(graph.cljc:43)
	at clojure.lang.Util.equiv(Util.java:33)
	at clojure.lang.PersistentHashMap$BitmapIndexedNode.find(PersistentHashMap.java:788)
	at clojure.lang.PersistentHashMap.containsKey(PersistentHashMap.java:126)
	at clojure.lang.APersistentSet.contains(APersistentSet.java:34)
	at clojure.lang.PersistentHashSet.cons(PersistentHashSet.java:97)
	at clojure.lang.PersistentHashSet.cons(PersistentHashSet.java:17)
	at clojure.lang.RT.conj(RT.java:677)
	at clojure.core$conj__5407.invokeStatic(core.clj:87)
	at clojure.core$conj__5407.invoke(core.clj:84)
	at zuko.entity.writer$fn__11355$map__GT_triples__11360$fn__11361$fn__11369.invoke(writer.cljc:167)
	at zuko.entity.writer$fn__11355$map__GT_triples__11360$fn__11361.invoke(writer.cljc:166)
	at zuko.entity.writer$fn__11355$map__GT_triples__11360.invoke(writer.cljc:153)
	at zuko.entity.writer$fn__11409$ident_map__GT_triples__11422$fn__11427.invoke(writer.cljc:197)
	at zuko.entity.writer$fn__11409$ident_map__GT_triples__11422.invoke(writer.cljc:181)
	at zuko.entity.writer$fn__11409$ident_map__GT_triples__11422$fn__11425.invoke(writer.cljc:194)
	at zuko.entity.writer$fn__11409$ident_map__GT_triples__11422.invoke(writer.cljc:181)
	at asami.entities$fn__11638$entity_triples__11643$fn__11647.invoke(entities.cljc:111)
	at asami.entities$fn__11638$entity_triples__11643.invoke(entities.cljc:44)
	at asami.entities$fn__11714$build_triples__11719$fn__11720$add_triples__11730.invoke(entities.cljc:147)
	at clojure.lang.PersistentVector.reduce(PersistentVector.java:343)
	at clojure.core$reduce.invokeStatic(core.clj:6829)
	at clojure.core$reduce.invoke(core.clj:6812)
	at asami.entities$fn__11714$build_triples__11719$fn__11720.invoke(entities.cljc:163)
	at asami.entities$fn__11714$build_triples__11719.invoke(entities.cljc:135)
	at asami.core$fn__11938$transact__11943$fn__11950$fn__11958$fn__11973.invoke(core.cljc:191)
	at asami.durable.store$fn__11050$transact_data_STAR___11059$fn__11064$fn__11065.invoke(store.cljc:219)
	at asami.durable.store$fn__11015$transact_update_STAR___11020$fn__11024.invoke(store.cljc:193)
	at asami.durable.store$fn__11015$transact_update_STAR___11020.invoke(store.cljc:178)
	at asami.durable.store$fn__11050$transact_data_STAR___11059$fn__11064.invoke(store.cljc:217)
	at asami.durable.store$fn__11050$transact_data_STAR___11059.invoke(store.cljc:208)
	at asami.durable.store.DurableConnection.transact_data(store.cljc:232)
	at asami.core$fn__11938$transact__11943$fn__11950$fn__11958.invoke(core.cljc:187)
	at asami.core$fn__11938$transact__11943$fn$reify__11977.get(core.cljc:200)
	at java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1700)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
	at java.lang.Thread.run(Thread.java:829)
	at com.oracle.svm.core.thread.JavaThreads.threadStartRoutine(JavaThreads.java:553)
	at com.oracle.svm.core.posix.thread.PosixJavaThreads.pthreadStartRoutine(PosixJavaThreads.java:192)

borkdude · 2021-06-05T21:06:33Z

The binary works again with PR threatgrid/asami#157 (which supersedes PR 156).

Note that there are some remaining reflections in zuko/logging.clj but this isn't hit in the above example.

$ clj -M -e "(set! *warn-on-reflection* true)" -e "(require 'asami.core)"
true
Reflection warning, zuko/logging.clj:14:36 - reference to field getClassName can't be resolved.
Reflection warning, zuko/logging.clj:13:30 - reference to field getStackTrace can't be resolved.
Reflection warning, zuko/logging.clj:78:5 - call to method append on java.io.Writer can't be resolved (argument types: unknown).
planner.cljc:107 recur arg for primitive local: mcount is not matching primitive, had: Object, needed: long
Auto-boxing loop arg: mcount
Reflection warning, asami/durable/block/file/voodoo.clj:18:14 - reference to field clean on java.lang.Object can't be resolved.

quoll · 2021-06-05T23:54:46Z

@quoll

With "after compilation" do you mean graalvm native-image compilation?

In this case don't require at runtime, rather pull the require to the top-level.

Can you give an example of how you would call your jar / native binary?

I think it's probably better to make the operations of your application explicit rather than going through clojure.main, but this depends on your example.

We discussed this, and it’s embarrassingly obvious why a plain repl could never work for this.

But I’ll just explain the require: it was a hack to make the namespace alias and :refer symbols available to the repl. This does not happen when done at the top level.

BrunoBonacci added the WIP label Jan 23, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

asami #29

asami #29

borkdude commented Nov 12, 2020

BrunoBonacci commented Nov 13, 2020

quoll commented Jun 5, 2021

borkdude commented Jun 5, 2021 •

edited

Loading

borkdude commented Jun 5, 2021 •

edited

Loading

borkdude commented Jun 5, 2021

borkdude commented Jun 5, 2021

borkdude commented Jun 5, 2021

borkdude commented Jun 5, 2021

borkdude commented Jun 5, 2021

borkdude commented Jun 5, 2021

quoll commented Jun 5, 2021

asami #29

asami #29

Comments

borkdude commented Nov 12, 2020

BrunoBonacci commented Nov 13, 2020

quoll commented Jun 5, 2021

borkdude commented Jun 5, 2021 • edited Loading

borkdude commented Jun 5, 2021 • edited Loading

borkdude commented Jun 5, 2021

borkdude commented Jun 5, 2021

borkdude commented Jun 5, 2021

borkdude commented Jun 5, 2021

borkdude commented Jun 5, 2021

borkdude commented Jun 5, 2021

quoll commented Jun 5, 2021

borkdude commented Jun 5, 2021 •

edited

Loading

borkdude commented Jun 5, 2021 •

edited

Loading