Error with database reading

[fab6]

Hi,

I am trying to use your script as I planning to got to papis from zotero as well.
The installation seemed fine with my julia 1.11.1 installation on macos.

I set
alias ZoteroToPapis="julia --project=@ZoteroToPapis -e 'using ZoteroToPapis; exit(ZoteroToPapis.main(ARGS))' --"

Though I am having trouble with reading any of the databases, e.g. here is the output for better-bibtex:

➜ ZoteroToPapis --better-bibtex-db ~/better-bibtex.sqlite
ERROR: SQLite.SQLiteException("unable to open database file")
Stacktrace:
 [1] sqliteerror(args::Ptr{SQLite.C.sqlite3})
   @ SQLite ~/.julia/packages/SQLite/UqCGE/src/SQLite.jl:34
 [2] SQLite.DB(f::String)
   @ SQLite ~/.julia/packages/SQLite/UqCGE/src/SQLite.jl:71
 [3] main(args::Vector{String})
   @ ZoteroToPapis ~/.julia/packages/ZoteroToPapis/X6AMO/src/main.jl:63
 [4] top-level scope
   @ none:1

Similar error occurs with the zotero-db option.

Do you have a hint, what I am doing wrong?

And what are the differences of zotero-db, better-bibtex-db and zotero-storage?
I had zotfile running in combination with zotero and would like to keep also the folder structure and main naming.

Thank you in advance!
Fab

[Klafyvel]

Hi! for the SQLite error, are you able to open the database manually (for example from the Julia REPL)?
Here are the meanings of the options:

• zotero-db: main zotero database
• zotero-storage: path to the directory where all your pdf are stored
• better-bibtex-db: better-bibtex uses its own database to store the citation keys. In my case it was in ~/Zotero/better-bibtex.sqlite.

I don't know zotfile, if you want a special folder structure, you'll have to edit the code a bit (right now it's creating a folder per publication year). The output path is handled here and defined there.

Hope this helps!

[fab6]

Hi,
thank you, I will check, for my test zotero was not running.

With the meaning of the different options, I was curious about, for which purpose I need to define which option. So, e.g., is it enough to read the better-bibtex-db and all information is available in the info.yaml in papis? Or are there some settings missing. Or would it be more comprehensive to use the zotero-db option?

With the zotfile addon, I moved the pdf locally out of the zotero storage folder into a complete folder structure which corresponds to the folder and subfolder organinzation in the zotero gui.

So if the file path of the current actual location of each pdf is read with some of the options, I might be straight forwarded to add a folder structure to the line you pointed out:
filepath = papis_directory(papis_root, entry.year, entry.citationkey)
to something like
filepath = papis_directory(papis_root, entry.filepath, entry.year, entry.citationkey)

But I am not sure, if the filepath is read with all options; e.g. I have in my bibtex from better-bibtex a field for each item:

  file = {<PATH_ZOTERO_OUTSIDE_STORAGE>/06_ML_General/02_ROM/Zuo_Chen_2009_Real-time_or_faster-than-real-time_simulation_of_airflow_in_buildings.pdf}

Is this available at the moment?

Thank you!

[Klafyvel]

Hi!

I think you need all the databases because most of the data relative to entries is only available in the main database. See here for the data collection, db is the main zotero database, and dbbb is the better-bibtex database.

For the zotfile extension, you'll need to research how the file entry in the main database is affected. For now the path is retrieved here, so you see it's assumed to be relative to the zotero storage. To do your research you can simply open the sqlite database, for example using the SQLite manager extension in Firefox.

[fab6]

ok, now it works like intended, I needed to set all options, better-bibtex-db, zotero-db and zotero-storage... cool

[fab6]

I assume that every entry key from zotero is transfered (only with a change of the path). Or do you know that certain keys are negleted? Maybe the notes?

[fab6]

I just check for one

author: Oliver Fuß
author_list:
- Oliver Fuß
date: 2004-00-00 2004
files:
- Fuß_2004_Ermittlung_und_Berechnung_der_Sauerstoffgrenzkonzentration_von_brennbaren_Gasen.pdf
institution: Universität Duisburg - Essen
language: de
library: Zotero
location: Fachbereich Chemie
papis_id: d6623fc898b3c3cc24149d172256c403
ref: fussErmittlungUndBerechnung2004
tags:
- 29_Explosion
- CLEAN
- 02_FIRE_THEORY
title: Ermittlung und Berechnung der Sauerstoffgrenzkonzentration von brennbaren Gasen
type: thesis
zotero_extra_fields:
  database_id: '26650'
  database_key: QZJFKRDX
zotero_import_date: 2025-01-17 17:08:27.027000

The corresponding entry in the better bibtex bib file

@thesis{fussErmittlungUndBerechnung2004,
	type = {phdthesis},
	title = {Ermittlung und Berechnung der Sauerstoffgrenzkonzentration von brennbaren Gasen},
	author = {Fuß, Oliver},
	date = {2004},
	institution = {Universität Duisburg - Essen},
	location = {Fachbereich Chemie},
	langid = {ngerman},
	keywords = {CLEAN},
	file = {<path>/ZOTERO_zotfile/02_FIRE_THEORY/29_Explosion/Fuß_2004_Ermittlung_und_Berechnung_der_Sauerstoffgrenzkonzentration_von_brennbaren_Gasen.pdf}
}

Here the thesis type is not recognized and translated.

The files is adjusted. Here the aim is now to use the subfolder organization from the better bibtex file to organize the pdfs in the same subfolders for papis.

[Klafyvel]

ok, now it works like intended, I needed to set all options, better-bibtex-db, zotero-db and zotero-storage... cool

Nice, would you like to add a note in the README to explain that both are needed? maybe it could help other people!

I assume that every entry key from zotero is transfered (only with a change of the path). Or do you know that certain keys are negleted? Maybe the notes?

You can actually check what gets exported here, I think I explicitely remove the notes because I never really used them, and I guess they would need to be treated in a different fashion than other items. If you decide to add a pass to add the notes to the export, please do share your contribution!

I just check for one

author: Oliver Fuß
author_list:
- Oliver Fuß
date: 2004-00-00 2004
files:
- Fuß_2004_Ermittlung_und_Berechnung_der_Sauerstoffgrenzkonzentration_von_brennbaren_Gasen.pdf
institution: Universität Duisburg - Essen
language: de
library: Zotero
location: Fachbereich Chemie
papis_id: d6623fc898b3c3cc24149d172256c403
ref: fussErmittlungUndBerechnung2004
tags:
- 29_Explosion
- CLEAN
- 02_FIRE_THEORY
title: Ermittlung und Berechnung der Sauerstoffgrenzkonzentration von brennbaren Gasen
type: thesis
zotero_extra_fields:
  database_id: '26650'
  database_key: QZJFKRDX
zotero_import_date: 2025-01-17 17:08:27.027000

The corresponding entry in the better bibtex bib file

@thesis{fussErmittlungUndBerechnung2004,
	type = {phdthesis},
	title = {Ermittlung und Berechnung der Sauerstoffgrenzkonzentration von brennbaren Gasen},
	author = {Fuß, Oliver},
	date = {2004},
	institution = {Universität Duisburg - Essen},
	location = {Fachbereich Chemie},
	langid = {ngerman},
	keywords = {CLEAN},
	file = {<path>/ZOTERO_zotfile/02_FIRE_THEORY/29_Explosion/Fuß_2004_Ermittlung_und_Berechnung_der_Sauerstoffgrenzkonzentration_von_brennbaren_Gasen.pdf}
}

Here the thesis type is not recognized and translated.

The files is adjusted. Here the aim is now to use the subfolder organization from the better bibtex file to organize the pdfs in the same subfolders for papis.

if some type translation is missing, I think you need to add them here

[fab6]

Hi,

it works almost... I adjusted quite a few minor things to be able to keep the folder/collection structure.
For this I added a few basic commands to replace some characters, splitting folder and/or passing the collection information to the path or adding some random number to duplicates.

What I am missing now is, that I wanted to add the unicode package to normalize some strings, maybe you have an advice how this can be done in combination with a julia environment and the reinstallation from the julia repository. I am lacking here basic julia experience and I could not figure it out yet...

[Klafyvel]

Hi, I don't know what the unicode package is. Julia's standard library has things to deal with unicode. Otherwise, if you are referring to a Python library, you can call it directly from Julia. I can assist if you provide more details.

[fab6]

Sorry, I was not clear, there is a Unicode package which adjust string, e.g.,

... but I am not sure how I get the package installed, I tried the usual ways via pkg add and tried to resolve it and instantiate it...

when your are working on your code, how do you add additional packages to the manisfest and/or project.tomls? Somehow I never got it working.

[Klafyvel]

Ok sorry I got confused!
Unicode is part of the standard library, and it came with your Julia installation. For other packages (e.g. not standard libraries) you usually manage them using Pkg. See here.

[fab6]

yes, sorry, I was looking at this, but did not understand the details yet... anyway I am almost there. Thank you for your help!