Editor unusable with large file upload

[nschneid]

I uploaded a large .conllu file (1400 sentences), and while it loaded successfully and I can browse the trees, the editor is too laggy for me to make edits—e.g. if I try to change a deprel it gets stuck. If I just upload the first few sentences from the file, everything works fine.

[keggsmurph21]

Yeah, this is a bit of an architectural problem. Currently, we just serialize the entire treebank and try to load it on the client here:

ud-annotatrix/client/server.js

Line 122 in f6865f1

this.app.load(serial);

via

ud-annotatrix/server/routes.js

Lines 125 to 131 in f6865f1

	app.get("/load", get_treebank, (req, res) => {
	CorpusDB(req.treebank).load((err, data) => {
	if (err)
	res.json({error: err.message});
	res.json(data);
	});

This is a bit sad, since we really only need to load the sentences one at a time, which we already sorta-support (but just don't use?):

ud-annotatrix/server/models/corpus.js

Line 33 in f6865f1

getSentence(id, next) {

[keggsmurph21]

FWIW, I think we could improve here, but it's a bit tough to make these large changes with confidence. I think maybe if we added some static type information it would be more feasible 😄

[Zensho]

I am currently investigating the issue. The problem might be caused by too much load on the working memory or the local storage, will have to look further to see which.

[jonorthwash]

if I try to change a deprel it gets stuck

@nschneid, when you say "gets stuck", do you mean that Annotatrix freezes, or just a UI problem like the label staying where it is after updating the tree?

[nschneid]

As I recall it refused to change the tree or allow further edits.

[Zensho]

The problem seems to be that localStorage in general allows no more than 5200000 characters in total, and with it only taking in strings any large corpus simply exceeds the quota. I will try to use IndexedDB to persist the data as an alternative.

[Zensho]

I think maybe if we added some static type information it would be more feasible 😄

Hi Kevin, I am currently working on this issue, and I am wondering what does the above sentence mean, and how would it help in achieving reading only one sentence at a time?