Known problems in Search Results

[julianharty]

This issue is to help track various reported problems related to searching. We already have some open issues related to searching e.g. #311

The main reason for gathering them here (at least for me) is to keep them in focus when we try to improve the app and the tests for the app. We have the mini-hackathon later this week http://wiki.kiwix.org/wiki/UkHackathon2017 and it'd be great to address at least some of these issues during the hackathon.

[julianharty]

Known, existing bugs
#311 Search crashes Kiwix...
#255 Improve Content Search functionality (see also #307 which claims to fix #255)
#240 Crash after selecting "Get Content". Hopefully this has been addressed by now, worth testing extensively.
#236 Can't open ZIM files on Android. While this isn't directly a Search issue, it includes search aspects. Adding more comprehensive logging may help diagnose the problem(s).

There are 19 open issues currently that mention search https://github.com/kiwix/kiwix-android/issues?utf8=%E2%9C%93&q=is%3Aissue+is%3Aopen+search

[julianharty]

It's also worth considering common user-centric concerns such as being able to search for abbreviations. Perhaps we can improve the communication with the user e.g. to inform them some functionality won't work if the ZIM file doesn't have a full-text index present on the device. See https://github.com/kiwix/kiwix-android/issues/84 as an example of the problem from a user's perspective.

[julianharty]

I've skimmed through about 2 months of reviews for the Android apps. WikiMed has received at leasts 9 reviews related to poor searching (there may be some in other languages I've not spotted). See #316 for the details.

[julianharty]

Building on some of the search issues reported in #311 here are some interim observations from testing searches in kiwix-android (rather than any of the custom apps). The ZIM file is Wikipedia Maths, 15,000 articles, English 604MB. They're not definitive - testing search effectively is likely to be a major undertaking - more indicative, and perhaps useful :)

As a user I'd expect search to return the term I searched for in the search results. When I search for function or algebra (both are popular words in this ZIM file) I can see the exact word while I'm still typing the relevant letters e.g. f u n c t i o however the exact word disappears from the search results before I complete typing the term. I can accept and understand why it might not be the most relevant result, but don't expect it to be missing entirely from the results.

Sometimes searching for a partial word e.g. func (for instance) shows no results. This seems to happen more often when I've deleted letters (where I often see blank results) and sometimes when I delete and then add a letter back. e.g. after doing something similar to:

1. type funct
2. delete the last 2 letters leaving fun
3. add the letter c

I've seen the search results return as a flash on the screen before being replaced by blank results, this might be a timing issue, with delayed responses to what's being typed on screen.

That's it for now, I want to add some logging to the app so I've a better idea of what the search is doing and how it's behaving. More to come...

[julianharty]

Well this has been interesting. There seems to be a pattern that affects when search results are shown. I noticed that odd numbers of space characters showed search results while even numbers didn't. I don't think the pattern's quite this predictable or consistent across all searches, but it was sufficiently repeatable I was able to record it (using the opensource AndroidTool https://github.com/mortenjust/androidtool-mac) A zip of the recording is attached.

coreprimeveltexxLMY48Bjulianharty12162017110851.mp4.zip

I've added some logging statements, in particular to SearchActivity.java as follows:

      @Override
      public boolean onQueryTextChange(String s) {
        Log.i("kiwix-search-activity", "onQueryTextChanged to: " + s);

This indicates that each addition and removal of characters calls this method. I'd next like to investigate how to log the interactions with the JNI Search code. Thoughts and suggestions are welcome @mgautierfr @mhutti1

[julianharty]

I've created the start of a Wikipage on Search and Search Results. Contributions and suggestions are welcome on ways to improve our definition, understanding, and testing of 'search'.

[okkebal]

"user-centric concerns such as being able to search for abbreviations....."

Wow... users are complaining to no end about not being able to find anything! Its not just IBM that returns no results in wikipedia NL. Try to find Anything in WikiSource EN, without typing in the EXACT title. I challenge you!

What would be great if there was any way to get a Zim file that contains an index. I would not know how or where to get these fabled Zim files.

[kelson42]

@okkebal All the recent Wikimed EN ZIM files have a fulltext index in them. We decided to create all our new ZIM files with FT index in them in January... but discovered quite an annoying surprise openzim/libzim#110. This "breaks" any ZIM with an FT index > 4GB. This should be fixed before end of February and then we will generate all new ZIM with FT index. That said quite interesting about your detailed analysis of search feature on Wikimed EN Zim files.

[okkebal]

The new nighly sais: "dumping memory app will freeze .... brr..." all the time (no crash occurs, it just waits and then continues). And can't do any testing like this. Hoprrible: can I turn that off somehow? I did try the '2017-10-10' medical wiki, but Full text search does not work. 0 results when I type 'Exhaustion', although that word appears on many pages. I will try a later version soon. When downloading the new version I see a "nodet" version. What does "nodet" mean??

PS: When I turn on the 'multi-file search' I do get many results, but when I click any of them nothing happens.

[kelson42]

@okkebal A memory leak detection tools has been included to dev version. Nightly is a dev tool. For that reason I do not think we are going to remove it from there. That said, now that we can see them, we will remove most of them ASAP.

[JessRiedel]

@okkebal:

When downloading the new version I see a "nodet" version. What does "nodet" mean??

It means it keeps only the first paragraph from each article:

"limit to lead paragraph (_nodet format) and keep only one image if possible (a few articles have more than one illustration in the intro)"

kiwix/kiwix-android-custom#13

[okkebal]

This issue is opened more than a year ago and Search just keeps getting worse. I personally use Aard to search: then, when I know the exact title of an article i type that in to kiwix and usually I can then find it on Kiwix. But that stopped working as well. Even if I type in the exact article name still nothing comes up for a lot of articles. Please reopen https://github.com/kiwix/kiwix-android/issues/84 since it is not fixed. None of the examples given in the bug discussion work on the latest Kiwix.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be now be reviewed manually. Thank you for your contributions.

[kelson42]

I close that ticket as many of the things are quite old and the situation as been seriously improved with Kiwix Android 3.0 (and newest ZIM files).