MXD incorrectly identifies some books as missing from calibre #32
Comments
|
I'm not understanding the issue. Please be more specific which books are missing. |
|
The books missing are shown in the MXD window with a white background, apart from the Cassandra Clare books. ie: Black Wings, Blood Magic, Bloody Skies, Picked at the Peak, The Preachers's Son, When Extended Family is Discovered and, When I Saw You. Some of these are shown in the attached screenshot which has the current calibre library showing in the background. eg: Black Starr Cross, Blood Magic and, Bloody Skies. The others do exist in calibre but are not visible in the current screenshot. The MXD window has nearly two thousand books with a white background and my assumption is that this is probably correct as I synchronize Marvin from two calibre libraries. I only spotted these, because I sorted by reading percentage to find the most recently read books. Consequently I can't say if the books noted above are the only ones shown in error. |
|
Hi Wyndham, Happy new year. If the MXD window shows a book in white, that means that the plugin did not find a matching book in your current calibre library. In your screenshot, the most likely explanation for the books you're expecting to shown as matched is that the author doesn't match between calibre and Marvin, even though the title does match. For a match to be detected, both the author and title must match exactly. Additionally, the content of the book needs to match. If you have a book in Marvin, and a different version of the same book in calibre, they won't match. This can often happen if you add books from different sources. The easiest way to detect this is to simply open the book in calibre's viewer, then open the same book in Marvin. Most likely you will see that they are different versions of the same book. It's also possible that there's another problem with the plugin, but these are the most likely explanations. Please close this issue if that's what you find, or provide additional metadata details about the books shown incorrectly. |
|
I can confirm they have the same author. All the books in Marvin have been sourced from Calibre only, so it seems unlikely to me that the books are different. If I update a book and then convert again, I usually download straight away to the mobile devices. In this case, I am fairly confident about these books as they are all recent reads. Simple visual checks on the books show the following: · Black Wings appears identical · Blood Magic appears identical · Bloody Skies appears identical As a final test I sent these three books from Calibre to Marvin and re-ran MXD, and they no longer show as missing from Calibre. This leaves me a little frustrated. I have to assume that it is either not possible, or easy, programmatically to distinguish between a book not existing (under the current rules, because the content is different in some, possibly small, respect) and to identify the book as possibly existing and the same book, but with differences. This could include slight differences in the title or author, such as Midnight Heir and The Midnight Heir, J.D. Robb and JD Robb, or Arthur Clarke and Arthur C. Clarke, etc. There remain a few other books which are not being correctly identified (given the aforementioned provisions), so I can do further testing if you like. |
|
I have the same problem. I figured it might be a glitch from multiple versions of calibre/the plugins/Marvin, so I deleted every book from Marvin, Polished all the books to verify calibre and book metadata was identical and finally transferred them from Calibre. Around half my Library of 500 books is 'not in calibre' even though I JUST copied it directly from calibre. After some more experimenting, I also discovered when I select a single white colored book - in the Marvin Library subwindow of calibre, I don't even have the correct options if the books was missing (I only have View Metadata, and Delete Book). However, if I select multiple books, I have the options to Delete Book, Export Metadata or Import Metadata. Even though the Library window says they are missing from calibre, I am not allowed to import the book to calibre. Choosing 'Export Metadata from calibre' causes every selected white book to turn green. Once I close the Marvin Library and reopen it, they are white again. This only occurs with these mistakenly white books - I altered metadata in calibre to make a few books yellow and they updated properly and stayed green. Hope that helps pinpoint the problem. |
|
If you can give me a specific set of steps that lead to the failure, along with a debug log, that would help pinpoint the problem. |
|
Here are a few more steps that would help in pinpointing the problem.
How is the book displayed in MXD? If it's other than green, please post the debug log here in a new comment |
|
Ok, here you go. Did as you said. Book is still white. Focused on a single book - Harry Crow by robst. I use Marvin for Fanfics mostly. Sorry the debug log is so long, but I still have a bunch of fanfic covers to make. Also, I use the latest version of all apps/plugins and run calibre on OSX 10.9.1 calibre 1.19 isfrozen: True is64bit: True |
|
@uten, please send a copy of this book (Harry Crow) to me. Is it generally true that the books you're having problems with are fan fiction? |
|
Book sent. Also, while the majority is fanfic, that is only because most of my books are fanfic.. But there are also novels, and non-fiction with the same problem. And there is no common link. Some of the fanfic were created by FLAG, calibre's fanfic downloader, various other websites, and even some I've converted. Same with the Novels and Non-Fiction, they all come from various different places, including smashwords. |
|
@uten, thanks for sending the book. I added it to my test library, then added it to Marvin using calibre. Upon opening MXD it properly displayed as green when Show match status is enabled. I examined the book to be sure that there was nothing unusual about its internal metadata. In studying your debug log more closely, it appears that you have a total of 1400 books in your calibre library (reported just after DEBUG 8.2). At DEBUG 67.3, it reports that you have 571 EPUBS in your library, and at DEBUG 68.1, it reports that you have 533 books installed in Marvin. One possible explanation is that you have some filtering applied to your calibre library so that only certain books are displayed at the time you launch MXD. MXD only evaluates books currently displayed in your calibre window when showing matches. This was an optimization added for users with very large libraries. Can you confirm that before launching MXD, Harry Crow is visible in the main calibre window? Are you using a virtual library, or is there any filter that would prevent the Harry Crow book from being listed in the main calibre window at the time MXD is launched? |
|
I do use virtual Libraries. Specifically, the one I was using at the time is entitled "Books for Marvin" and contains all epub books. I did verify that the book was showing. It also shows a tick in calibre. The remaining epubs that I have still to transfer from calibre need to have metadata entered, or something similar. That was the first thing I checked. |
|
Also, while the book shows as white, view metadata shows the Marvin metadata and the calibre metadata. The only difference is it doesn't have the option underneath each window to transfer metadata. But selecting multiple white files does allow me to export metadata from calibre, turning the books green. The few yellow books allow me to export metadata from calibre, turning the books green. So, since the metadata and status change is obviously not being saved on my ipad, is it possible that the data saying the book is in calibre isn't being written either? Is that stored on the ipad? |
|
Please do an experiment to update the metadata on the iPad from calibre for a given book, then opening the book in Marvin to see if the metadata was actually updated from the calibre values, regardless of the colors shown by the plugin. Instead of reporting "the metadata shows up as the old bad stuff again", it would be more helpful if you could report the actual before/after metadata. Also, can you do a manual inspection of the books in Marvin to see if there are duplicates of the problem books? |
|
@uten, please try this as well. In calibre, open Preferences|Advanced|Miscellaneous, then click Open calibre configuration directory. Exit calibre. In that directory, open the plugins folder, then locate the file Marvin XD.json. Open that file with a text editor, and find the line "show_match_colors": true, Copy the entire line, and paste a copy below it, making sure that the indentation is exactly the same for the pasted copy. Edit that line to read "development_mode": true, Save and close the file. This will add some additional diagnostic information to the debug log. In that same directory, open iOS reader applications.json Find the line "debug_plugin": true, Change true to false Save and close the file. That will turn off debugging for the iOSRA plugin. Do the same sequence you did before, deleting the book in Marvin, then starting Calibre in debug mode, adding the book, starting MXD. After the book shows white, close calibre, post the debug log. |
|
I chose Harry Crow to alter the metadata. I removed all the tags. Which turned it yellow. Updated the metadata which turned it white. The metadata did update however and it has stayed updated. Also picked a green colored book. Worked correctly. Green > Yellow > Green.. Weird thing number one. With yellow books. Just updating the metadata causes the following. They turn green until you disconnect, then next time you connect, they are yellow. However, one of them with this problem had an author written "Brighton, Patrick". I switched it to "Patrick Brighton" in calibre, then updated the metadata to Marvin. The book is now green. Since that worked, I altered the author of Harry Crow and updated. Still white. One thing I did notice. Just looking at a books metadata in calibre MIGHT turn the book yellow. I chose the "Edit Metadata Individually" and then selecting next (or cancel). Some books turned yellow from that, and some didn't. After choosing to update the metadata to Marvin, they would switch back to either white or green. But it wasn't every book, and it wasn't just white or green colored books. That happening made me go WTF?? :) |
|
Debug File is way toooooooo long now, so I'll email it to you. |
|
BTW, what checksum format do you use to determine the hash? I figured I could run some files through to see if they match, but the only thing I know so far is what type it isn't.. :) |
|
The checksum is computed by generating an MD5 hash of all the text and CSS files, ignoring images and the OPF file. The objective is to uniquely identify the content of the book, so that matches may be identified when the metadata doesn't match. |
|
I will review the debug file, but in the meantime, please try this:
Any differences? |
|
(Edited version of debug log follows) |
|
Deleting the hashes has taken my white books from around 250 to 13. Main problem fixed. Something was obviously wrong with the hashes. While fixing up some of the last thirteen white books, I did discover something. It seems that editing the book in calibre, then updating it to calibre left the book white. I tried just deleting the calibre hash info. Still White. On deleting both hash sets, a refresh turned the book green. Also happened that way if I just deleted the Marvin hash info. How does the plugin handle it if the hash it has is different to the files? Does it check for that? Is it possible the plugin has old hash data? I do have a couple of ebooks that have a different problem. The cover shows up in Calibre, and in Marvin, but not on Marvin XD. Even after deleting the book, and re-adding it (after polishing of course), the problem remains. And causes the book to be white. I'll send you a copy of the ebook to see if it shows up for you. Thanks so much for all your help so far! |
|
@uten, I've built an updated version of MXD with more diagnostics that should help pin down the remaining unmatched books. You can download it from this link. Save it to your desktop. Start calibre with your iPad disconnected, then install the downloaded plugin from Preferences|Advanced|Plugins. Create a debug log again with the same settings for development_mode (true). This version of MXD should show the unmatched books in gray. Please confirm that the unmatched books are shown in gray, and that the other books are displayed with the proper colors. The debug log will be large, email it to me as you have before. If you're having different issues with MXD, please open a new issue with those details. I will try loading the book you sent me to see if there's an obvious problem with it. In answer to your question about stored hashes, deleting the hashes from calibre and Marvin clears them completely, then they are recreated then next time you run the plugin. Calibre library hashes are stored in calibre's db, Marvin's hashes are stored on the iDevice. In general, is the common denominator with the books that are missing that you've polished them in calibre and sent them to Marvin again after polishing? |
|
FYI, the book you sent me The Trouble With Families loaded into Marvin and displayed in MXD without issue, so it's probably related to this issue. No need to open a new one. A couple of questions:
|
|
I sorta figured the problem was related - hence why I mentioned it in this thread. In hindsight, using the word different wasn't the best idea as by different, I meant the cover issue only. BTW, when you viewed the metadata of the book I sent you, did it show the cover on the copy that was sent to Marvin?? As for Polish, I only use the Update Metadata, and Update Cover options. I use the Replace option When Sending Books already in Marvin, as I discovered early on that it wasn't sending updated epubs to Marvin. I'll send you the log as soon as I can. Thanks again! |
|
With respect to the book you sent, MXD is not showing the cover in the metadata dialog, although the cover does display properly in Marvin's library view, calibre's library view and calibre's device view. The book is shown as green in MXD. There's clearly something wrong with the cover not showing in MXD. I'll track that down. |
|
The problem with the missing covers in the MXD metadata comparison dialog is now fixed. It was actually a typo in the iOS reader applications plugin, so you will need to download a revised version of that from this link.
That fix will not affect any of the matching color issues we've been working on in this issue. Thanks for spotting that. |
|
Ok, Debug file on the way. Sorry it took so long, but I had a migraine. :( The book used is the one I sent you - "The trouble with families" by LaraBr |
|
(Edited version of debug log focusing on missing books only) |
|
@uten, here's what I'm observing from the latest debug log. There are 12 unmatched books. 9 of those 12 - the ones with multiple UUID entries in the debug log - should have been flagged with a different color. I've added pink to represent books meeting the following conditions:
I've updated MXD. Please download and install it from this link, then confirm that there are 9 pink books and 3 gray books displayed. The other 3 - First Heart, The Trouble With Families, Waiting for Sunrise don't match because the hashes don't match between calibre and Marvin. We will need to do some more experiments to isolate why the hashes aren't matching. Please confirm that the updated plugin behaves as described, then we'll consider the next step. |
|
@uten, I had an insight about why the updated fanfics were improperly identified as yellow instead of gray. I've updated both MXD and iOSRA, you will need to download and install both: It's worth noting that an updated fanfic (or an edited epub) should never show yellow, only gray. If, after downloading a fanfic update a book is displayed as yellow, don't do the metadata update, but instead capture a debug log and post it. Please let me know if this works properly. |
|
Tested it twice so far.. The book shows up as Gray. I had to delete hashes and rebuild them. Do you want a debug log of this? |
|
Yes, please send a debug log. |
|
Also, please clarify… |
|
While everything is working right now - after rebuilding the hashes - I have made sure I left some fanfics to update so I can get the error to replicate. I'll send you the debug log in a few minutes. |
|
Another silly typo. Please do the following:
I apologize, I should have caught that one but didn't. I really appreciate your assistance and persistence. |
|
Done. And done.. Works perfectly now. And I was glad to help. :) I will let you know if anything does crop up, but it looks awesome now. |
|
Whew. I will close this issue, feel free to reopen it. I'm planning to release this version after revising the help file to describe the new colors. |
|
@uten, I'd like your input on a further refinement please. I'm planning to display a dialog when gray books are detected, just before showing the MXD window. This would be similar to the dialog displayed when duplicate content is detected. Additionally, I'm planning to mark duplicates and updated books with a temporary marker in the main calibre window to make it easier to isolate the books needing further attention. An updated version of MXD with this feature is available for download from this link. Here's how this will appear to the user:
The most likely scenario is that either one condition or the other is active - in other words, there is some duplicated content, or some updated content, but not both. In the event that both duplicated content and updated content are detected, all books would be marked with the temporary marker. Questions:
|
|
You read my mind.. I was trying to figure out a way you could output a list of duplicates for the magenta books as just knowing I had them did make them hard to track down, but I had completely forgotten about temp markers. I'll create some magenta books, and some gray books, so I can see what the output looks like. As for the third question, I think as long as they know they might lose their current temporary markers, there shouldn't be a problem - although given human nature, I'm sure someone will complain.. Regardless, both these conditions can't be found or fixed any other way, and help streamline both calibre's and Marvin's library. Lord knows enough of calibre's features already change the temp markers around. :) |
|
Ok. Got to have a good look at it and I think it's awesome!! I didn't find it confusing at all, for multiple reasons. Firstly, you can copy the info, so you can refer to it if you want, and secondly, and most importantly, the information is in two separate windows, that appear one after the other, so the information isn't really given to you at the same time anyway. If it was all on the one screen, it may have been a little more confusing, depending on layout of course, but this isn't confusing at all. I like that temp markers are added to all the books in both categories. The only thing I can think of to make everyone happy (which would be a miracle), is to make the temp marker selecting optional, which may not be possible. IE, add a tick box or something that lets them choose to select the Magenta books, the Gray Books, or both, or neither.. Hope that made sense, but it is probably far too much work to do that for the one or two grumpy people. :) During one of my debug saving sessions, this error popped up. It happened either when I closed the MXD window, or when I quit calibre - I did both a little too close together to know for certain which. And it didn't happen the next time. calibre, version 1.20.0 calibre 1.20 isfrozen: True is64bit: True |
|
I've uploaded a new build of MXD at this link. This version includes a checkbox in the config dialog General options section to disable the temp markers. I also added some code to recover from the error you reported above. |
|
Very nice. Only other suggestion is to split the checkbox into two - one for Magenta, and one for Gray - to let people work on just one group at a time if they want. |
|
Yes, definitely the right way to go. Download and install the latest version from this link. Feel free to suggest better wording for the dialogs, I know it's awkward. Note that the dialog text changes depending on whether flagging is enabled for each condition. |
|
Error found in the new selection. If both options are ticked, only the updated content gets temporary markers. If they are selected independently (only one at a time), they work as they should. As for the dialogs, I don't think they are awkward at all. They state the conditions and info needed, without any extraneous, confusing text added. You also have separated the points by paragraph breaks instead of bunching the information all together. I think the dialogs are well done. |
|
You are good tester. I thought I had tested for that. Updated version available from this link. A few more tweaks to the text in both the dialog tooltips and the dialogs. |
|
Very nice. Works perfectly now.. I've been thinking, and we have been using the term updated. We should have been using differing, as it is possible to add the book to Marvin, then transfer it to calibre. And it shows as gray, which it should. So instead of updated content, maybe call it differing content. What really called it out to me, was you use the sentence "Marvin books with outdated content" in the dialog window, but it's not necessarily true. As an example - I could use an online downloading service on my idevice to get an updated novel/fanfic/magazine whatever, then save it to Marvin. That puts the new version in Marvin, and the old one in calibre - and is a lot more work given that you have to also delete the old copy from calibre and you lose any metadata edits.... But you see what I mean??? |
|
You're right, that path is perfectly valid. I will think about the wording, and also how to possibly handle a backload from Marvin to calibre without losing metadata. |
|
@uten, here's a lengthy brain dump of the situation as I see it. First, here's a summary of how content mismatches (displayed as GRAY) are detected:
Now, let's consider typical workflows for adding books from calibre to Marvin, then tweaking in calibre:
Here's the workflow when adding a book from Marvin to calibre:
We've already seen that if a user-initiated change occurs to a book in the calibre library, say downloading an updated fanfic or editing the book, the UUID doesn't change, allowing MXD to detect that the book's identity is the same, but the content has been updated, resulting in GRAY. But what happens when the user replaces a book in Marvin? (Users cannot directly edit a book's content in Marvin, only some of the metadata.) If the user tries to re-download a book whose author and title match a book already installed in Marvin, Marvin does not replace it. This is a safety feature, as Marvin retains lots of state information about a book, including last read position, annotations, bookmarks, etc, all of which would be invalidated by replacing the underlying content of the book. So the only way to truly replace a book in Marvin, using Marvin's UI, is to a) delete the book from Marvin's library, then b) re-download it. Doing so creates an new entry in Marvin's database, without a matching calibre UUID. This suggests a particular scenario in which it might be possible to fool the current color algorithm:
My sense is that this is an unlikely scenario - once a Marvin user starts using calibre to manage content, they would tend to do all content management from calibre, not Marvin. The recommended approach would be to add the latest version of the book to the existing entry in calibre, retaining metadata and UUID. If I were to make a change, here's how I think it would work:
The implementation would be a context menu item for TEAL books, Update in calibre library, which would fetch a copy of the Marvin version of the book, apply that copy to the existing calibre entry, then update Marvin's UUID for that book to match the already-existing calibre entry for that title/author. This would retain existing metadata in calibre and get everything back in sync. I need to do more research to confirm that this is possible. It would be challenging to test. So, in summary, your scenario is certainly possible, but an informed user could avoid the problem by adding the updated content to calibre rather than Marvin. A more elegant solution would be to add a new color TEAL representing updated Marvin content, with a context menu action updating the calibre library from the Marvin book. |
|
I've updated the plugin to revise the terminology from 'outdated content' to 'differing content'. You can download the latest version of the plugin from this link. |
|
Wow. Lots of info to think about. I happen to agree that this is an unlikely situation. I only discovered it because I was actively attempting to check everything. I personally would never use Marvin in that way, unless I had to travel for a while. And that point there was why I brought it up. I do like your solution, both simple and elegant. Once you determine whether or not it's even possible, maybe a poll to see how many people would like the Teal color. Until then, a simple note, letting them know how to avoid the 'problem', would work. I'll keep thinking on it, and if I think of anything else, I'll let you know. |
|
With respect to the colors, what I've been doing is subdividing the color wheel, trying to pick colors as far apart as possible so they can be distinguished by folks with vision problems. So it's not personal taste, it's maximizing visible difference. I picked GRAY because it suggested something broken. One option with GRAY would be to do a lighter shade and a darker shade, which would keep them grouped and visibly related. But first I need to confirm the other requirements before fine-tuning the color assignment. I do appreciate the time and effort you've invested to help improve MXD - I wish you had been around when I was originally developing it. The recent improvements are significant. |
|
Out of curiosity, I added the code to subdivide GRAY into two groups - those where the content has been updated in calibre, and those where the content has been updated in Marvin. Calibre updates are still shown as GRAY, Marvin updates are shown as CYAN. You can download it from this link. To clarify, this is the only change - the CYAN group is treated identically to the GRAY group. I hacked Marvin's database to force a CYAN book in my test library for testing purposes, but I'm curious to know if any of your books show up as CYAN? |
|
I added a few books and edited a few more to try and get all the colors to show. I had all the colors show up. Cyan included. On a side note. There may be a problem with using light gray to do with viewing angle and color profiles... My three year old knocked my monitor askew while I was fixing up the library, and I went from having a white book and two gray books to having three off-white books. I took a screenshot (attached below) and checked on a windows desktop, a windows laptop and an ipad, and the only one that displayed it correctly at any angle was my ipad. Both windows computers displayed the gray lighter than the Mac, but the worst was the laptop despite the fact I got it looking identical to the ipad at the correct viewing angle. And even minor messing with the color profile also turned all three off-white.. Not sure if it's worth worrying about too much, but I also know there aren't many monitors that have the correct color profile. And it was only white and light gray that were affected like this. On the upside, the white and light gray were able to help me fine tune my color profiles so they showed correctly on my Windows computers. I couldn't decide if this was nitpicking or not, so I figured I'd let you decide. :) |
|
I darkened GRAY by 5%, that should help. Download from this link. |
|
That did it. You can still tell the difference, no matter what now. :) |
|
I've made another revision which you can download from this link. I'm still thinking about whether or not to add the functionality to update books from within the MXD dialog. If I do this, it will need to be bidirectional - in other words, you should be able to send a book updated in calibre to Marvin, and a book updated in Marvin should update the calibre copy. For the moment, the books are identified with enough details that the user can resolve the issues manually. |
|
Very nice. There is certainly enough information to resolve things quite easily. The problem with updating books via MXD isn't just that. Sometimes, the updated book will alter the metadata (especially true in Marvin as it creates a new book - in calibre you can usually disable it). So not only do you need to transfer the books in both directions, the option would need to be there to transfer the 'old' metadata to the updated book, so that any edits someone made aren't lost. Resolving the issue manually make take a little longer, but it isn't difficult at all. |
The attached screenshot shows several eBooks as missing from calibre.
The Cassandra Clare books are in a different calibre library so that is correct. The others however, do exist in the current calibre library.
Or have I misunderstood something?
ps Very best wishes for a great 2014. :)
The text was updated successfully, but these errors were encountered: