Mobileread
Large library in KOReader
#1  ctop 02-02-2020, 08:29 PM
Hi there,
I wonder how folks here are handling large libraries in KOReader. I currently have about 4500 epubs on my devices, indexed using the book scan within KOReader. Whenever I go to the filemanager to look for a new book, it takes ages (like in: several minutes) to open the first screen. On the bottom it says something like page 12 of 430 pages, so I know, there is a lot of calculation going on to put my in the right place (although that is not the place I want to be in..). So this is clearly not the best way to handle this, but what are the alternatives? BTW, my library is handled in Calibre and sent to the device, but not using collections, just a simple structure of Authorname/Title on the file system.
There must be better ways to do this!
All the best,
Ctop
Reply 

#2  NiLuJe 02-02-2020, 11:57 PM
What is the FM sorting by? IIRC, some sort methods are massively slower than others.

(Also, on which device is that ).

And what, exactly, are you calling "the book scan"?
Reply 

#3  ctop 02-03-2020, 01:13 AM
Quote NiLuJe
What is the FM sorting by? IIRC, some sort methods are massively slower than others.

(Also, on which device is that ).

And what, exactly, are you calling "the book scan"?
I am currently using KOReader on a Likebook Ares and a Kobo Forma. The behaviour can be observed on both devices.

The sorting is currently on "name mixed" (this takes more than 10 minutes to display the first screen on my Ares), but I have tried other sorting methods, like "last read" or "filename" to see if this speeds things up, but to no avail. Any suggestions on what might be most effective are welcome.

What I mean by book scan is the "Book info cache management", which scans all the books for metadata and puts them in a cache (if I understand correctly).

All the best,
Ctop
Reply 

#4  ilovejedd 02-03-2020, 01:35 AM
Which part is slow? By file manager or by Calibre metadata search?

If it's the file manager, add some more hierarchies to the directory structure. A folder on FAT32 with thousands of entries in it doesn't make for good performance.

For example:

Code
{author_sort:.1}/{author_sort}/{title}
If I remembered the syntax correctly, that should give you:

Code
/L/Last Name, First Name/Title
Reply 

#5  Frenzie 02-03-2020, 04:12 AM
I think filename, date added, date modified, and size should be the fastest (based on a quick peek at the code, not experiments). Mixed filenames & folders adds another step on top of regular filename, for example.

Basically you'll just want to subdivide a little further; even without speed concerns I imagine 400+ pages are very inefficient to navigate. ilovejedd posted a good suggestion for that.
Reply 

#6  pazos 02-03-2020, 11:04 AM
Quote ctop
What I mean by book scan is the "Book info cache management", which scans all the books for metadata and puts them in a cache (if I understand correctly).
That's not needed. Specially if you put a huge number of documents on your reader. AFAIK it is used for the cover view. The alternative: do not index. CoverBrowser will do that for you the first time you open a folder, which hopefully will have less than 50-100 books, instead of the thousands you're trying to index at once.

FWIW, I have a bunch of pdf documents on my cervantes, stored in pdfs/work/year/month. Some months I gather 50-100 documents, other months less than 10. In any case I don't do any indexing and most of the time I do not read them on the device, but with the device hooked to a computer.

I just did the test on September 2019 folder (121 documents, most of them single page receipts but a few large memos). Never opened from the filemanager before. The filemanager opens them in "classic mode view" and start updating them one by one (1s each one aprox).
Reply 

#7  Frenzie 02-03-2020, 11:18 AM
Actually CoverBrowser does it per page/view, not for the whole folder, afaik. :-)
Reply 

#8  Ken Maltby 02-03-2020, 12:01 PM
I have over 7000 documents/ebooks that are on my external micro-SD card. They are in some 50+ subject/category folders, behind a .books hidden folder. My "Home" is that .books folder, it is currently three pages, at ~14 entries/folders per page.

Luck;
Ken
Reply 

#9  NiLuJe 02-03-2020, 01:32 PM
Yeah, definitely don't play with the BookInfo cache, it's not doing what you think it does (like, at all) .

It's hidden seventy nesting layers deep for a reason ^^.

(The only non-developer valid use-case for messing with that is if you've massively changed your Library's content, and want to purge old thumbnails from no-longer-available books... to end up saving, like, 5MB.

TL;DR: Don't.).
Reply 

#10  ilovejedd 02-03-2020, 02:13 PM
Quote NiLuJe
(The only non-developer valid use-case for messing with that is if you've massively changed your Library's content, and want to purge old thumbnails from no-longer-available books... to end up saving, like, 5MB.
Lol, I actually use it quite often to batch update metadata for in-progress fics.

For kicks, I tried copying 5,458 EPUB files to a new test folder on the PW4. Indexing took 1 hour 3 minutes.

/Fandom/A/Authors - Series 01 - Title

Most number of files in a single folder is 654. Browsing is reasonably fast with file system based sort methods although I probably should've nested it some more.

It seems when you open a "parent" folder, the "child" folders also get scanned so it can display how many items/"grandchildren" there are in the the "child" folders. That said, even with 3125 "grandchildren" inside 27 "child" folders, opening the parent folder didn't take too long (2-3 seconds).
Reply 

  Next »  Last »  (1/4)
Today's Posts | Search this Thread | Login | Register