LibrarianSync: Kindle collections builder
#1  barsanuphe 08-22-2014, 07:25 PM
What it is.

LibrarianSync is a Kindle collection builder, originally meant as a complement to Librarian, but which can be used completely independently.

It has been tested on a Kindle PW2 (EU, Wi-Fi) FW, and on a Touch FW 5.3. It probably works on other K5 devices. It apparently works on the Voyage as well.

It can use as a source:
Using the json file, the entries can either remplace all existing collections or update existing collections. Using folder structure, all previously existing collections are dropped.

The native json format looks as follows:

{ "library/Alexandre Dumas/Alexandre Dumas (2004) Les Trois": ["gutenberg","french","not read"], "library/Alexandre Dumas/Alexandre Dumas (2011) Le Comte De": ["gutenberg","french","most awesome book ever"], "re:Alexandre Dumas (Père|Fils)": ["dumas"]
As shown in the last line, you can add (python) regular expressions instead of filenames (with a re: prefix); all matches will be added to the associated collection.

What is needed.

LibrarianSync comes as a KUAL menu entry. As such, it requires: a jailbroken PW2, MKK, KUAL, and python (snapshot > 0.10N-r10867).

For more information, see the README file on github.

Where it is.

The code resides on github, but since I do not use this personally anymore, the better place to look for changes is NiLuJe's fork (thanks!).
See attachments for an up to date Kindle package.

Note: the code has moved to a separate repository as LibrarianSync has become something useful in its own right. Please update your git remotes to the new repository.

Where it comes from.

LibrarianSync is inspired by this thread.

Thanks to NiLuJe for all the contributions and continuous support, and kaznelson for his initial courageous sqlite investigations.

Feedback is very much welcome.

LibrarianSync Changelog

v20201109 - Same as below . -- NiLuJe
v20190103 - Refreshed package from latest sources, so as not to leave four year old packages around... -- NiLuJe
v1.10 - fixed when getting a null cdetype, added Voyage support.
v1.9 - fixed support of non-cloud aware devices
v1.8 - fixed collection appearing twice in FW 5.6
v1.7 - added support for regular expressions in librarian json when building collections with LibrarianSync.
v1.6 - fixed exporting collections, added downloading ebooks hosted over http by librarian.
v1.5 - package available for all k5 models.
v1.4 - updating collections is now faster.
v1.3 - calibre plugin json support & ability to export current collections (librarian and calibre plugin versions). While running the script, current operation now logged and displayed.
v1.2 - experimental calibre plugin json support & ability to export current collections to extensions/exported_collections.json.
v1.1 - experimental calibre plugin json support
v1.0 - original version
[gz] librariansync-1.0.tar.gz (7.0 KB, 1706 views)
[gz] librariansync-1.1.tar.gz (7.4 KB, 1275 views)
[gz] librariansync-1.2.tar.gz (8.0 KB, 1312 views)
[gz] librariansync-1.3.tar.gz (10.6 KB, 1408 views)
[gz] librariansync-1.4.tar.gz (10.9 KB, 1409 views)
[gz] librariansync-1.5.tar.gz (11.4 KB, 1365 views)
[gz] librariansync-1.6.tar.gz (12.4 KB, 1387 views)
[gz] librariansync-1.7.tar.gz (12.7 KB, 1774 views)
[gz] librariansync-1.8.tar.gz (12.7 KB, 1415 views)
[gz] librariansync-1.9.tar.gz (12.7 KB, 1645 views)
[gz] librariansync-1.10.tar.gz (13.3 KB, 5457 views)
[gz] librariansync-20190103.tar.gz (13.8 KB, 1566 views)
[gz] librariansync-20201109.tar.gz (14.0 KB, 628 views)

#2  knc1 08-22-2014, 09:02 PM
Quote barsanuphe
Thanks to all the (old but still relevant) information in this thread I managed to write a python script to automatically generate collections on my pw2.
Instead of looking at the files or directory structure, it uses information in a json file to create and populate collections. This way it's possible to manage collections just with one simple text file, such as:

In case someone might find this helpful, the script and KUAL menu entry can be found at
Thanks for this.

It comes at a good time, since the interactive Collection Manager is not (yet) ported to all of the PW2 firmware / hardware combinations.

I think this will also work on other models of Kindles - that DB probably doesn't (didn't) change that often.

#3  barsanuphe 08-24-2014, 04:43 AM
Yes, I guess it should work with other models, after all this thread was originally about the Kindle Touch. I only have a pw2 (EU), so I can't verify that. If anyone is willing to try it on a different model, feedback would be welcome.

Also, I updated the script and KUAL menu to allow for incremental collection updates, that do not rebuild all collections but merely add the json entries into existing or new collections.

#4  NiLuJe 08-24-2014, 07:08 PM
@barsanuphe: I'll probably get my ass in gear and bundle the sqlite3 & requests Python modules in the next snapshots, FYI; that should hopefully let you get rid of the curl & sqlite3 forks.

EDIT: Done .

#5  knc1 08-24-2014, 07:55 PM
The Amazon Lua install includes the sqlite and json add-on libraries as I recall (at least on the PW1, I think on the KT also, PW2 is unknown).

If someone wanted to get ambitious, they could re-write this in Lua and avoid any add-in support packages.

The same code (and libraries) shipped by Amazon can also be run by the ARMhf Luajit if the user had those installed.

- - - -

Just a thought -
Not something I have time to do.
And twobob is still off, dealing with "life".

#6  barsanuphe 08-28-2014, 08:01 PM
Quote NiLuJe
@barsanuphe: I'll probably get my ass in gear and bundle the sqlite3 & requests Python modules in the next snapshots, FYI; that should hopefully let you get rid of the curl & sqlite3 forks.

EDIT: Done .
Great! I hadn't seen your update. I updated my script to use the new modules.
I didn't know the requests module, it's pretty nifty.

#7  NiLuJe 08-28-2014, 08:57 PM
@barsanuphe: That was fast ^^.

Yep, requests is great . Did the switch affect performance in any measurable way?

#8  kaznelson 08-29-2014, 01:13 AM
Whether it's possible to make "lite" version?
To create collections on folders names (without metainfo), and worked ONLY on Kindle? (without computer)

#9  barsanuphe 08-29-2014, 05:23 AM

I have not looked at the performance aspect yet, and am testing with only a few ebooks on my Kindle. I expect it to be faster though, but couldn't tell you by how much. Also, all operations are now sent in a single POST request, which should help too.


I updated the script and added a new entry in the KUAL menu to that effect. You should now be able to create collections from your folder structure, without json file. The collection name is the path of each ebook (without filename), relative to /mnt/us/documents.

ie if you have "/mnt/us/documents/sf/franck herbert/dune/dune.epub", the collection name will be "sf/franck herbert/dune".

I tried earlier to make nested collections, but I found only top collections are displayed on my Kindle. A subcollection can be added to a collection but does not show up on the interface. Reading through the Collection Manager thread, it seems it's a firmware issue...

#10  NiLuJe 08-29-2014, 09:44 AM
@barsanuphe: Yup, nesting was broken in FW 5.4.2, and it hasn't resurfaced since.

I was thinking of using this with the Calibre Kindle Collections plugin, most likely by making it understand the json schema used by the plugin, since the plugin already takes care of a number of sanity checks (namely the uuid/ASIN handling of each book).

FWIW, the schema is as follows:

{	"<CollectionName>@<Language>":	{	"items":	[	"<hash>"	],	"lastAccess": <epoch>	}
So, as an example with a few different types of hash, that would look something like:

{	"Apps@en-US":	{	"items":	[	"#B006VZUZV2^AZW2"	],	"lastAccess": 1344387337000	},	"Brandon Sanderson@en-US":	{	"items":	[ "*c98b6567481d950dccad195379053e2539f1d75c", "#12e45a36-0410-4caf-b32d-dc807713d8e9^EBOK"	],	"lastAccess": 1234567889976	}
(The indentation is all mine, for formatting purposes only).

  Next »  Last »  (1/54)
Today's Posts | Search this Thread | Login | Register