Mobileread
Calibre + Kindle: Fetching User Annotations
#1  GRiker 03-05-2010, 07:37 PM
Calibre 0.6.44 is now able to fetch user annotations from a connected Kindle. This means that any Bookmarks, Highlights or Notes you have created on the Kindle (as displayed in 'My Notes & Marks') may be merged into the book's comments field in calibre's database. 0.6.46 adds archiving of 'My Clippings.txt' to calibre's database.

Important note about migrating books from Kindle to calibre:
If you have not yet migrated your books from the Kindle to calibre, please read this section carefully.

Kindle supports three types of user annotation, plus Last Page Read:To utilize the Fetch annotations feature, you will need to migrate your books into calibre following the steps listed below.

If you have existing annotations for Kindle books that you wish to keep, including Last Page Read locations, you need to migrate your annotations into calibre then resave them to the Kindle following the steps below.

If you don't care about any existing annotations, or keeping your Last Page Read locations, you may skip steps 3 & 7 for each book.

This is a one-time process – once you've migrated a book from the Kindle to calibre, you won't need to do it again.

Migrating books and annotations into calibre
(0.6.45 required for step 7)
:
  1. Make a backup copy of the documents folder on the connected Kindle. A USB memory stick works well for this important step. If there's a problem, you'll be able to restore the contents of your Kindle/documents directory.
  2. Add the book to the calibre database
    • Click the down arrow to the right of the Add books icon, then select Add books from a single directory
    • Navigate to the 'documents' folder on your connected Kindle
    • Select the book to add. Supported Kindle book extensions include azw, mobi, prc, tpz, and azw1.
    • Click 'Open'. The Adding ... dialog is displayed. When the dialog closes, the book has been added to the calibre database.
  3. Add the annotations to the database
    • Select the book just added to calibre
    • Click the Edit meta information icon
    • Click the Add a new format for this book to the database icon.
    • Navigate to the 'documents' folder on your connected Kindle
    • Select the annotations file matching the book you added in step 2. Supported Kindle annotation extensions include .mbp and .tan. For example, if you added 'The-War-of-the-Worlds.azw' in step 2, you will now add 'The-War-of-the-Worlds.mbp' in this step.
    • Click 'Open'. The Adding ... dialog is displayed. When the dialog closes, the book's annotation file has been added to the calibre database.
  4. Edit the metadata
    • Edit title, author, comments, etc. to your liking.
    • Click OK to accept the changes and close the Edit Meta Information dialog.
  5. Delete the original file on the Kindle
    • Click on the Reader icon
    • Select the book
    • Remember that you backed everything up in step 1.
    • Click the Remove books icon to remove the original book from the Kindle
  6. Send the book with edited metadata to the Kindle
    • Click on the Library icon to display the books in your database.
    • Select the book
    • Click the Send to device icon
  7. Save the annotations to the Kindle (This step available with 0.6.45 and later)
    • Click the down arrow to the right of the Save to disk, then Save single format to disk ....
    • Select either MBP or TAN, matching the extension of the annotation file you added in step 3.
    • Select the 'documents' folder on your connected Kindle as the destination directory
    • The Saving dialog is displayed. When the dialog closes, the annotation file has been saved to the Kindle in the new location.
  8. Repeat for each book to be migrated
  9. Confirm the changes
    • Eject the Kindle
    • Confirm that the titles/authors conform to the metadata values in calibre
    • Select the book, toggle right, click 'My Notes & Marks' to view the annotations on the Kindle.
    • Confirm that the annotations have arrived safely.
Using Fetch annotations
Here's a 'My Notes & Marks' menu with some sample user annotations:

image »
Fetch annotations merges user annotations created on the Kindle with a book's comments in the database.

Invoking Fetch annotations:
After the dialog is dismissed, you can review your annotations in calibre. Select the book in the main window, then type 'I' or click in the lower metadata preview window ...
image »
... which will bring up the metadata viewer:
image »
Caveats:
  1. Fetch annotations is an experimental feature introduced in version 0.6.44. I don't believe that it will harm your Kindle or your annotations.
  2. If you're a new calibre user and you haven't yet migrated your books to calibre, see the section 'Migrating books and annotations into calibre' above to get started.
  3. The Kindle must be connected for Send to device↓Fetch annotations to be enabled. Wait for the 'Reader' icon to display in the upper window.
  4. Calibre looks for the annotation file in the same location that it stores the eBook on the Kindle.
  5. Fetch annotations will not find annotation files (*.mbp, *.tan) in locations other than where calibre would store the eBook. See #4.
  6. If you use more than one Kindle with calibre, Fetch annotations will import annotations from the currently connected Kindle, replacing any previous annotations.
  7. Currently supported formats include MOBI, PRC, AZW, (*.mbp) TPZ and AZW1 (*.tan). PDF is not currently supported.
  8. Fetch annotations is implemented only for the Kindle at this time.

Question, comments, suggestions? Please post them in this thread.

G
Reply 

#2  daffy4u 03-05-2010, 08:18 PM
This is most excellent! Thanks for all your hard work!
Reply 

#3  TuxGirl 03-05-2010, 08:57 PM
Yay!!!!!!!!!!!!!!!! thank you!!!!!!!!!!!!!!!!!!!!!!!!!!
Reply 

#4  FoolforBooks 03-05-2010, 09:58 PM
Great, but.. when you edit the metadata, changing the title, and send that book back to the Kindle with Send to Device, are you re-naming the .mbp files to match the re-named book file? That is the only way those notes and highlights are still accessible through MyNotes&Marks.

The alternative would be not to re-name the file, leaving in Amazon's format, but change only the metadata within the file. Then the mbp files would still match up. I was very curious why you re-named the file at all.

And does #4 mean we must be using that sub-directory business with all the separate author and book files. I have disabled that.
Reply 

#5  GRiker 03-06-2010, 06:54 AM
Quote FoolforBooks
Great, but.. when you edit the metadata, changing the title, and send that book back to the Kindle with Send to Device, are you re-naming the .mbp files to match the re-named book file? That is the only way those notes and highlights are still accessible through MyNotes&Marks.
I'll take a look at migrating the last-cached .mbp file with the book when using Send to device. That would obviously be helpful.

Quote FoolforBooks
The alternative would be not to re-name the file, leaving in Amazon's format, but change only the metadata within the file. Then the mbp files would still match up. I was very curious why you re-named the file at all.
That design decision precedes my involvement with calibre, but my sense is renaming the files is the right choice.

Quote FoolforBooks
And does #4 mean we must be using that sub-directory business with all the separate author and book files. I have disabled that.
No, you're not forced to use any particular scheme, but the scheme you use must be declared in Preferences|Add/Save|Sending to device. The storage pattern declared there is used to generate the path to find the associated .mbp file. So as long as you're using calibre to Send to device, Fetch annotations should be able to find the associated .mbp file. If the scheme you're using isn't finding the .mbp file, please post a bug report.

G
Reply 

#6  FoolforBooks 03-06-2010, 03:42 PM
G--I misunderstood. I thought you were going to be managing the retention of the actual mbp and tan files. That is a concern when someone makes a change to the title of a book. Calibre changes the file name, and the mbp file no longer matches, so the notes and highlights are *lost*. They need re-named to the exact file name you used when Calibre sends the book to the device.

After doing metadata changes, I went in via usb and changed the mbp file names myself. I had left them on the Kindle. My understanding is that you are recommending people to clear their Kindle completely, and edit metadata, and re-send all their books via Send to Device. I think this is a little radical. If you are going to recommend this, then you need to tell people to backup those mbp files first (or leave them on the Kindle), and to rename them to match the new filenames on the books after they are sent back via Calibre. And to do it before they open the book, because the Kindle software will immediately create an mbp file with the new name. The re-naming process is complicated by having to delete that one and re-name the one containing the notes. (Or do all this within Calibre.)

I want to make sure I understand what you are going to do. If I do this, and I have 50 screens of highlights in a book, you will put all 50 pages in text format into the Comments box in the Calibre library? I'm not sure I understand the objective, but if people asked for this enhancement they must know what they will do with the data.

When you build a Calibre Catalog, and send it back to the Kindle, the Catalog will contain all 50 pages of highlights as Comments?

Does any of this require allowing Calibre to read the metadata when the Kindle connects to Calibre? I have disabled that as it was taking too long, and I didn't see why you needed to do it. I asked Kovid, but he didn't answer that part of the question.
Reply 

#7  GRiker 03-06-2010, 07:46 PM
Quote FoolforBooks
G--I misunderstood. I thought you were going to be managing the retention of the actual mbp and tan files. That is a concern when someone makes a change to the title of a book. Calibre changes the file name, and the mbp file no longer matches, so the notes and highlights are *lost*. They need re-named to the exact file name you used when Calibre sends the book to the device.
There are some internal architecture changes for device drivers that need to be in place before calibre can store/replace annotations files. That work is currently under way.

Quote FoolforBooks
After doing metadata changes, I went in via usb and changed the mbp file names myself. I had left them on the Kindle. My understanding is that you are recommending people to clear their Kindle completely, and edit metadata, and re-send all their books via Send to Device. I think this is a little radical. If you are going to recommend this, then you need to tell people to backup those mbp files first (or leave them on the Kindle), and to rename them to match the new filenames on the books after they are sent back via Calibre. And to do it before they open the book, because the Kindle software will immediately create an mbp file with the new name. The re-naming process is complicated by having to delete that one and re-name the one containing the notes. (Or do all this within Calibre.)
I am working on some additional documentation to explain how to migrate from simple Kindle storage to calibre-assisted storage. You are correct that any pre-existing annotations that the user wishes to retain need to be renamed to continue to be available after calibre begins managing file storage on the Kindle.

Quote FoolforBooks
I want to make sure I understand what you are going to do. If I do this, and I have 50 screens of highlights in a book, you will put all 50 pages in text format into the Comments box in the Calibre library? I'm not sure I understand the objective, but if people asked for this enhancement they must know what they will do with the data.
Yes, that's what happens. If you're not interested in being able to see annotations in calibre, then you wouldn't use Fetch annotations.

Quote FoolforBooks
When you build a Calibre Catalog, and send it back to the Kindle, the Catalog will contain all 50 pages of highlights as Comments?
Currently (0.6.44), the user annotations are stored with the comments in the generated catalog, but that will change in the next release - user annotations will not be included in the generated catalog.

Quote FoolforBooks
Does any of this require allowing Calibre to read the metadata when the Kindle connects to Calibre? I have disabled that as it was taking too long, and I didn't see why you needed to do it. I asked Kovid, but he didn't answer that part of the question.
No, you don't need to have metadata reading enabled for the feature to work.

G
Reply 

#8  andrys 03-06-2010, 09:00 PM
Quote FoolforBooks
G--I misunderstood. I thought you were going to be managing the retention of the actual mbp and tan files. That is a concern when someone makes a change to the title of a book. Calibre changes the file name, and the mbp file no longer matches, so the notes and highlights are *lost*. They need re-named to the exact file name you used when Calibre sends the book to the device.

After doing metadata changes, I went in via usb and changed the mbp file names myself. I had left them on the Kindle. My understanding is that you are recommending people to clear their Kindle completely, and edit metadata, and re-send all their books via Send to Device. I think this is a little radical. If you are going to recommend this, then you need to tell people to backup those mbp files first (or leave them on the Kindle), and to rename them to match the new filenames on the books after they are sent back via Calibre. And to do it before they open the book, because the Kindle software will immediately create an mbp file with the new name. The re-naming process is complicated by having to delete that one and re-name the one containing the notes. (Or do all this within Calibre.)

I want to make sure I understand what you are going to do. If I do this, and I have 50 screens of highlights in a book, you will put all 50 pages in text format into the Comments box in the Calibre library? I'm not sure I understand the objective, but if people asked for this enhancement they must know what they will do with the data.

When you build a Calibre Catalog, and send it back to the Kindle, the Catalog will contain all 50 pages of highlights as Comments?

Does any of this require allowing Calibre to read the metadata when the Kindle connects to Calibre? I have disabled that as it was taking too long, and I didn't see why you needed to do it. I asked Kovid, but he didn't answer that part of the question.
I'm one who makes highlights and notes on almost every book I read, but especially the non-fiction ones (which is most of what I read), so it's important to me what you're saying here...

I don't want to make or recommend making changes that would cause the notes and highlights to be gone unless we went to some work aligning the annotation/status auxiliary file with the newly edited book file when the title is actually changed.

I use the http://kindle.amazon.com page a lot, and it's sort of crucial I don't lose those annotations.


For non-Amazon books, no worry (I think).
Reply 

#9  GRiker 03-07-2010, 12:07 PM
@FoolForBooks, @andrys: I've revised the first posting to help users who have not yet migrated book management to calibre with the process. I made some changes to the Save books functionality which won't be available until 0.6.45. With those changes, it is possible (though somewhat tedious) to migrate books and annotations from the native Kindle-style storage to calibre-style storage.

After the architectural changes are in place for the device drivers, it should be simpler for new users to migrate annotations with their books.

Thanks for the feedback.

G
Reply 

#10  GRiker 03-07-2010, 03:16 PM
Quote andrys
I'm one who makes highlights and notes on almost every book I read, but especially the non-fiction ones (which is most of what I read), so it's important to me what you're saying here...

I don't want to make or recommend making changes that would cause the notes and highlights to be gone unless we went to some work aligning the annotation/status auxiliary file with the newly edited book file when the title is actually changed.

I use the http://kindle.amazon.com page a lot, and it's sort of crucial I don't lose those annotations.
@andrys, if you'd be willing to help me test the interactions with the Amazon page, that would be appreciated.

If you're already using calibre to manage the books on your Kindle, everything should just work. If it doesn't, I'd like to resolve any problems.

G
Reply 

  Next »  Last »  (1/3)
Today's Posts | Search this Thread | Login | Register