Mobileread
Update size is different every attempt?
#1  Wolferl 09-22-2009, 07:27 AM
Hi folks,

I am just new to SunriseXP (having read the FAQ )
After AvantGo shut down, I wanted to have another possibility to have a TV guide on my Compaq iPAQ (PPC, Vade Mecum reader). I tried Plucker, but it is not able to handle cookies, which is important for www.klack-channels.de. So the search continued...
SunriseXP did the job well at first sight. Then I discovered, that some links were not downloaded. I increased link depth and retried. Now, even less stuff is created! The pdb file, originally 220kb, is now between 2k and 50kb, it changes on every "update" event I do.

Question 1:
How come it does not update all the stuff?

Question 2:
Is there a setting "maximum file size" or something which prevents SunriseXP downloading ALL the stuff?

I'm a bit lost...

Thanks for help,
Wolferl

#2  Wolferl 09-23-2009, 06:43 AM
I've done a bit of research and found out that SunriseXP gets errors when retrieving some of the HTML files (notHandled: 403 Forbidden).

It seems there is a problem requesting the files too quickly.

Is there a setting to slow spidering down a bit? Or re-fetching the HTML if it fails?

Cheers,
Wolfgang

#3  Laurens 09-23-2009, 09:13 AM
No, there is no setting to configure spidering speed. You might want to limit the max. number of connections to get around this. Unfortunately, this is a global setting that affects all sites.

#4  Wolferl 09-23-2009, 10:24 AM
Hi Laurens,

Quote Laurens
No, there is no setting to configure spidering speed. You might want to limit the max. number of connections to get around this.
You mean "View -> Preferences -> Network tab -> "maximum active updates"?
I already set this to "1". Unfortunately, Sunrise runs two requests at the same time then. There seems to be an "off by one" bug...

But anyways, that does not really solve the problem:
Klack-channels.de seems to block access with a "403 forbidden" if one retrieves pages too quickly.
I tried to use a special proxy software that allows throtteling (Charles Web Debugging Proxy), and using this, it was possible to get the whole thing spidered without any errors. It just takes 20 minutes (only 600kb file size)...and that proxy is not freeware.

If there would be configurable delay "delay before request" in seconds, that would be perfect.

Cheers,
Wolferl

#5  Laurens 09-23-2009, 04:27 PM
There's not much I can do for you, unfortunately, since I'm no longer maintaining the application. You might want to look around for some other (hopefully free) proxy that can handles bandwidth throttling.

#6  Wolferl 09-24-2009, 04:32 AM
Hi Laurens,

Quote Laurens
There's not much I can do for you, unfortunately, since I'm no longer maintaining the application.
Since I have lots of programming experience...what do you think about doing this modification myself? I have searched around for the source, but wasn't successful...

Cheers,
Wolferl

#7  Laurens 09-25-2009, 08:43 AM
Quote Wolferl
Hi Laurens,



Since I have lots of programming experience...what do you think about doing this modification myself? I have searched around for the source, but wasn't successful...

Cheers,
Wolferl
SunriseXP (C++) is unfortunately not open-source. It uses some closed-source libraries, preventing its release as an open-source project.

You might want to consider switching to the older Java version, which is open-source.

http://sourceforge.net/projects/sunrisexp/

#8  Wolferl 09-26-2009, 05:04 AM
Hi Laurens,

Quote Laurens
SunriseXP (C++) is unfortunately not open-source. It uses some closed-source libraries, preventing its release as an open-source project.
Oh, too bad

Quote Laurens
You might want to consider switching to the older Java version, which is open-source.
http://sourceforge.net/projects/sunrisexp/
OK, I'll stick with this one.

Thanks for your help!

Wolferl

Today's Posts | Search this Thread | Login | Register