stackexchange-favs-to-pinboard

Check-in [29383c0dea]
Login

Many hyperlinks are disabled.
Use anonymous login to enable hyperlinks.

Overview
Comment:Add basic caching to speed up adding new bookmarks

Since have to (well should do, to be nice) rate limit quite a lot it can
be slow to add new bookmarks to Pinboard when we are checking Pinboard
for every single StackExchange fav. Keep a local cache of items already
added to Pinboard.

Some things I have to do (note to self):

- Check for success on Pinboard add before adding to cache
- Consider removing bookmarks on Pinboard for unfaved StackExchange
items?

Downloads: Tarball | ZIP archive | SQL archive
Timelines: family | ancestors | descendants | both | origin/master | trunk
Files: files | file ages | folders
SHA3-256: 29383c0dea82c000cb483f09eaff5260c93d8a3d9d1308db14ab9394b64ba265
User & Date: base@atomicules.co.uk 2015-06-20 18:40:57
Context
2017-03-12
14:54
Make this actually work for lots of favs

Don't know if somehting changed or this was just always semi-broken
(probable), but noticed today it wasn't adding any new favs when I knew
for a fact I had some; Perhaps, also, I have a lot more favs than when I
first wrote this.

Previously this assumed that the response returned all favs for a site
whereas actually by default it returns just 30. Perhaps by luck I'd been
picking up new ones because of using "sort=activity"; a strange choice
on it's own as if I'd used "sort=added" and run this fairly regularly
perhaps I'd never had noticed any issue? Anyway, now it does the correct
thing which is pull 100 at a time and look for the has_more parameter in
what is returned so it knows whether it needs to fetch another page.

This does mean get_favs now returns an array of items (makes more sense)
than the actual encompassing response as it used to. Leaf check-in: 09c6947036 user: base@atomicules.co.uk tags: master, trunk

2015-06-20
18:40
Add basic caching to speed up adding new bookmarks

Since have to (well should do, to be nice) rate limit quite a lot it can
be slow to add new bookmarks to Pinboard when we are checking Pinboard
for every single StackExchange fav. Keep a local cache of items already
added to Pinboard.

Some things I have to do (note to self):

- Check for success on Pinboard add before adding to cache
- Consider removing bookmarks on Pinboard for unfaved StackExchange
items? Leaf check-in: 29383c0dea user: base@atomicules.co.uk tags: origin/master, trunk

2015-04-06
10:21
:Merge branch 'master' of github.com:atomicules/stackexchange-favs-to-pinboard check-in: 2f9937b611 user: base@atomicules.co.uk tags: origin/master, trunk
Changes
Hide Diffs Unified Diffs Ignore Whitespace Patch

Changes to stackexchange-favs-to-pinboard.rb.

86
87
88
89
90
91
92

93





94
95
96
97
98
99
100
101
102


103

104


105
106
107





		end
		posted
	end
end


if defined?(StackID) and defined?(User) and defined?(Token)

	pb = Pinboard.new(User, Token)





	parsed = get_sites(StackID)
	parsed["items"].each do |site|
		favs = get_favs(site["site_url"].sub("http://", "").sub(".stackexchange", "").sub(".com", ""), site["user_id"])
		#Don't make more than 30 requests per second (unlikely too anyway, what with Pinboard adding)
		sleep (1.0/30)
		favs["items"].each do |fav|
			title = fav["title"]
			tags = fav["tags"]
			link = fav["link"]


			#Need to unescape so can re-escape in Pinboard code

			pb.add(link, CGI.unescape_html(title), nil, tags.join(", ")+", stackexchangefavs", "no")


		end
	end
end












>

>
>
>
>
>



|





>
>
|
>
|
>
>
|
|
|
>
>
>
>
>
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
		end
		posted
	end
end


if defined?(StackID) and defined?(User) and defined?(Token)
	cache_file = ENV["HOME"]+"/.stackexchange_favs_to_pinboard"
	pb = Pinboard.new(User, Token)
	if File.exists?(cache_file)
		cache = JSON.parse(File.read(cache_file))
	else
		cache = {}
	end
	parsed = get_sites(StackID)
	parsed["items"].each do |site|
		favs = get_favs(site["site_url"].sub("http://", "").sub(".stackexchange", "").sub(".com", ""), site["user_id"])
		#Don't make more than 30 requests per second
		sleep (1.0/30)
		favs["items"].each do |fav|
			title = fav["title"]
			tags = fav["tags"]
			link = fav["link"]
			#Check cache to see if already added to pinboard
			unless cache.has_key?(link)
				#Need to unescape so can re-escape in Pinboard code
				#Still want no default replace, just in case cache doesn't exist
				pb.add(link, CGI.unescape_html(title), nil, tags.join(", ")+", stackexchangefavs", "no")
				#Add to cache. Should check for success really
				cache[link] = true
			end
		end
	end
	#Write out cache
	File.open(cache_file, "w") do |file|
		file << cache.to_json
	end
end