stackexchange-favs-to-pinboard

Check-in [09c6947036]
Login
Home Timeline Branches Tags Tickets Wiki

Many hyperlinks are disabled.
Use anonymous login to enable hyperlinks.

Overview
Comment:Make this actually work for lots of favs

Don't know if somehting changed or this was just always semi-broken
(probable), but noticed today it wasn't adding any new favs when I knew
for a fact I had some; Perhaps, also, I have a lot more favs than when I
first wrote this.

Previously this assumed that the response returned all favs for a site
whereas actually by default it returns just 30. Perhaps by luck I'd been
picking up new ones because of using "sort=activity"; a strange choice
on it's own as if I'd used "sort=added" and run this fairly regularly
perhaps I'd never had noticed any issue? Anyway, now it does the correct
thing which is pull 100 at a time and look for the has_more parameter in
what is returned so it knows whether it needs to fetch another page.

This does mean get_favs now returns an array of items (makes more sense)
than the actual encompassing response as it used to.

Downloads: Tarball | ZIP archive | SQL archive
Timelines: family | ancestors | trunk | master
Files: files | file ages | folders
SHA3-256: 09c69470368aa60945a1a3d1d277eef304c39e15540c60b864a9c6fef4e3feae
User & Date: base@atomicules.co.uk 2017-03-12 14:54:59
Context
2017-03-12
14:54
Make this actually work for lots of favs

Don't know if somehting changed or this was just always semi-broken
(probable), but noticed today it wasn't adding any new favs when I knew
for a fact I had some; Perhaps, also, I have a lot more favs than when I
first wrote this.

Previously this assumed that the response returned all favs for a site
whereas actually by default it returns just 30. Perhaps by luck I'd been
picking up new ones because of using "sort=activity"; a strange choice
on it's own as if I'd used "sort=added" and run this fairly regularly
perhaps I'd never had noticed any issue? Anyway, now it does the correct
thing which is pull 100 at a time and look for the has_more parameter in
what is returned so it knows whether it needs to fetch another page.

This does mean get_favs now returns an array of items (makes more sense)
than the actual encompassing response as it used to. Leaf check-in: 09c6947036 user: base@atomicules.co.uk tags: master, trunk

2015-06-20
18:40
Add basic caching to speed up adding new bookmarks

Since have to (well should do, to be nice) rate limit quite a lot it can
be slow to add new bookmarks to Pinboard when we are checking Pinboard
for every single StackExchange fav. Keep a local cache of items already
added to Pinboard.

Some things I have to do (note to self):

- Check for success on Pinboard add before adding to cache
- Consider removing bookmarks on Pinboard for unfaved StackExchange
items? Leaf check-in: 29383c0dea user: base@atomicules.co.uk tags: origin/master, trunk

Changes
Hide Diffs Unified Diffs Ignore Whitespace Patch

Changes to stackexchange-favs-to-pinboard.rb.

︙︙
23
24
25
26
27
28
29





30
31
32






33
34
35
36
37
38
39

def parse(response_string)
	#Seems like Ruby automatically handles gzip compression now. No need to decompress.
	parsed = JSON.parse(response_string)
end


def get_favs(site, id)





	response = open("https://api.stackexchange.com/2.2/users/#{id}/favorites?order=desc&sort=activity&site=#{site}")
	#Use response.read not .string as some returns are large enough to go to a temp file
	parsed = parse(response.read)






end


class Pinboard
	@@logger = Logger.new(STDOUT)
	@@logger.level = Logger::INFO
	@@rate_limit = 3








>
>
>
>
>
|
|
|
>
>
>
>
>
>








23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50

def parse(response_string)
	#Seems like Ruby automatically handles gzip compression now. No need to decompress.
	parsed = JSON.parse(response_string)
end


def get_favs(site, id)
	puts "Fetching favs for #{site}"
	has_more = true
	items = []
	page = 1
	while has_more
		response = open("https://api.stackexchange.com/2.2/users/#{id}/favorites?order=desc&sort=added&pageSize=100&page=#{page}&site=#{site}")
		#Use response.read not .string as some returns are large enough to go to a temp file
		parsed = parse(response.read)
		items += parsed["items"]
		has_more = parsed["has_more"]
		page += 1
	end
	puts "#{items.length} favs"
	items
end


class Pinboard
	@@logger = Logger.new(STDOUT)
	@@logger.level = Logger::INFO
	@@rate_limit = 3
︙︙
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112

	if File.exists?(cache_file)
		cache = JSON.parse(File.read(cache_file))
	else
		cache = {}
	end
	parsed = get_sites(StackID)
	parsed["items"].each do |site|
		favs = get_favs(site["site_url"].sub("http://", "").sub(".stackexchange", "").sub(".com", ""), site["user_id"])
		#Don't make more than 30 requests per second
		sleep (1.0/30)
		favs["items"].each do |fav|
			title = fav["title"]
			tags = fav["tags"]
			link = fav["link"]
			#Check cache to see if already added to pinboard
			unless cache.has_key?(link)
				#Need to unescape so can re-escape in Pinboard code
				#Still want no default replace, just in case cache doesn't exist








|


|








106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123

	if File.exists?(cache_file)
		cache = JSON.parse(File.read(cache_file))
	else
		cache = {}
	end
	parsed = get_sites(StackID)
	parsed["items"].each do |site|
		favs = get_favs(site["site_url"].sub("http://", "").sub("https://", "").sub(".stackexchange", "").sub(".com", ""), site["user_id"])
		#Don't make more than 30 requests per second
		sleep (1.0/30)
		favs.each do |fav|
			title = fav["title"]
			tags = fav["tags"]
			link = fav["link"]
			#Check cache to see if already added to pinboard
			unless cache.has_key?(link)
				#Need to unescape so can re-escape in Pinboard code
				#Still want no default replace, just in case cache doesn't exist
︙︙

This page was generated in about 0.008s by Fossil 2.23 [47362306a7] 2023-11-01 18:56:47