Many hyperlinks are disabled.
Use anonymous login
to enable hyperlinks.
Overview
| Comment: | Make this actually work for lots of favs Don't know if somehting changed or this was just always semi-broken (probable), but noticed today it wasn't adding any new favs when I knew for a fact I had some; Perhaps, also, I have a lot more favs than when I first wrote this. Previously this assumed that the response returned all favs for a site whereas actually by default it returns just 30. Perhaps by luck I'd been picking up new ones because of using "sort=activity"; a strange choice on it's own as if I'd used "sort=added" and run this fairly regularly perhaps I'd never had noticed any issue? Anyway, now it does the correct thing which is pull 100 at a time and look for the has_more parameter in what is returned so it knows whether it needs to fetch another page. This does mean get_favs now returns an array of items (makes more sense) than the actual encompassing response as it used to. |
|---|---|
| Downloads: | Tarball | ZIP archive | SQL archive |
| Timelines: | family | ancestors | trunk | master |
| Files: | files | file ages | folders |
| SHA3-256: |
09c69470368aa60945a1a3d1d277eef3 |
| User & Date: | base@atomicules.co.uk 2017-03-12 14:54:59 |
Context
|
2017-03-12
| ||
| 14:54 | Make this actually work for lots of favs Don't know if somehting changed or this was just always semi-broken (probable), but noticed today it wasn't adding any new favs when I knew for a fact I had some; Perhaps, also, I have a lot more favs than when I first wrote this. Previously this assumed that the response returned all favs for a site whereas actually by default it returns just 30. Perhaps by luck I'd been picking up new ones because of using "sort=activity"; a strange choice on it's own as if I'd used "sort=added" and run this fairly regularly perhaps I'd never had noticed any issue? Anyway, now it does the correct thing which is pull 100 at a time and look for the has_more parameter in what is returned so it knows whether it needs to fetch another page. This does mean get_favs now returns an array of items (makes more sense) than the actual encompassing response as it used to. Leaf check-in: 09c6947036 user: base@atomicules.co.uk tags: trunk, master | |
|
2015-06-20
| ||
| 18:40 | Add basic caching to speed up adding new bookmarks Since have to (well should do, to be nice) rate limit quite a lot it can be slow to add new bookmarks to Pinboard when we are checking Pinboard for every single StackExchange fav. Keep a local cache of items already added to Pinboard. Some things I have to do (note to self): - Check for success on Pinboard add before adding to cache - Consider removing bookmarks on Pinboard for unfaved StackExchange items? Leaf check-in: 29383c0dea user: base@atomicules.co.uk tags: origin/master, trunk | |
Changes
Changes to stackexchange-favs-to-pinboard.rb.
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
..
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
|
def parse(response_string)
#Seems like Ruby automatically handles gzip compression now. No need to decompress.
parsed = JSON.parse(response_string)
end
def get_favs(site, id)
response = open("https://api.stackexchange.com/2.2/users/#{id}/favorites?order=desc&sort=activity&site=#{site}")
#Use response.read not .string as some returns are large enough to go to a temp file
parsed = parse(response.read)
end
class Pinboard
@@logger = Logger.new(STDOUT)
@@logger.level = Logger::INFO
@@rate_limit = 3
................................................................................
if File.exists?(cache_file)
cache = JSON.parse(File.read(cache_file))
else
cache = {}
end
parsed = get_sites(StackID)
parsed["items"].each do |site|
favs = get_favs(site["site_url"].sub("http://", "").sub(".stackexchange", "").sub(".com", ""), site["user_id"])
#Don't make more than 30 requests per second
sleep (1.0/30)
favs["items"].each do |fav|
title = fav["title"]
tags = fav["tags"]
link = fav["link"]
#Check cache to see if already added to pinboard
unless cache.has_key?(link)
#Need to unescape so can re-escape in Pinboard code
#Still want no default replace, just in case cache doesn't exist
|
>
>
>
>
>
|
|
|
>
>
>
>
>
>
|
|
|
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
...
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
|
def parse(response_string) #Seems like Ruby automatically handles gzip compression now. No need to decompress. parsed = JSON.parse(response_string) end def get_favs(site, id) puts "Fetching favs for #{site}" has_more = true items = [] page = 1 while has_more response = open("https://api.stackexchange.com/2.2/users/#{id}/favorites?order=desc&sort=added&pageSize=100&page=#{page}&site=#{site}") #Use response.read not .string as some returns are large enough to go to a temp file parsed = parse(response.read) items += parsed["items"] has_more = parsed["has_more"] page += 1 end puts "#{items.length} favs" items end class Pinboard @@logger = Logger.new(STDOUT) @@logger.level = Logger::INFO @@rate_limit = 3 ................................................................................ if File.exists?(cache_file) cache = JSON.parse(File.read(cache_file)) else cache = {} end parsed = get_sites(StackID) parsed["items"].each do |site| favs = get_favs(site["site_url"].sub("http://", "").sub("https://", "").sub(".stackexchange", "").sub(".com", ""), site["user_id"]) #Don't make more than 30 requests per second sleep (1.0/30) favs.each do |fav| title = fav["title"] tags = fav["tags"] link = fav["link"] #Check cache to see if already added to pinboard unless cache.has_key?(link) #Need to unescape so can re-escape in Pinboard code #Still want no default replace, just in case cache doesn't exist |