stackexchange-favs-to-pinboard

Check-in [c8517a3a50]
Login
Home Timeline Branches Tags Tickets Wiki

Many hyperlinks are disabled.
Use anonymous login to enable hyperlinks.

Overview
Comment:Improvements to Pinboard code. Now a class, uses json and logger

I used the Pinboard code from this script as a basis for some Jekyll
syndication code I'm putting together elsewhere. I then made
changes/improvements to it which I've decided to port back over here. I
guess eventually I'll end up making my own Pinboard gem. Maybe.

Some corresponding changes to the script were necessary such as command
line arguments for Pinboard username and token, and re-ordering of method
arguments.

Downloads: Tarball | ZIP archive | SQL archive
Timelines: family | ancestors | descendants | both | origin/master | trunk
Files: files | file ages | folders
SHA3-256: c8517a3a50d3885bbedff65aa8d172139364a2430657efe5b89f278563c4c0ff
User & Date: base@atomicules.co.uk 2015-04-06 10:00:34
Context
2015-04-06
10:14
Updates for StackExchange API and Ruby 2.X

- As a result of previous commit, discovered that Ruby open-uri now
seems to automatically decompress Gzip data so parsing of responses is
more simple now.
- Also, update the API version
- Use response.read instead of string as some responses are too long for
string and get written to a tempfile
- Ensure we are making no more than 30 requests/sec check-in: cb04e6d1bf user: base@atomicules.co.uk tags: origin/master, trunk

10:00
Improvements to Pinboard code. Now a class, uses json and logger

I used the Pinboard code from this script as a basis for some Jekyll
syndication code I'm putting together elsewhere. I then made
changes/improvements to it which I've decided to port back over here. I
guess eventually I'll end up making my own Pinboard gem. Maybe.

Some corresponding changes to the script were necessary such as command
line arguments for Pinboard username and token, and re-ordering of method
arguments. check-in: c8517a3a50 user: base@atomicules.co.uk tags: origin/master, trunk

2013-02-17
23:08
Typical, spot something I missed as soon as I push check-in: d45e2bb088 user: atomicules@lavabit.com tags: origin/master, trunk
Changes
Hide Diffs Unified Diffs Ignore Whitespace Patch

Changes to stackexchange-favs-to-pinboard.rb. 

1
2
3
4
5


6
7
8

9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31

32
33
34
35
36


37

38
39


40





















41
42
43
44
45
46
47
48
49
50
51
52
53
54

55
56
57
58

59
60
61
62
63
64
65
66
67
68
69

70
71
72

#StackExchange Favs to Pinboard
require 'open-uri'
require 'json'
require 'optparse'
require 'cgi'



optparse = OptionParser.new do |opts|
	opts.on('-i', '--id ID', "Stackexchange ID") { |i| StackID = i }

	opts.on('-t', '--token TOKEN', "Pinboard API Token") { |t| Token = t }
end
optparse.parse!

def get_sites(id)
	response = open("https://api.stackexchange.com/2.1/users/#{id}/associated")
	parsed = parse(response)
end

def parse(response) 
	#From: Garth, http://stackoverflow.com/a/1366187/208793
	gz = Zlib::GzipReader.new(StringIO.new(response.string))
	parsed = JSON.parse(gz.read)
end

def get_favs(site, id)
	response = open("https://api.stackexchange.com/2.1/users/#{id}/favorites?order=desc&sort=activity&site=#{site}")
	parsed = parse(response)
end

def pinboard_add(auth_token, url, description, replace, tags)
	attempts = 1
	posted = false

	until ($rate_limit > 60) | (attempts > 3) | posted 
		response = open(URI.encode("https://api.pinboard.in/v1/posts/add?auth_token=#{auth_token}&url=#{url}&description=#{description}&replace=#{replace}&tags=#{tags}").gsub("'", "%27"))
		#Bit of a hacky kludge. For some reason URI.encode doesn't catch apostrophes.
		if (response.status[0] == "200") & response.string.include?("done")
			puts "Added #{url}"


			posted = true

		elsif (response.status[0] == "200") & response.string.include?("exists")
			puts "Skipping #{url}, already exists"


			posted = true





















		elsif response.status[0] == "429"
			# 429 Too Many Requests, increase rate limit
			$rate_limit *= 2
			puts "Rate Limit increased to #{$rate_limit} seconds"
		end
		attempts += 1
		#Rate limit as per Pinboard API requirements
		sleep $rate_limit
	end
	if $rate_limit > 60
		puts "Rate limit has exceeded 60 secs, let's try again another time"
		quit
	elsif attempts > 3
		puts "Failed 3 times to save #{url}, bombing out"

		quit
	end
end


if defined?(StackID) and defined?(Token)
	$rate_limit = 3
	parsed = get_sites(StackID)
	parsed["items"].each do |site|
		favs = get_favs(site["site_url"].sub("http://", "").sub(".stackexchange", "").sub(".com", ""), site["user_id"])
		favs["items"].each do |fav|
			title = fav["title"]
			tags = fav["tags"]
			link = fav["link"]
			pinboard_add(Token, link, CGI.unescape_html(title), "no", tags.join(", ")+", stackexchangefavs")
			#Need to unescape so can URI encode

		end
	end
end






>
>



>




















|
|
|
>
|
|
|
|
<
>
>
|
>
|
|
>
>
|
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
|
|
|
|
|
|
|
|
|
|
|
<
|
|
>
|



>
|
|







<
|
>




1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39

40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80

81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97

98
99
100
101
102

#StackExchange Favs to Pinboard
require 'open-uri'
require 'json'
require 'optparse'
require 'cgi'
require 'logger'


optparse = OptionParser.new do |opts|
	opts.on('-i', '--id ID', "Stackexchange ID") { |i| StackID = i }
	opts.on('-u', '--user USER', "Pinboard username") { |u| User = u }
	opts.on('-t', '--token TOKEN', "Pinboard API Token") { |t| Token = t }
end
optparse.parse!

def get_sites(id)
	response = open("https://api.stackexchange.com/2.1/users/#{id}/associated")
	parsed = parse(response)
end

def parse(response) 
	#From: Garth, http://stackoverflow.com/a/1366187/208793
	gz = Zlib::GzipReader.new(StringIO.new(response.string))
	parsed = JSON.parse(gz.read)
end

def get_favs(site, id)
	response = open("https://api.stackexchange.com/2.1/users/#{id}/favorites?order=desc&sort=activity&site=#{site}")
	parsed = parse(response)
end


class Pinboard
	@@logger = Logger.new(STDOUT)
	@@logger.level = Logger::INFO
	@@rate_limit = 3
	Api_url = "https://api.pinboard.in/v1/"



	def initialize(user, token)
		@user = user
		@token = token
	end


	def add(url, description, extended=nil, tags=nil, replace="no")
		attempts = 1
		posted = false
		#At minimum must have url and description
		array_parameters = "&url=#{CGI.escape(url)}&description=#{CGI.escape(description)}"
		#Could loop through the below
		unless extended.nil?
			array_parameters += "&extended=#{CGI.escape(extended)}"
		end
		unless tags.nil?
			#TODO: Need to check whether tags_escaped will work
			array_parameters += "&tags=#{CGI.escape(tags)}"
		end
		until (@@rate_limit > 60) | (attempts > 3) | posted 
			response = open("#{Api_url}posts/add?auth_token=#{@user}:#{@token}"+array_parameters+"&replace=#{replace}&format=json")
			@@logger.debug(response.string)
			@@logger.debug(response.status)
			response_json = JSON.parse(response.string)
			if (response.status[0] == "200") & (response_json["result_code"] == "done")
				@@logger.info("Added #{url}")
				posted = true
			elsif (response.status[0] == "200") & (response_json["result_code"] == "item already exists")
				@@logger.info("Skipping #{url}, already exists")
				posted = true
			elsif response.status[0] == "429"
				# 429 Too Many Requests, increase rate limit
				@@rate_limit *= 2
				@@logger.warn("Rate Limit increased to #{$rate_limit} seconds")
			end
			attempts += 1
			#Rate limit as per Pinboard API requirements
			sleep @@rate_limit
		end
		if @@rate_limit > 60
			@@logger.error("Rate limit has exceeded 60 secs, let's try again another time")

		elsif attempts > 3
			@@logger.error("Failed 3 times to save #{url}, bombing out")
		end
		posted
	end
end


if defined?(StackID) and defined?(User) and defined?(Token)
	pb = Pinboard.new(User, Token)
	parsed = get_sites(StackID)
	parsed["items"].each do |site|
		favs = get_favs(site["site_url"].sub("http://", "").sub(".stackexchange", "").sub(".com", ""), site["user_id"])
		favs["items"].each do |fav|
			title = fav["title"]
			tags = fav["tags"]
			link = fav["link"]

			#Need to unescape so can re-escape in Pinboard code
			pb.add(link, CGI.unescape_html(title), nil, tags.join(", ")+", stackexchangefavs", "no")
		end
	end
end

This page was generated in about 0.012s by Fossil 2.23 [47362306a7] 2023-11-01 18:56:47