Smile Bank to Ledger

Check-in [4ca78e73c4]
Login

Many hyperlinks are disabled.
Use anonymous login to enable hyperlinks.

Overview
Comment:Convert to csv parsing only

Really I don't need all that class stuff in there anymore, but I thought I
would retain and re-use what I could. Now it simply parses the csv file that
Smile provide for download and converts that to Ledger format. Sob-sob - I
loved being able to logon on the command line and download transactions.

Downloads: Tarball | ZIP archive | SQL archive
Timelines: family | ancestors | descendants | both | trunk
Files: files | file ages | folders
SHA1: 4ca78e73c4d679ae810936d30baa5db134c3cf71
User & Date: atomicules 2016-10-02 12:30:20
Context
2016-10-16
21:46
Fix date filtering and order of transactions check-in: 9332fb989f user: atomicules tags: trunk
2016-10-02
12:30
Convert to csv parsing only

Really I don't need all that class stuff in there anymore, but I thought I
would retain and re-use what I could. Now it simply parses the csv file that
Smile provide for download and converts that to Ledger format. Sob-sob - I
loved being able to logon on the command line and download transactions. check-in: 4ca78e73c4 user: atomicules tags: trunk

2016-09-04
14:15
Last tweaks I'd made. Commit for reference

I'd forgotten to commit these. Committing for reference since none of this
works anymore. Only option is to use a webbrowser and download the csv. Will be
changing this into a CSV parser. Maybe. Might even just do that in Vim. check-in: ce95ff0dde user: atomicules tags: old-smile-site, trunk

Changes
Hide Diffs Unified Diffs Ignore Whitespace Patch

Changes to smile.rb.

1
2
3
4
5
6
7

8
9
10
11
12

13
14
15
16
17
18
19




20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134

135
136
137

138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
require 'optparse'
require 'mechanize'
require 'nokogiri'
require 'highline/import'

optparse = OptionParser.new do |opts|
	opts.on('-d', '--date DATE', "Date") { |d| Date_back = d }

end
optparse.parse!

class Smile
	attr_writer :date_back 

	attr_reader :transactions
	attr_reader :balance
	attr_reader :available

	def initialize
		@a = Mechanize.new
		@transactions = []




	end


	def transactions(date_back)
		@date_back = Date.parse(date_back)
		previous_statements
		#previous calls recent (I could change this though so it reads more sensibly here)
		write_ledger
	end


	#All these private
	def logon_start
		page = @a.get("https://banking.smile.co.uk/SmileWeb/start.do")
		#Since dealing with sensitive logon data don't want to get it from anywhere apart from keyboard input
		#Sortcode and Account number login
		sortCode = ask("Enter sortcode") { |a| a.echo = "*" }
		accountNumber = ask("Enter Account number") { |a| a.echo = "*" }
		page.form.sortCode = sortCode
		page.form.accountNumber = accountNumber
		page = page.form.submit
	end


	def logon_passcode
		#Requires
		page = logon_start
		#Passcode login
		doc = Nokogiri::HTML(page.body)
		page.form.firstPassCodeDigit = ask(doc.xpath("//label")[0].text) { |a| a.echo = "*" }
		page.form.secondPassCodeDigit = ask(doc.xpath("//label")[1].text) { |a| a.echo = "*" }
		page = page.form.submit
	end


	def logon_other
		#requires
		page = logon_passcode
		#Other logon details
		page.form.fields.each do |field|
			unless field.type == "hidden"
				field.value = ask(field.name) { |a| a.echo = "*" }
			end
		end
		page = page.form.submit
	end


	def logon_bulletin
		#requires
		page = logon_other
		if page.form
			if page.form.action.include?("bulletinBoard")
				page = page.form.submit
			end
		end
		#Need this to be the last thing so it gets returned if still logon_other
		page
	end


	def recent_transactions
		#requires
		page = logon_bulletin
		page = @a.click(page.link_with(:text => "current account"))
		get_transactions(page)
		doc = Nokogiri::HTML(page.body)
		#Any trailiing minus needs to be moved to before the number, but any trailing plus needs to be removed, don't think this can be done in one gsub.
		@balance = doc.xpath("//td[@class='recentTransactionsAccountData']/table/tr[2]/td[2]").text.strip.gsub(/(£)(\d+\.\d+)(\-)/, "\\1\\3\\2").gsub(/\+/, "")
		@available = doc.xpath("//td[@class='recentTransactionsAccountData']/table/tr[3]/td[2]").text.strip
		#Need to return this
		page
	end


	def previous_statements
		#requires
		page = recent_transactions
		#Previous statements
		page = @a.click(page.link_with(:text => "previous statements"))
		#Get first page of previous statements
		page = @a.click(page.links_with(:href => %r{getDomestic} )[0])
		date = Date.today
		until date < @date_back
			get_transactions(page)
			#On previous statement pages most recent transaction is at bottom
			#This is opposite to recent item page
			date = @transactions[-1][:date]
			page = @a.click(page.link_with(:text => "previous statement page"))
		end
	end


	def get_transactions(page)
		doc = Nokogiri::HTML(page.body)
		#Look for summaryTable, sometimes can be two, get the last
		rows = doc.xpath("//table[@class='summaryTable'][position()=last()]//tr")
		#Skip first row (table headers, ths)
		rows = rows[1..-1]
		transactions = []
		rows.each do |row|
			#Skip last row, maybe
			#Also, sometimes bugs or invalid markup and so even though last row does have just one td doesn't get reported that way
			#So use begin/rescue
			begin
				#Might as well parse date here
				date = Date.parse(row.elements[0].text.strip)
				name = row.elements[1].text.strip
				credit = row.elements[2].text.strip
				debit = row.elements[3].text.strip
				unless name.include?("BROUGHT FORWARD")
					transactions << { :date => date, :name => name, :credit => credit, :debit => debit}
				end
			rescue
			end

		end	
		#Sort in order of newest date first so can add on
		@transactions += transactions.sort { | t1, t2 | t2[:date] <=> t1[:date] }

	end


	def write_ledger
		balance_assertion = ""
		File.open("smile.dat", "w") do |file|
			#Reverse for writing out
			@transactions.reverse!
			@transactions.each.with_index do |transaction, idx|
				file << transaction[:date].strftime("%Y/%m/%d")+"\t*\t"+transaction[:name]+"\n"
				if transaction[:debit].include?("£")
					#Quote all commodities because having ledger utf-8 trouble
					amount = transaction[:debit].gsub(/£/, "\"£\"-")
					type = "Expenses:"
				else
					amount = transaction[:credit].gsub(/£/, "\"£\"")
					type = "Income:"
				end
				if idx == @transactions.length-1
					#Include balance assertion at the end.
					balance_assertion = "\t= "+@balance
					balance_assertion = balance_assertion.gsub(/£/, "\"£\"")
				end
				file << "\t"+"Assets:Smile:Current"+"\t"+amount+balance_assertion+"\n"
				file << "\t"+type+"\n"
				file << "\n"
			end
			#Write out account balance as a comment at the end
			file << "#Available: "+@available+"\n"
		end
	end

	private :logon_start
	private :logon_passcode
	private :logon_other
	private :logon_bulletin
	private :recent_transactions
	private :previous_statements
	private :write_ledger
end


if defined?(Date_back)
	smile = Smile.new
	smile.transactions(Date_back)
end

<
|
<



>





>


<

|
<

>
>
>
>



|
<
|
<




<
<
<
<
<
<
<
<
<
<
<
|
|
<
<
<
<
<
<
<
<
<
|
|
<
<
<
<
<
<
<
<
<
<
<
|
|
<
<
<
<
<
<
<
<
<
<
<
|
<
<
<
<
<
<
<
<
<
<
<
<
<
|
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
|
|
|
|

<

>
|
<
<
>










|

|


|




|
<





<
|



<
<
<
<
|
<




|
|
|

1

2

3
4
5
6
7
8
9
10
11
12
13
14

15
16

17
18
19
20
21
22
23
24
25

26

27
28
29
30











31
32









33
34











35
36











37













38


































39
40
41
42
43

44
45
46


47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68

69
70
71
72
73

74
75
76
77




78

79
80
81
82
83
84
85
86
require 'optparse'

require 'csv'


optparse = OptionParser.new do |opts|
	opts.on('-d', '--date DATE', "Date") { |d| Date_back = d }
	opts.on('-f', '--file FILE', "File") { |f| File_csv = f }
end
optparse.parse!

class Smile
	attr_writer :date_back 
	attr_writer :file_csv
	attr_reader :transactions
	attr_reader :balance


	def initialize args

		@transactions = []
		#http://stackoverflow.com/posts/36177200/revisions
		args.each do |k,v|
			instance_variable_set("@#{k}", v) unless v.nil?
		end
	end


	def transactions

		get_transactions

		write_ledger
	end













	def get_transactions
		csv = CSV.open(@file_csv, 'r')









		csv.reverse_each do |line|
			#Need to skip first/last line











			unless line[0] == "Date"
				@balance = line[5]











				date = Date.parse(line[0])













				name = line[1]


































				credit = line[3]
				debit = line[4]
				unless date > @date_back
					@transactions << { :date => date, :name => name, :credit => credit, :debit => debit}
				end

			end
			#No need to sort any more
		end


		csv.close
	end


	def write_ledger
		balance_assertion = ""
		File.open("smile.dat", "w") do |file|
			#Reverse for writing out
			@transactions.reverse!
			@transactions.each.with_index do |transaction, idx|
				file << transaction[:date].strftime("%Y/%m/%d")+"\t*\t"+transaction[:name]+"\n"
				if transaction[:debit]
					#Quote all commodities because having ledger utf-8 trouble
					amount = "\"£\"-"+transaction[:debit]
					type = "Expenses:"
				else
					amount = "\"£\""+transaction[:credit]
					type = "Income:"
				end
				if idx == @transactions.length-1
					#Include balance assertion at the end.
					balance_assertion = "\t= \"£\""+@balance

				end
				file << "\t"+"Assets:Smile:Current"+"\t"+amount+balance_assertion+"\n"
				file << "\t"+type+"\n"
				file << "\n"
			end

			#Can't get available balance any more
		end
	end





	private :get_transactions

	private :write_ledger
end


if defined?(Date_back) && defined?(File_csv)
	smile = Smile.new(:date_back => Date.parse(Date_back), :file_csv => File_csv)
	smile.transactions
end