I plan to take a Ruby on Rails scraping mooc , to get the most out of the course I am reading some tutorials on how to scrape and anticipate possible tasks.
I created a web page on github that simulates the data I want to extract:
https://rrg1459.github.io/extractData/
And I have the following helper called clientes_helper.rb
that extracts the data correctly:
module ClientesHelper
def consulta_cliente
require 'open-uri'
require 'nokogiri'
url = "https://rrg1459.github.io/extraerDatos/"
begin
@hayInternet = true
doc = Nokogiri.HTML(open(url))
inline_script = doc.search('//tr')
inline_script.each do |script|
linea = script.text.strip
linea = linea.gsub("\t","")
linea = linea.gsub("\n","")
linea = linea.gsub("</table>","")
linea = linea.gsub("<tr>","")
linea = linea.gsub("<td>","")
linea = linea.gsub("</td>","")
linea = linea.gsub("<b>","")
linea = linea.gsub("</b>","")
linea = linea.gsub("</tr>","")
linea = linea.gsub("<td align=\"left\">","")
linea = linea.gsub(" ","")
linea = linea.gsub("<font color=\"#00387b\">","")
if /ID/ =~ linea
@id_cliente = linea.split(':')[1]
elsif /Nombre/ =~ linea
@cliente_nombre = linea.split(':')[1]
elsif /stado/ =~ linea
@cliente_estado = linea.split(':')[1]
elsif /REGULAR/ =~ linea
@cliente_regular = linea.split(':')[1]
elsif /Dirección/ =~ linea
@direccion_cliente = linea.split(':')[1]
end
end
rescue
@hayInternet = false
end
end
end
But it looks horrible and it's not the best technique, I think it might be offensive to some experienced eyes and I apologize in advance.
I am quite aware of the Ruby on Rails principle of "convention over configuration" and I try to stick to this principle and fail to make it look clean, spare, precise.
And the truth is, I'm more tangled up than a dog eating gum.
Can someone guide me on what is the best practice to extract this data in an elegant Ruby-like way.
Thank you in advance for any help you can give me.
I think the most important improvement you can make to your code is to use nokogiri correctly.
By searching
//tr
you are returning and iterating through 14 tags that it found in your HTML, which is inefficient, since you iterate through more tags than you should and on top of that you have to remove all those line breaks and html tags (which by the way, you don't are necessary, since it.text
removes them).You'd be better off directly accessing the 4 tr's you need via their xpath :