Documentation / Dates and times guide

Back to contents PHP Python Ruby Choose a language:

An important part of scraping is turning string data into structured data. Two very common things this happens with are date and times.

For more details, read the Ruby Date docs, the Time docs, or this tutorial.

Parsing dates/times

The easiest way is to use a general purpose function that detects many common date formats, and converts them into a Ruby Date object.

puts Date.parse('21 June 2010') # 2010-06-21
puts Date.parse('10-Jul-1899')  # 1899-07-10
puts Date.parse('01/01/01')     # 2001-01-01

puts Date.parse('21 June 2010').class # Date

Or you can parse times as well, making a Ruby Time object. puts Time.parse('Tue 27 Sep 2011 00:25:48') # 2011-09-27 00:25:48 +0000 puts Time.parse('21 June 2010 6am').class # Time

Ambiguous cases

This sometimes goes wrong. For example, is this the 2nd March (US) or 3rd February (UK)?

puts Date.parse('3/2/1999') # 1999-02-03

You can fix it using a completely explicit format string.

puts Date.strptime('3/2/1999', '%m/%d/%Y') # 1999-03-02

Saving to the datestore

This is easy as pie. You just save either the Ruby date or datetime object, and ScraperWiki will convert it into the format SQLite needs.

birth_date = Date.parse('1/2/1997 9pm') 
birth_time = Time.parse('1/2/1997 9pm') 
data = { 
    :name => 'stilton', 
    :birth_datetime => birth_date,
    :birth_date => birth_time
}
ScraperWiki::save_sqlite(unique_keys=['name'], data=data)

Times are saved as UTC, as SQLite doesn't parse explicit timezones.

Querying dates

From the Web API for a scraper, you can do queries based on dates. See SQLite's date/time functions for more.

select * from swdata where birth_date < '2000-01-01'