Pulling stockdata from the Yahoo

GARP

I'm trying to get stock data from Yahoo. Wrote a couple of lines that seemed to work on the desktop. But does anybody knows if I'm doing it right and where I can find the .csv file with the pulled stock data? I don't get an error and can't find my data.

import os
import urllib
import csv

def pulldata(stock):
        urllib.urlretrieve('http://ichart.finance.yahoo.com/table.csv?s='+stock+'&
        a=07&b=25&c=2014&d=07&e=29&f=2014&g=d&ignore=.csv')
        return

pulldata('RUT')

SpotlightKid

Just pass the path of the output file as a second argument to urlretrieve(). The filename is also the first item of the tuple returned by this function.

import urllib

STOCK_URL = 'http://ichart.finance.yahoo.com/table.csv?s=%s&a=07&b=25&c=2014&d=07&e=29&f=2014&g=d&ignore=.csv'

def pulldata(stock, filename):
    return urllib.urlretrieve(STOCK_URL % urllib.quote(stock), filename)

filename, headers = pulldata("RUT", "RUT-data.csv")

If you want to save the file in your documents folder, build the filename like this (there are several ways to do this, but this one will work on desktop Python as well):

import os
from os.path import exists, dirname, join

# create output directory if it doesn't exist yet
documents = join(os.getenv('HOME'), 'Documents')
if not exists(documents):
    os.mkdir(documents)

filename = join(documents, 'RUT-data.csv')

ccc

Do you have to be logged into Yahoo for this to work?

When I ran it, RUT-data.csv was an HTML file that contained (on line 22):

<h1>Sorry, the page you requested was not found.</h1>

What am I doing wrong?

SpotlightKid

RUT doesn't seem to be a valid stock identifier. Try '^GDAXI' for the German DAX.

SpotlightKid

BTW, if you want to play with the URL parameters (params a-f can be used to set a date range), you can construct the URL params from a dictionary like so (note that the month number has to be minus one):

STOCK_URL = 'http://ichart.finance.yahoo.com/table.csv'
PARAMS = {
    'a': 9,
    'b': 1,
    'c': 2014,
    'd': 9,
    'e': 31,
    'f': 2014,
    'g': 'd',
    'ignore': '.csv'
}

def pulldata(stock, filename):
    params = PARAMS.copy()
    params['s'] = stock
    url = "%s?%s" % (STOCK_URL, urllib.urlencode(params))
    return urllib.urlretrieve(url, filename)

Here's the documentation of the API:

https://code.google.com/p/yahoo-finance-managed/wiki/csvHistQuotesDownload

GARP

Thanks guys for that quick response, worked like a charm,and sorry it is indeed ^RUT instead of RUT :-)

techteej

Expanding on @SpotlightKid 's example, this grabs the current date and brings up stock info for that month.

from datetime import datetime, date
import urllib

month = int(datetime.strftime(date.today(), "%m")) - 1
year = int(datetime.strftime(date.today(), "%Y"))

STOCK_URL = 'http://ichart.finance.yahoo.com/table.csv'
PARAMS = {
    'a': month,
    'b': 1,
    'c': year,
    'd': month,
    'e': 31,
    'f': year,
    'g': 'd',
    'ignore': '.csv'
}

def pulldata(stock, filename):
    params = PARAMS.copy()
    params['s'] = stock
    url = "%s?%s" % (STOCK_URL, urllib.urlencode(params))
    return urllib.urlretrieve(url, filename)

filename, headers = pulldata("^RUT", "RUT-data.csv")

SpotlightKid

Small suggestion:

from datetime import datetime, date

[...]

month = int(datetime.strftime(date.today(), "%m")) - 1
year = int(datetime.strftime(date.today(), "%Y"))

This can be just written as:

from datetime import date

month = date.today().month - 1
year = date.today().year

No need to format the month/year number into a string and then into an integer again.

techteej

@SpotlightKid whoops! Copied from another program and forgot to take that out. Thanks!

techteej

How would I tweak this to show data for a certain stock? Like APPL for example?

ccc

Change the last line to: filename, headers = pulldata("AAPL", "AAPL-data.csv")

Exersize for the reader: Where is the "January Bug" in the code above and how would you fix it?

techteej

Can't find the January bug , and would like to print the name of the stock if possible

ccc

Hint 1:

stock_dict = { 'AAPL' : 'Apple, Inc.',
               'GOOG' : 'Google',
               'HPQ'  : 'Hewlett-Packard Company',
               'IBM'  : 'Internationa Business Machines Corp.' }

for s in 'AAPL GOOG HPQ IBM COKE'.split():
    print(stock_dict.get(s, 'Unknown'))

Hint 2: You will only see the January Bug when the current month is January... In six weeks it will become clear.

JonB

Ok, I'm stumped ccc. Since python dates use a 1-based month, (January is 1), the above will result in the variable month==0, which is how yahoo wants the data (yahoo uses 0 based month, yet 1 based day for whatever reason).
Manually creating a date object in January and using the above code works fine.

There is one bug and one quirk that I see:
The bug happens st the stroke of midnight on 12/31, in which case the first call to today, to get the month, will return December, but the next call to today, to get the year, will return next year. Thus you won't pull any data (yahoo will return html rather than csv when the dates are in the future)

The quirk is that techteejs proposal of pulling data from the start of the current month is not how most people look at stock data.... Last 30 days, sure, but on the first of the month you might not pull any data if the market is not open( for example, market is always closed on January first, so of you ran the script on January first, yahoo would return a file not found error html)

Here's a version that pulls last 30 days, and only calls today() once.

from datetime import datetime, date, timedelta
import urllib

enddate = date.today()
startdate = enddate + timedelta(-30)

STOCK_URL = 'http://ichart.finance.yahoo.com/table.csv'
PARAMS = {
    'a': startdate.month-1,
    'b': startdate.day,
    'c': startdate.year,
    'd': enddate.month-1,
    'e': enddate.day,
    'f': enddate.year,
    'g': 'd',
    'ignore': '.csv'
}

def pulldata(stock, filename):
    params = PARAMS.copy()
    params['s'] = stock
    url = "%s?%s" % (STOCK_URL, urllib.urlencode(params))
    return urllib.urlretrieve(url, filename)

filename, headers = pulldata("^RUT", "RUT-data.csv")

ccc

Ahhh... You have it right and I had it wrong. Your analysis and code above is the correct approach.

techteej

@JonB originally I had planned to have it retrieve last months data if ran on the first and it was a weekend.

@ccc Any way to put the stock name in the .csv? I would like to make this for more than one stock.

SpotlightKid

How about putting the month in the filename:

filename = "data-%s-%02i.csv" % (stock.replace('^', ''), startdate.month)