Python and Quantopian Tools¶
Reasons to use Python¶
- Speed
- Python is much faster
- Big Data
- Python tools for blocked/parallel computing is more well regarded, and older than R alternatives
- Blaze/Dask
- pyspark
- sklearn (partial fit)
- Python tools for blocked/parallel computing is more well regarded, and older than R alternatives
Reasons to use Python¶
- Packages
- NLTK
- Web Scraping
- beautifulsoup, Scrapy, Selenium
- Deep Learning
- Quantopian (other backtesting frameworks)
- Implement into production (app/software)
Speed¶
- Python's data science tools are better optimized for speed.
- Numpy arrays are the building blocks for most of python's data science tools, and numpy arrays are optimized for speed.
- Pandas provides high-level, easy-to-use, data manipulation tools built off of numpy.
Speed Testing¶
- Comparing Python and R using the "Million Songs" dataset
- Tidyverse vs Pandas/Numpy Aggregate
- Random Forest in R vs sklearn
Data¶
- 438 MB
- Response is the year of the song release
- 90 features taken from the audio
- Added an id column for the purpose of comparing pandas and dplyr performance
In [4]:
songs.shape
Out[4]:
In [5]:
songs.head(10)
Out[5]:
Aggregation - Mean¶
songs %>%
group_by(id) %>%
summarise_all(funs(mean(., na.rm = TRUE))) %>%
select(-id)
On my system using a single system.time call, execution time of 1.25 seconds.
In [6]:
%%timeit
(songs
.groupby("id")
.mean()
.reset_index(drop = True))
Aggregation - Median¶
songs %>%
group_by(id) %>%
summarise_all(funs(median(., na.rm = TRUE))) %>%
select(-id)
Execution time was 497.5 seconds (8 minutes)
In [7]:
%%timeit
(songs
.groupby("id")
.median()
.reset_index(drop = True))
Aggregation - Custom¶
songs %>%
group_by(id) %>%
summarise_all(funs(sum(. > 0))) %>%
select(-id)
Execution time was 185.8 seconds (2 minutes)
In [8]:
%%timeit -r 3
(songs
.groupby('id')
.apply(lambda df: (df > 0).sum())
.reset_index(drop = True))
Additional gains can be made by converting to numpy, and letting numpy handle the calculations.
In [9]:
def number_positive(df):
numpy_df = df.values
numpy_df = numpy_df > 0
counts = numpy_df.sum(axis = 0)
pandas_series = pd.Series(counts)
return(pandas_series)
In [10]:
%%timeit -r 3
(songs
.groupby('id')
.apply(number_positive)
.reset_index(drop = True))
Running a model¶
What do all good data scientists do when they first want to analyze a dataset?
Blindly run a random forest on it!
In [11]:
songs = songs.groupby("id").mean().reset_index(drop = True)
In [12]:
songs.shape
Out[12]:
Random Forest¶
fit <- randomForest(X, y, ntree=10)
Execution time of 525.8 seconds (9 minutes)
In [15]:
X = songs.iloc[:, range(1, 91)]
y = songs[0]
rf = RandomForestRegressor(n_estimators=10, n_jobs = 1, max_features = "sqrt", min_samples_leaf = 5)
%timeit -r 3 rf.fit(X, y)
Quantopian Tools¶
- Alphalens
- Zipline
- Pyfolio
Alphalens¶
Analyzes how predictive a single feature is at predicting stock returns.
The core functionality of alphalens is as follows.
For each day:
- Split candidate stocks into groups by the feature (quantiles)
- Calculates 1, 5, and 10 day returns for each group
Then aggregates over all days
Zipline¶
- First quantopian tool
- Algorithmic trading simulator
- Used for backtesting a trading strategy/algorithm
Zipline - Reasons to use¶
- Well tested and thought out platform
- Can automatically handle slippage/transaction costs
- Calculates many important metrics out of the box
- Beta
- Measures of variability (Sharpe/Sortino)
- Integrates with Pyfolio
Pyfolio¶
Pretty tables and plots
Simple example¶
In [4]:
from zipline import run_algorithm
from zipline.api import symbol, get_datetime, order_target_percent
import pyfolio
import pandas as pd
Our simple strategy is to long Apple on even days of the month, and short it on odd days.
In [2]:
def initialize(context):
pass
def handle_data(context, data):
day_of_month = get_datetime().day
if (int(day_of_month) % 2 == 0):
order_target_percent(symbol("AAPL"), 1)
else:
order_target_percent(symbol("AAPL"), -1)
In [5]:
result = run_algorithm(start = pd.Timestamp("2017-01-01", tz = "UTC"),
end = pd.Timestamp("2017-06-30", tz = "UTC"),
initialize = initialize,
capital_base = 1000000,
handle_data = handle_data,
data_frequency = "daily",
bundle = "quantopian-quandl")
In [6]:
result.positions.head()
Out[6]:
In [8]:
result.ending_cash.head(5)
Out[8]:
In [9]:
pf_data = pyfolio.utils.extract_rets_pos_txn_from_zipline(result)
pyfolio.create_full_tear_sheet(*pf_data)
Vardon Backtest¶
In [10]:
def initialize(context):
context.transactions = picks
z.set_commission(f.commission.PerShare(cost=0, min_trade_cost=0))
z.set_slippage(f.slippage.FixedSlippage(spread=0.0))
# Scheduling function to run one minute before close will result in it executing at close
z.schedule_function(func = order_stocks, time_rule = z.time_rules.market_close(minutes = 1),
half_days=True)
def order_stocks(context, data):
# Get date that zipline is on in the backtest
current_date = (z.get_datetime()
.tz_localize(None)
.normalize())
transactions = context.transactions.loc[current_date]
# Place order for each transaction
for row in transactions.itertuples(index=True, name='Pandas'):
z.order(z.symbol(row[1]), row[2])