Daniel E. Cook

A tool for writing TILs (Today I Learned)

Wed, 22 Apr 2020 01:15:53 +0000

There is a great repo on GitHub of TILs. The author (jbranchaud) states:

A collection of concise write-ups on small things I learn day to day across a variety of languages and technologies. These are things that don’t really warrant a full blog post.

I think this is a pretty cool idea, but putting together the repo with the index can take time and interrupt your workflow. I wanted to make it very quick and easy to add TILs, so I wrote TIL-Tool. TIL-tool is a command line application invoked using til that makes it very easy to write TILs and generate an index.

To create a new TIL, run til open topic/title:

til open Python/list_comprehensions

This will open up a new text document which you can edit. Once you are done you can save the document and til. If you have configured a git remote (e.g. on GitHub), then you can then run til push which will build an index and push changes.

All TILs are stored in ~/.til

TIL-Tool on GitHub
TIL Example Repo

You can walk to In-N-Out from LAX

Thu, 30 Jan 2020 00:26:29 +0100

Don’t believe everything you read on the internet. There is an In-N-Out just outside of LAX on Sepulveda Blvd which I had thought was walkable. I was discouraged after reading this thread which suggests the walk to In-N-Out from LAX is not possible during a layover. Turns out, on a decent layover (~2.5 hours), it is totally possible!

We gave it a go because I am a big fan of In-N-Out. Go to the lower-level (arrivals) and head east. Turn left on Sky Way and follow it until you hit the stairs down to S. Sepulveda Blvd. Then head North and cross once you see In-N-Out on the left. It is not he most glamorous of walks, but a great way to get out of the airport. It is about a mile each way (~22 min).

Parallelize by iterating over chromosomal ranges

Wed, 29 Jan 2020 01:15:53 +0000

I have added a new utility to seq-collection called iter which generates chromosomal ranges. Lists of genomic ranges can be easily plugged into utilities such as xargs or gnu-parallel to parallelize commands.

For example:

sc iter test.bam 100,000 # Iterate on bins of 100k base pairs

# Outputs
> I:0-999999
> I:1000000-1999999
> I:2000000-2999999
> I:3000000-3999999
> I:4000000-4999999

Note: BAMs use a 0-based coordinate system; VCFs are 1-based

This list of genomic ranges can be used to process a BAM or VCF in parallel:


function process_chunk {
  # Code to process chunk
  vcf=$1
  region=$2
  # e.g. bcftools call -m --region 
  echo bcftools call --region $region $vcf # ...
}

# Export the function to make it available to GNU parallel
export -f process_chunk

parallel --verbose process_chunk ::: test.bam ::: $(sc iter test.bam)

You can also set the [width] option to 0 to generate a list of chromosomes.

See Using GNU-Parallel for Bioinformatics for a comprehensive guide on using Parallel for bioinformatics.

seq-collection (sc) is a set of tools written in nim and using the fantastic hts-nim package.

Calculate Insert Size Metrics Faster

Wed, 29 Jan 2020 00:00:00 +0000

Picard tools is a great set of utilities by the Broad Institute for performing sequence analysis. however, some of the utilities run on the slower side.

To speed things up, I created a new command: insert-size as part of seq-collection. The command runs much faster, owing in part to parallelization of insert-size calculations.

insert-size does not operate in exactly the same way as picard CollectInsertSizeMetrics, but the results are very close.

insert-size has some nice advantages over picard. The output is a lot more interpretable and parsable than standard picard output.

For example, if you run:

sc insert-size --basename --header tests/data/test.bam

The outputted table will be:

median	mean	std_dev	min	percentile_99.5	max_all	n_reads	n_accept	n_use	sample	basename
179	176.5	63.954	38	358	359	237	101	100	AB1	test.bam

You can also output the distribution of insert-sizes by count by specifying the --dist=<filename> argument.

seq-collection (sc) is a set of tools written in nim and using the fantastic hts-nim package.

From Pandas to Google Sheets

Fri, 25 Oct 2019 01:15:53 +0000

I wrote the following snippet to post datasets (e.g. TSVs or CSVs) to google sheets. In order to get this to work you will need to authorize google sheets access.

Then you can set the content of any google sheets worksheet to the data from a pandas dataframe by using the pandas_to_sheets function.

#!/usr/bin/env python
import gspread
import pandas as pd
from oauth2client.service_account import ServiceAccountCredentials

def iter_pd(df):
    for val in df.columns:
        yield val
    for row in df.to_numpy():
        for val in row:
            if pd.isna(val):
                yield ""
            else:
                yield val

def pandas_to_sheets(pandas_df, sheet, clear = True):
    # Updates all values in a workbook to match a pandas dataframe
    if clear:
        sheet.clear()
    (row, col) = pandas_df.shape
    cells = sheet.range("A1:{}".format(gspread.utils.rowcol_to_a1(row + 1, col)))
    for cell, val in zip(cells, iter_pd(pandas_df)):
        cell.value = val
    sheet.update_cells(cells)

scope = ['https://spreadsheets.google.com/feeds',
         'https://www.googleapis.com/auth/drive']

credentials = ServiceAccountCredentials.from_json_keyfile_name('service.json', scope)

gc = gspread.authorize(credentials)

workbook = gc.open_by_key("<workbook id>")
sheet = workbook.worksheet("worksheet_name")

df = pd.read_csv("input_data.tsv")
pandas_to_sheets(df, workbook.worksheet("worksheet"))

Speeding up Reading and Writing in R

Sun, 20 Oct 2019 01:30:53 +0000

If you are relying on built-in functions to read and write large datasets you are losing out on efficiency and speed gains available through external packages in R. Below, I benchmark some of the options out there used for reading and writing files.

Sample Data

First, I’ll generate a sample dataset with ten million rows we can use for testing.

Generating a test dataset

library(tidyverse)
library(microbenchmark)
n <- 1e6
times <- 10 # Number of times to run each benchmark
data <- data.frame(a = runif(n),
                   b = sample(1:1000, n, T),
                   c = sample(month.name, n, T),
                   d = sample(LETTERS, n, T))

write.table(data, "data.tsv", quote = F, row.names = F)

Here are the first few rows of that dataset:

	a	b	c	d
1	0.1926477	789	August	R
2	0.8303095	156	March	D
3	0.1144189	742	July	P
4	0.2828960	337	April	S
5	0.2861664	43	November	W

Reading TSVs

Base R has some pretty slow functions for reading files that also are poorly designed (row numbers and quotes by default, issues reading column names with special characters, etc.). Lets see how they compare with more up to date packages.

vroom vs readr vs base R vs data.table

Below I use microbenchmark to compare the following methods for reading this 1M row dataset:

base::read.table
base::read.delim
readr::read_tsv
vroom::vroom
data.table::fread
tbl_df(data.table::fread) - This converts the data.table to a tibble::tbl_df object which is the type of data structure readr and vroom return and is what is used in the tidyverse.

These functions will output data in either a data.frame, tibble, or data.table.

bm <- microbenchmark(
  `base::read.table` = read.table("data.tsv"),
  `base::read.delim` = read.delim("data.tsv"),
  `readr::read_tsv` = readr::read_tsv("data.tsv"),
  `vroom ~ 1 thread` = vroom::vroom("data.tsv", num_threads = 1),
  `vroom ~ 8 threads` = vroom::vroom("data.tsv"),
  `tbl_df(data.table::fread) ~ 1 thread` = tbl_df(data.table::fread("data.tsv", nThread = 1)),
  `tbl_df(data.table::fread) ~ 8 threads` = tbl_df(data.table::fread("data.tsv")),
  `data.table::fread ~ 1 thread` = data.table::fread("data.tsv", nThread = 1),
  `data.table::fread ~ 8 threads` = data.table::fread("data.tsv"),
  times = times
)
autoplot(bm) + 
  labs(caption = glue::glue("{scales::comma(n)} rows; {times} times"))

Looks like the base R functions lose - by a lot. data.table::fread and vroom::vroom come out on top at ~ 100 milleseconds whereas the base functions take ~10 seconds or 100x longer!

Stop wasting your time with read.table, read.csv, and read.delim and move to something quicker like data.table::fread, or vroom::vroom both of which perform much faster. Both can also take advantage of multiple cores but outperform base R even when they only use a single thread!

Writing TSVs

vroom vs readr vs data.table vs base R

Next I compared methods for writing TSV files. Base R has the functions write.csv and write.table for writing delimited text files. Unfortunately, these too have poor defaults (quoting strings, adding rownames). I have turned these off for the comparison.

bm <- microbenchmark(
    `base::write.table` = write.table(data, file = "out.tsv", quote=F, sep = "\t"),
    `readr::write_tsv` = readr::write_tsv(data, "out.tsv"),
    `readr::write_tsv + gz` = readr::write_tsv(data, "out.tsv.gz"),
    `data.table::fwrite ~ 1 thread` = data.table::fwrite(data, "out.tsv", nThread = 1),
    `data.table::fwrite ~ 8 threads` = data.table::fwrite(data, "out.tsv", nThread = 8),
    `vroom::vroom_write ~ 1 thread` = vroom::vroom_write(data, "out.tsv", num_threads = 1),
    `vroom::vroom_write ~ 8 threads` = vroom::vroom_write(data, "out.tsv"),
    `vroom::vroom_write ~ 1 thread + gz` = vroom::vroom_write(data, "out.tsv.gz", num_threads = 1),
    `vroom::vroom_write ~ 8 threads + gz` = vroom::vroom_write(data, "out.tsv.gz"),
    times = times
)
autoplot(bm)

data.table::fwrite performs the fastest in multi-threaded mode with vroom::vroom not far behind. These are ~100x faster than base R. Apparently, applying gzip compression slows things down considerably but can save a lot of space.

Serializing Data

Serialized data formats retain column types and avoid data loss that may occur when writing and reading TSVs. Here I compare:

feather::write_feather
fst::write_fst
base::save
base::saveRDS

Note that these serialization formats each provide other benefits that should be considered. For example, feather files are a good interchange format between R and python using the python Pandas module.

bm <- microbenchmark(
   `base::save`=save(data, file = "out.Rda"),
   `saveRDS` = saveRDS(data, file = "out.rds"),
   `fst::write_fst` = fst::write_fst(data, path = "out.fst"),
   `feather::write_feather` = feather::write_feather(data, path = "out.feather"),
   times = times
)
autoplot(bm)

fst and feather perform about the same and again, about ~100x better than base R.

Reading Serialized Data

bm <- microbenchmark(
   `base::load`=load(file = "out.Rda"),
   `readRDS` = readRDS(file = "out.rds"),
   `fst::read_fst` = fst::read_fst(path = "out.fst"),
   `feather::read_feather` = feather::read_feather(path = "out.feather"),
   times = times
)
autoplot(bm)

fst reads the quickest but feather is not too far behind. These functions are about ~10x better than base R in this comparison.

Using GNU-Parallel for bioinformatics

Fri, 27 Sep 2019 01:30:53 +0000

GNU Parallel is an indispensible tool for speeding up bioinformatics. It allows you to easily parallelize commands. Below, I detail some of the basics regarding how it is used and how it can be applied to bioinformatics.

Many HPC clusters will have GNU-Parallel pre-installed or available as a module. You can also install it using homebrew or other package managers.

Basic Usage

Lets start with a basic example:

seq 1 5 | parallel -j 4 echo

Here we are (1) Printing a sequence of numbers from 1 to 5, and (2) piping this data into parallel. We have provided the command echo which will be parallelized across -j=4 jobs. We can see what this looks like by using the --dry-run flag which prints the commands to be run.

seq 1 5 | parallel --dry-run -j 4 echo

echo 3
echo 4
echo 5
echo 2
echo 1

The results are out of order! This is due to the nature of parallelization. Not all “jobs” initiate or take the same amount of time so it is common to observe outputs in a different order. We can enforce a “first in first out” result set by using the -k flag. Lets see what the output looks like by removing the --dry-run flag:

seq 1 5 | parallel -j 4 -k echo

Like any other command, you can send this output to a file:

seq 1 5 | parallel -j 4 -k echo > out.txt

`-j`

In order for GNU Parllel to work, you want to have a multi-core CPU. Parallelizing across more cores than you have available can actually make performance worse, so it is important to tune the -j parameter to the number of cores available.

Luckily, parallel allows you to specify -j using a percentage of cores or as a number relative to the total number of cores. For example:

parallel -j 100% # Uses 100% of cores.
parallel -j -1 # Uses 1 less than the total number of cores.
parallel -j +1 # Parallelize across the number of cores + 1

Use `:::` for args

Use ::: to specify arguments derived from commands or lists.

parallel -j 4 -k echo ::: `seq 1 5`

Note - There are limits to the number of arguments you can provide with a process substitution as shown above. In these instances, it may be better to pipe arguments or use a file (below) rather than supply them with a process substitution:

seq 1 5 | parallel -j 4 -k echo

Use `::::` for args within files

For large argument lists you can specify a file with a list of arguments. Specify a file of arguments (one per line) using ::::.

parallel -j 4 -k echo :::: my_args.txt

Use `

By default, parallel assumes the arguments are placed at the end of the input command, but you can explicitly define where arguments are substituted using {}:

parallel --dry-run -j 4 -k echo \"{} "<-- a number"\" ::: `seq 1 5`

echo "1 <-- a number"
echo "2 <-- a number"
echo "3 <-- a number"
echo "4 <-- a number"
echo "5 <-- a number"

Notice that we are having to escape quotes - there are ways around this.

Combinatorials

You can keep adding ::: and :::: to add additional arguments, and these will be combined to generate all possible combinations. This is extremely useful for testing commands with different combinations of input parameters.

parallel --dry-run -k -j 4 Rscript run_analysis.R {1} {2} ::: `seq 1 2` ::: A B C

Rscript run_analysis.R 1 A
Rscript run_analysis.R 1 B
Rscript run_analysis.R 1 C
Rscript run_analysis.R 2 A
Rscript run_analysis.R 2 B
Rscript run_analysis.R 2 C

Parallelize Functions

In some cases, you want to perform a series of commands. For example, the code below compute the number of ATCGs of the complement of a DNA sequence.

echo "ATTA" |  tr ATCG TAGC | \
    python -c "import sys; o=sys.stdin.read().strip(); print(o, o.count('T'), o.count('G'), o.count('C'), o.count('A'))"

This command has two operations. While is it possible to incorporate this into a ‘one-liner’, it is far easier to create a bash function, export it, and use that as input.

function count_nts {
    # $1 is the first argument passed to the function
    echo $1 | tr ATCG TAGC | \
    python -c "import sys; o=sys.stdin.read().strip(); print(o, o.count('T'), o.count('G'), o.count('C'), o.count('A'))"
}

# Use the `-f` flag to export functions
export -f count_nts

parallel -j 4 count_nts ::: TAAT TTT AAAAT GCGCAT | tr ' ' '\t'

With the basics down, lets see how we can use parallel to speed up bioinformatics.

GNU Parallel for Variant Calling

When working with BAMs or VCFs you can parallelize across chromosomes. Most variant callers or annotation tools allow you to operate on a single chromosome at a time by specifying a region. This allows us to apply a split-apply-combine strategy by splitting by chromosome, operating on each chromosome, and combining the results at the end.

Split chromosomes from a BAM


chrom_list=`samtools idxstats in.bam | cut -f 1 | grep -v '*'`

# For c. elegans you can would see the following 7
# I
# II
# III
# IV
# V
# X
# MtDNA

We can create a function so this operation is easier going forward:

function bam_chromosomes {
    samtools idxstats $1 | cut -f 1 | grep -v '*'
}

Apply an operation to each chromosome

Here is where GNU parallel comes into play: Parallelized variant calling by chromosome:

#!/bin/bash

genome=path/to/genome.fa
export genome # This is critical!

function parallel_call {
    bcftools mpileup \
        --fasta-ref ${genome} \
        --regions $2 \
        --output-type u \
        $1 | \
    bcftools call --multiallelic-caller \
                  --variants-only \
                  --output-type u - > ${1/.bam/}.$2.bcf
}

export -f parallel_call

chrom_set=`bam_chromosomes test.bam`
parallel --verbose -j 4 parallel_call sample_A.bam ::: ${chrom_set}

A few important notes regarding this step:

You must export any variables you use within a parallelized function. That is what I am doing here with the reference genome variable.
bcftools mpileup outputs an uncompressed pileup (--output-type=u). This is done for efficiency sake - there is no reason to pipe a compressed form of data for it to need to be uncompressed by the next tool.
Similarly, I also output an uncompressed set of variant calls ${1/.bam/}.$2.bcf because these are temporary files that we will remove later.
Finally, I use a variable substitution to remove the extension from the bam and to generate a <sample>.<chromsome>.bcf filename: ${1/.bam/}.$2.bcf → sample_A.I.bam, sample_A.II.bam, etc. This prevents filename collisions if we are calling many samples simultaneously.

Combine the variant calls.

Once we have completed variant calling we need to combine everything back in the right order. We can use a bash array to add a prefix and suffix to the list of chromosomes to reconstruct the output filenames and concatenate them into a single file.

# Generate an array of the resulting files
# to be concatenated.
sample_name="sample_A"
set -- `echo $chrom_set | tr "\n" " "`
set -- "${@/#/${sample_name}.}" && set -- "${@/%/.bcf}"
# This will generate a list of the output files:
# sample_A.I.bcf sample_B.II.bcf etc.

set -- "${@/#/test.}" && set -- "${@/%/.bcf}"

# Output compressed result
bcftools concat $@ --output-type b > $sample_name.bcf

# Remove intermediate files
rm $@

To ensure the intermediate files are removed even when errors occur you should use a bash trap.

Summary

GNU Parallel can greatly speed up simple parallelization scenerios. Additional code is often required to handle the “splitting” and “combining” steps, but this can allow for tremendous efficiency gains.

Converting VCF To JSON

Sun, 22 Sep 2019 01:15:53 +0000

Recently I started developing a set of utilities called seq-collection (sc) written in nim and using the fantastic hts-nim package.

The first utility I added was a tool to convert a VCF to JSON. This tool is useful for building out an API that reads genotype data directly from the VCF format. It is possible to read specific variants or intervals of VCF files when they are indexed, allowing for fast and efficient querying of genetic data without the need for a database. Furthermore, these queries can be made over http connections making it possible to use a VCF file as a database.

For example, as a graduate student I developed a genome browser for C. elegans wild isolates. Queries are made directly on specific genomic intervals of VCF files using bcftools. However, a large amount of python code is used to convert VCF output ot JSON format. The sc utility could replace this underlying python code on the server with a simple binary and a few command line arguments to accomplish the same task. Below I have a few examples illustrating how to use sc json.

Installation

You can download the MAC OSX binary here.

Or build from source at danielecook/seq-collection

Usage

json

Convert a VCF to JSON

Usage:
  json [options] vcf [region ...]

Arguments:
  vcf              VCF to convert to JSON
  [region ...]     List of regions

Options:
  -i, --info=INFO            comma-delimited INFO fields; Use 'ALL' for everything
  -f, --format=FORMAT        comma-delimited FORMAT fields; Use 'ALL' for everything
  -s, --samples=SAMPLES      Set Samples (default: ALL)
  -p, --pretty               Prettify result
  -a, --array                Output as a JSON array instead of individual JSON lines
  -z, --zip                  Zip sample names with FORMAT fields (e.g. {'sample1': 25, 'sample2': 34})
  -n, --annotation           Parse ANN Fields
  --pass                     Only output variants where FILTER=PASS
  --debug                    Debug
  -h, --help                 Show this help

Examples

List all sites

sc json tests/data/test.vcf.gz

Output

{"CHROM":"I","POS":41947,"ID":".","REF":"A","ALT":["T"],"QUAL":999.0,"FILTER":["PASS"]}
{"CHROM":"I","POS":105133,"ID":".","REF":"G","ALT":["A"],"QUAL":999.0,"FILTER":["PASS"]}
{"CHROM":"I","POS":176422,"ID":".","REF":"A","ALT":["G"],"QUAL":999.0,"FILTER":["PASS"]}

pretty output

We can “prettify” this output using the --pretty flag:

sc json --pretty tests/data/test.vcf.gz

{
  "CHROM": "I",
  "POS": 41947,
  "ID": ".",
  "REF": "A",
  "ALT": [
    "T"
  ],
  "QUAL": 999.0,
  "FILTER": [
    "PASS"
  ]
}

Fetch Genotypes

Next we can output genotype calls by specifying the --FORMAT flag with a GT argument:

> sc json --FORMAT=GT tests/data/test.vcf.gz

{"CHROM":"I","POS":41947,"ID":null,"REF":"A","ALT":["T"],"QUAL":999.0,"FILTER":["PASS"],"FORMAT":{"GT":[[0,0],[0,0],[0,0],[0,0],[0,0],[0,0],[0,0],[0,0],[0,0],[0,0],[0,0],[0,0],[0,0],[0,0]]}}
{"CHROM":"I","POS":105133,"ID":null,"REF":"G","ALT":["A"],"QUAL":999.0,"FILTER":["PASS"],"FORMAT":{"GT":[[0,0],[0,0],[0,0],[0,0],[0,0],[0,0],[0,0],[0,0],[0,0],[0,0],[0,0],[0,0],[0,0],[1,1]]}}
{"CHROM":"I","POS":176422,"ID":null,"REF":"A","ALT":["G"],"QUAL":999.0,"FILTER":["PASS"],"FORMAT":{"GT":[[0,0],[0,0],[0,0],[0,0],[0,0],[0,0],[1,1],[0,0],[0,0],[0,0],[0,0],[1,1],[1,1],[null,null]]}}

The genotypes are ordered by sample, and the numbers correspond as follows as follows:

-1 → Missing
0 → Reference
1 → First ALT allele
2 → Second ALT allele
3 → etc.

You can also use SGT to outut a string representation of genotypes (e.g. “0/1”). It is also possible to use TGT to output the actual bases:

 sc json --format=TGT tests/data/test.vcf.gz

{"CHROM":"I","POS":41947,"ID":null,"REF":"A","ALT":["T"],"QUAL":999.0,"FILTER":["PASS"],"FORMAT":{"TGT":["A/A","A/A","A/A","A/A","A/A","A/A","A/A","A/A","A/A","A/A","A/A","A/A","A/A","A/A"]}}
{"CHROM":"I","POS":105133,"ID":null,"REF":"G","ALT":["A"],"QUAL":999.0,"FILTER":["PASS"],"FORMAT":{"TGT":["G/G","G/G","G/G","G/G","G/G","G/G","G/G","G/G","G/G","G/G","G/G","G/G","G/G","A/A"]}}
{"CHROM":"I","POS":176422,"ID":null,"REF":"A","ALT":["G"],"QUAL":999.0,"FILTER":["PASS"],"FORMAT":{"TGT":["A/A","A/A","A/A","A/A","A/A","A/A","G/G","A/A","A/A","A/A","A/A","G/G","G/G","./."]}}

Setting the working directory in R

Tue, 09 Jul 2019 00:26:29 +0100

It is convenient to be able to set the working directory of a script to its parent directory. This allows you to point to the relative path of files associated with it. For example, if your working directory is set to the location of init.sh, then you will be able to read in data/file.dat without specifying its full path. If these files are in a git repo - you can also be assured they will travel together.

├── init.sh
└── data
    └── file.dat

In bash you can set the directory to the location of the script being executed using:

cd "${0%/*}"

# Or more obviously
cd "$(dirname "${BASH_SOURCE[0]}")"

In python you can set the directory to the location of the script being executed using:

import os
from os.path import dirname, abspath

os.chdir(dirname(abspath(__file__)))

In R, ~~unfortunately, no straightforward method exists~~ Update 2020-06-29 - there is a way to do this; see the 2020-06-29 update below. The update below demonstrates how to get the current directory a script is located in, followed by additional ways of setting the working directory based on the git repo or with Rstudio.

Update: 2020-06-29 - A way to get the script directory in R

There is a way to set the working directory to a script location, as shown here, you can get the location of a script using the function below.

LocationOfThisScript = function() # Function LocationOfThisScript returns the location of this .R script (may be needed to source other files in same dir)
{
	this.file = NULL
	# This file may be 'sourced'
	for (i in -(1:sys.nframe())) {
		if (identical(sys.function(i), base::source)) this.file = (normalizePath(sys.frame(i)$ofile))
	}

	if (!is.null(this.file)) return(dirname(this.file))

	# But it may also be called from the command line
	cmd.args = commandArgs(trailingOnly = FALSE)
	cmd.args.trailing = commandArgs(trailingOnly = TRUE)
	cmd.args = cmd.args[seq.int(from=1, length.out=length(cmd.args) - length(cmd.args.trailing))]
	res = gsub("^(?:--file=(.*)|.*)$", "\\1", cmd.args)

	# If multiple --file arguments are given, R uses the last one
	res = tail(res[res != ""], 1)
	if (0 < length(res)) return(dirname(res))

	# Both are not the case. Maybe we are in an R GUI?
	return(NULL)
}
current.dir = LocationOfThisScript()

setwd to script location when calling `Rscript`

getScriptPath <- function(){
    cmd.args <- commandArgs()
    m <- regexpr("(?<=^--file=).+", cmd.args, perl=TRUE)
    script.dir <- dirname(regmatches(cmd.args, m))
    if(length(script.dir) == 0) stop("can't determine script dir: please call the script with Rscript")
    if(length(script.dir) > 1) stop("can't determine script dir: more than one '--file' argument detected")
    return(script.dir)
}

# Setting the script path would then be:
setwd(getScriptPath())

setwd to script location in Rstudio

# This will not throw an error if you are not using rstudio.
try(setwd(dirname(rstudioapi::getActiveDocumentContext()$path)))

Setting the script path relative to the git repo

Most of my code resides in git repositories - so an alternative to setting the working directory to the location of a script is to set it to the location of a git repository. Here is how I do that in R:

# First set the working directory to the location of a script (useful for working in Rstudio)
try(setwd(dirname(rstudioapi::getActiveDocumentContext()$path)))
# Next, set the directory relative to the git repo
setwd(system("git rev-parse --show-toplevel", intern=T))

The big advantage of this approach is that you can call the script from anywhere while you are located in its git repo, and it will always execute from the base or top-level of that repo.

A bash alias for Microsoft Excel (Mac only)

Fri, 28 Jun 2019 00:26:29 +0100

Years ago I wrote a function for opening excel from R. While I would never use Excel for data analysis, it turns out it’s pretty good for sorting and browsing data. Thats why I wrote a simple bash alias for opening up text documents from the terminal.

function excel() {
    tmp=`mktemp`
    out=${1}
    cat ${out} > $tmp
    open -a "Microsoft Excel" $tmp
}

Usage:

cat spreadsheet.tsv | excel

Useful Nextflow bash functions for SLURM

Fri, 21 Jun 2019 00:00:00 +0000

If you use Nextflow on a cluster with the SLURM scheduler, then these bash functions may be useful to you and worth sticking in your .bashrc.

# Shortcut for going to work directories
# Usage: gw <workdir pattern>
# Replace the work directory below as needed
# Where workdir pattern is something like "ab/afedeu"
function gw {
        path=`ls --color=none -d /path/to/work/directory/$1*`
        cd $path
}

# sq squeue alternative
# Outputs more complete information about jobs including the work directory
function sq() {
    squeue --user `whoami` --format='%.18i %50j %10u %.10C %m %20J %M %.2t %n %R %Z' | awk -v OFS='\t' '{ match($10, /([a-f0-9]{2}\/[a-f0-9]{6})/, arr); print $1, $2, $3, $4, $5, $6, $7, $8, $9, arr[1] }' 
}

Now, typing sq will give:

JOBID	NAME	USER	CPUS	MIN_MEMORY	THREADS_PER_CORE	TIME	ST	REQ_NODES
17475076	nf-fastq	cookd	8	12G	*	0:00	PD	(Priority)	04/fbbe3e
17475077	nf-fastq	cookd	8	12G	*	0:00	PD	(Priority)	c9/9176eb
17475078	nf-fastq	cookd	8	12G	*	0:00	PD	(Priority)	80/6a233a

And you can type gw c9/9176eb which will take you to the work directory for that job.

gist

Log commands to Google Cloud Stackdriver Logs

Fri, 15 Dec 2017 00:00:00 +0000

Google Cloud Platform (GCP) has a service called Stackdriver logging which provides a nice interface for accessing logs.

Stackdriver logging is integrated with all GCP services but it can also be extended. Users can create custom logs and access them centrally using the web-based interface or the Google Cloud SDK.

This got me wondering whether there was a way to log terminal commands locally or on a server. It is possible by setting the PROMPT_COMMAND variable in BASH. After a command is submitted the value of PROMPT_COMMAND is interpretted (technically it is interpretted before the next prompt is printed to the screen).

I wrote up a quick function that looks to see whether the last command exited successfully (0) or resulted in an error (>0), and log using INFO or ERROR respectively. Then I set the function to the PROMPT_COMMAND variable. Note that you may need to activate the gcloud beta logging for this to work.

function prompt {
if [[ $? -eq 0 ]];then
    (gcloud beta logging write bash_log "`fc -nl -1`" --severity=INFO > /dev/null 2>&1 &)
else
    (gcloud beta logging write bash_log "`fc -nl -1`" --severity=ERROR > /dev/null 2>&1 &)
fi
}

PROMPT_COMMAND=prompt

Now check the logging interface and you will see your commands are logged!

Python Command-line skeleton

Thu, 02 Feb 2017 00:00:00 +0000

Writing a command-line interface (CLI) is an easy way to extend the functionality and ease of use of any code you write.

Python comes with the built-in module, argparse, that can be used to easily develop command-line interfaces. To speed up the process, I have developed a ‘skeleton’ application that can be forked on github and used to quickly develop CLI programs in python.

The repo has the following features added:

Testing with travis-ci and py.test
Coverage analysis using coveralls
A setup file that will install the command
a simple argparse interface

To get started, you should signup for an account on travis-ci and coveralls, and fork the repo!

python-cli-skeleton on Github

Introducing a Chicago Bioinformatics Slack Channel

Tue, 31 Jan 2017 00:00:00 +0000

Today I am introducing a new slack team for bioinformatians in Chicago.

Signup for the Chicago Bioinformatics Slack Channel!

Currently anyone with an email at the following domains can signup:

@northwestern.edu
@uchicago.edu
@uic.edu
@depaul.edu
@luc.edu
@iit.edu

Members can invite anyone. I am happy to add any Chicago-area domains. Please let me know which ones I am missing!

The slack team features channels for bioinformatics-help, general, introductions, meetups, and random currently. We can add more channels!

Alfred Image Utilities

Sun, 15 Jan 2017 00:00:00 +0000

A workflow for making quick changes to image files. Alfred-image-utilities grabs any selected images in the frontmost finder window and can apply changes to them. Most of the time a copy of the image is made and its extension is changed to <filename>.orig.<ext>. You can replace the original file by holding command when executing most commands.

Download

Main Menu

Convert to png or jpg

You can convert from a large number of formats to these jpg or png. The original file is retained unless you hold command.

Scale images by a maximum width/height, by percent, or generate thumbnails.

Hold command to replace original. This option is not available when generating thumbnails. Generating thumbnails will add a .thumb to the filename (<filename>.thumb.<ext>)

Rotate images (clockwise)

Hold command to replace original.

Convert images to black and white.

Hold command to replace original.

rdatastore

Thu, 15 Dec 2016 00:00:00 +0000

I’ve developed a new package for R known as rdatastore that is avaliable at cloudyr/rdatastore. rdatastore provides an interface for Google Cloud’s datastore service. Google Cloud Datastore is a NoSQL database, which makes provides a mechanism for storing and retrieving heterogeneous data. Although Google Datastore is not useful for storing large datasets, it has a number of useful applications within R. For example:

Saving and loading credentials for use with other services.
Caching data. This is implemented using datastore in my version of the memoise package.
Saving/loading universally used pieces of data (e.g. parameters, options, settings) across systems or between work/home.
Storage and retrieval of small (<10,000 row) datasets. Useful for integration of summary datasets.

The last two reasons are the primary motivation for developing rdatastore. Parallelized pipelines can simultaneously submit results to datastore (across many nodes or machines), and the results are obtainable for analysis within R. Settings can be updated on one machine and retrieved on others as well, obviating the need to modify virtual machines or scripts in many cases.

__The datastore interface can be used to view and edit data.__

Setup

Setup a Google Cloud Platform and create a new project.
Download the Google Cloud SDK. This provides a command line based gcloud command.
Install rdatastore

devtools::install_github("cloudyr/rdatastore")

Usage

Authentication

library(rdatastore)
authenticate_datastore("andersen-lab") # Enter your project ID here. rdatastore will authenticate using Oauth.

Storing Data

commit()

Individual entitites can be stored using commit(). You have to supply a kind (which is analogous to a table in relational database systems). You may optionally submit a name. Any additional arguments supplied are added as properties. Datatypes are inferred from R datatypes. For example:

commit(kind = "Car", name = "Tesla", wheels = 4) # Stores a new entity named 'Tesla'

Result

kind	name	wheels
car	Tesla	4

Important! Stick with basic datatypes like character vectors, integers, doubles, binary, and datetime objects. Not all datatypes are supported.

I designed rdatastore to make it easier to append data rather than overwrite it. This is abit against the grain as far as other datastore libraries go. For example:

commit(kind = "Car", name = "Tesla", electric = TRUE) # Stores a new entity named 'Tesla'

The entity will now be:

kind	name	wheels	electric
Car	Tesla	4	TRUE

If you want to overwrite the entity, you can use keep_existing = FALSE, and the original data will be wiped and replaced.

When using commit() you can omit the name parameter in which case Google datastore will autogenerate an ID for the entity. I’m not sure where this is useful. You won’t be able to look the item up without knowing its ID or by performing a query on the entities data.

lookup()

Retrieve data by specifying its kind and name.

lookup("Car", "Tesla")

kind	name	wheels	electric
Car	Tesla	4	TRUE

gql()

You can query items using the Google Query Language (GQL). GQL is a lot like SQL.

# Lets commit a few more items
commit("Car", "VW", electric = FALSE)
commit("Car", "Honda", make = "Odyssey", wheels = 4)
commit("Car", "Reliant", make = "Robin", wheels = 3)

gql("SELECT * FROM Car")

kind	name	make	wheels	electric
Car	Honda	Odyssey	4	NA
Car	Reliant	Robin	3	NA
Car	Tesla	NA	4	TRUE
Car	VW	NA	NA	FALSE

Notice that some some properties are NA because they were never specified.

We can also query specific properties - but this will only return entitites with those properties defined.

gql("SELECT make FROM Car")

kind	name	make
Car	Honda	Odyssey
Car	Reliant	Robin

You can also filter on properties with GQL:

gql("SELECT * FROM Car WHERE wheels = 3")

kind	name	make	wheels
Car	Reliant	Robin	3

A big list of favorites

Tue, 29 Nov 2016 00:00:00 +0000

Here it is! My favorite things in life across all domains. This is a work in progress, but hopefully you’ll find one (or a few) things you like and add it to your life. It’s a bit sparse currently, but it will fill in over time.

Note: There are no referral links here and I am not being paid to advertise anything here. Any companies/products listed have earned it.

Programming/Software

Terminal

Homebrew - A phenomenal package manage. Use with homebrew/science!
Autojump - Jump among directories by typing their j and their name. Even works if you type it incorrectly! Install with brew install autojump.
pyenv - An easy way to manage multiple installations of python, and set which versions to open globally, locally, or by directory. Install with brew install pyenv.
direnv - Set the terminal environment in a .envrc file based on the current directory.

Software (OSX)

Dropbox - The best backup/syncing solution. I pay for a subscription. I’ve used Box and Google Drive as well. Both are inferior.
Transmit - FTP Client.
Sublime Text - The best text editor. Extend its functionality with package control.
Alfred - Like spotlight but with a lot more functionality. Workflows extend its functionality considerably. See the ones I’ve written!
Sequal Pro - A MYSQL GUI. The best GUI for a database I’ve seen.

Programming

Github - A great place to work on projects using git.
Python - My favorite all-purpose programming language.

R

Below I list some of my favorite R packages.

Cowplot - Arrange plots in a grid Vignette.

Python

Jekyll - Static sites
Flask - Python framework
peewee - A very easy to use ORM for simple projects.

Future

Personal

iOS Apps

Reeder - A great RSS reader.
Strava - Fitness tracker. I’ve used Runkeeper in the past. It’s worth switching. If you want to switch fitness apps and not lose everything, check out tapiriik.

Websites

News

New York Times
Washington Post

Tech News

news.ycombinator.com

Financial

Vanguard - Retirement accounts and investing.

Blogs

Chicago Architecture Blog - Follow Chicago Architecture.

Services

Pinboard - Simple bookmarks.
tapiriik - Sync fitness data across fitness tracking services.

Biking

Quad Lock - The best phone mount.

Kitchen

Coffee

Bodum French Press - I use this every single day.
Coldbrew press - For iced coffee during the summer.

Wikipedia Pages

These are mostly a roundup of interesting ideas/facts/concepts I have come across.

Guitar Printouts

Wed, 17 Aug 2016 00:00:00 +0000

I put these guitar-related printouts (A chord diagram sheet and a fretboard diagram sheet) together years ago:

Quiver-alfred

Thu, 04 Aug 2016 00:00:00 +0000

Search Quiver from Alfred! Quiver-alfred quickly constructs a database of your notes for fast and easy querying.

Download

Usage

Type qset to set your quiver library location. Quiver-Alfred constructs a database of your notes to make querying as fast as possible. The database should refresh once every hour and should only take a few seconds to create.

Type q to use!

You can search tags by hitting q #.

Browse Notes within notebook:

Full Text Search using sqlite:

memoise: Caching in the cloud

Wed, 27 Jul 2016 00:00:00 +0000

Update: 2019-06-22

Based on my suggestions, out-of-memory caching was implemented in the “official” memoise package here. The memoise package now caches based on files and AWS.

Original Post

Memoisation is a technique for caching the results of functions based on inputs. For example, the following function calculates the fibonnaci sequence in R.

fib <- function(n) {
  if (n < 2) return(1)
  fib(n - 2) + fib(n - 1)
}

This is an innefficient way of calculating values of the fibonnacci sequence. However, it is a useful example for understanding memoisation. The following code uses Hadley Wickhams package memoise.

library(memoise)

# fib() generates the nth element of the fibonnaci seqeuence
fib <- function(n) {
  if (n < 2) return(1)
  fib(n - 2) + fib(n - 1)
}

# Memoize fib
mem_fib <- memoise(fib)

mem_fib(30) # Initial run caches the value

In the above example, the memoise() function generates a memoised function, which will automatically cache results. If the function is run again with the same parameters, it will return the cached result rather than recompute the result. Implementing memoisation can significantly speed up analysis when functions that take time to run are repeatedly called.

What if you are running similar analyses within a cluster environment? The ability to cache results in a centralized datastore could increase the speed of analysis across all machines. Alternatively, perhaps you work on different computers at work and at home. Forgetting to save/load intermediate files may require long-running functions to be run again. Further, managing and retaining intermediate files can be cumbersome and annoying. Again, caching the results of the memoised function in a central location (e.g., cloud-based storage) can speed up analytical pipelines across machines.

Recently I’ve put some work into developing new types of out-of-memory caches for the memoise package available here. This forked version can be used to cache items locally or remotely in a variety of environments. Supported environments include:

R environment (cache_local)
Google Datastore (cache_datastore)
Amazon S3 (cache_aws_s3)
File system (cache_filesystem; allows dropbox, google drive to be used for caching)

There are a few caveats to consider when using this version of memoise. If you use the external cache options, it will take additional time to retrieve cached items. This is preferable in cluster environments where syncing files across instances/nodes can be difficult. However, when working at home/work, using locally synced files is preferred.

Installation

devtools::install_github("danielecook/memoise")

Usage

Google Datastore

library(memoise)

# fib() generates the nth element of the fibonnaci seqeuence
fib <- function(n) {
  if (n < 2) return(1)
  fib(n - 2) + fib(n - 1)
}

# Define a cache
ds <- cache_datastore(project = "your-project-name", cache_name = "rcache2")

# Memoize fib
mem_fib <- memoise(fib, cache = ds)

mem_fib(30) # Initial run caches the value

Amazon S3

library(memoise)

# fib() generates the nth element of the fibonnaci seqeuence
fib <- function(n) {
  if (n < 2) return(1)
  fib(n - 2) + fib(n - 1)
}

# Set up credentials
Sys.setenv("AWS_ACCESS_KEY_ID" = "<access key>",
           "AWS_SECRET_ACCESS_KEY" = "<access secret>")

# Define a cache
# Your bucket name must be unique among all s3 users - so use something like 'rcache-<initials>'
aws_s3 <- cache_s3("<unique bucket name>")

# Memoize fib
mem_fib <- memoise(fib, cache = aws_s3)

mem_fib(30) # Initial run caches the value

Automatically construct / infer / sense bigquery schema

Wed, 30 Dec 2015 00:00:00 +0000

Update: BigQuery adds schema auto-detection (2019-06-22)

BigQuery now offers a schema auto-detection features making the work I had done below no longer necessary.

Original Post

BigQuery is a phenomenal tool for analyzing large datasets. It enables you to upload large datasets and perform sophisticated SQL queries on millions of rows in seconds. Moreover, it can be integrated with R using BigRQuery, which can be used to interact with bigquery using some of the functions in dplyr.

It is easy to upload datasets to bigquery, although it requires you to specify a schema. If you have a lot of columns in a dataset this can be a pain to do manually - so I wrote a script to automate the process. The script automatically determines the variable types within the first 500 rows of a tab-delimited dataset. To get started, download the python script below and save it as schema.py.

import mimetypes
import sys
from collections import OrderedDict

filename = sys.argv[1]

def file_type(filename):
    type = mimetypes.guess_type(filename)
    return type

filetype = file_type(filename)[1]
if filetype == "gzip":
    import gzip
    readfile = gzip.GzipFile(filename, 'r')
else:
    readfile = open(filename,'r')

with readfile as f:
    header = next(f).strip().split("\t")
    lines = [dict(zip(header,next(f).strip().split("\t"))) for x in xrange(50000)]

schema = OrderedDict(zip(header, [bool]*len(header)))
def boolify(s):
    if s == 'True' or s == "TRUE" or s == "T":
        return True
    if s == 'False' or s == "FALSE" or s == "F":
        return False
    raise ValueError("huh?")

def autoconvert(s):
    for fn in (boolify, int, float):
        try:
            return fn(s)
        except ValueError:
            pass
    return s

type_precedence = {str:0, float:1, int:2,bool:3}
type_map = {str:"STRING", float:"FLOAT", int:"INTEGER", bool:"BOOLEAN"}

# Sense header
for line in lines:
    for k,v in line.items():
        if v == "" or v == ".":
            pass
        else:
            sense_type = type(autoconvert(v))
            if schema[k] == sense_type or schema[k] == str:
                pass
            elif type_precedence[schema[k]] > type_precedence[sense_type]:
                schema[k] = sense_type

print ','.join([ k.replace("/","_") + ":" + type_map[v] for k,v in schema.items()])

Usage

Save the gist as a script and run it as follows:

python schema.py <file>

The script supports plain text and gzipped files (which bigquery can load).

Output Example

CHROM:STRING,POS:INTEGER,REF_Original:STRING,ALT_Change:STRING,avg_cover:FLOAT,spikein_snvfrac:FLOAT,maxfrac:FLOAT,in_spikein:BOOLEAN,in_varset:BOOLEAN

Note that support for RECORD and TIMESTAMP fieldtypes is not supported.

Parallelize bcftools functions

Sat, 21 Nov 2015 00:00:00 +0000

bcftools is a great for working with variant call files. In general, it is fast. However, I have found that the process of merging VCF files (using bcftools merge) and performing concordance checking (using bcftools gtcheck) can be a little bit slow. That is why I wrote two functions that take advantage of GNU Parallel to parallelize them.

# ~/.bashrc: executed by bash(1) for non-login shells.
# see /usr/share/doc/bash/examples/startup-files (in the package bash-doc)
# for examples

function bam_chromosomes() {
    # Fetch chromosomes from a bam file
    samtools view -H $1 | \
    grep -Po 'SN:(.*)\t' | \
    cut -c 4-1000
}

function vcf_chromosomes() {
    # Fetch contigs from a vcf file.
    bcftools view -h $1 | \
    grep 'contig' | \
    egrep -o "ID=([^,]+)" | \
    sed 's/ID=//g' | \
    tr '\n' ' '
}


PARALLEL_CORES=6
function parallel_bcftools_merge() {
    file_set=`echo $@ | egrep -o '(\-l|\-\-file-list)(=|[ ]+)[^ ]+' | tr '=' ' ' | cut -f 2 -d ' '`
    if [ -n "${file_set}" ]
        then
            find_vcf=`cat ${file_set} | head -n 1`
        else
            find_vcf=`echo $@ | tr '\t' '\n' | egrep -o '[^ ]+.vcf.gz' | awk 'NR == 1 { print }' - `
    fi
    contigs=`vcf_chromosomes ${find_vcf}`
    current_dir=$(dirname ${find_vcf})
    hash_merge=`echo "$@" | md5sum | cut -c 1-5`
    output_prefix="${current_dir}/parallel_merge.${hash_merge}"
    
    parallel --gnu --workdir ${current_dir} \
    --env args -j ${PARALLEL_CORES} \
    'bcftools merge -r {1} -O u ' $@ ' > ' ${output_prefix}'.{1}.bcf' ::: ${contigs} 
    
    order=`echo $contigs | tr ' ' '\n' | awk -v "prefix=${output_prefix}" '{ print prefix "." $0 ".bcf" }'`
    bcftools concat -O v ${order} | grep -v 'parallel_merge' | sed 's/##bcftools_mergeCommand=merge -r I -O u /##bcftools_mergeCommand=merge /g' | bcftools view -O u
    rm ${order}
}


PARALLEL_CORES=6
function parallel_bcftools_gtcheck() {
    find_vcf=`echo $@ | tr '\t' '\n' | egrep -o '[^ ]+.vcf.gz' | awk 'NR == 1 { print }' - `
    contigs=`vcf_chromosomes ${find_vcf}`
    current_dir=$(dirname ${find_vcf})
    hash_merge=`echo "$@" | md5sum | cut -c 1-5`
    output_prefix="${current_dir}/parallel_concordance.${hash_merge}"
    gtcheck_options=`echo $@ | awk -v vcf=${find_vcf} '{ gsub(vcf,"",$0); print $0; }'`
    parallel --gnu  -j ${PARALLEL_CORES} --workdir ${current_dir} 'bcftools view ' ${find_vcf} ' {} | \
    bcftools gtcheck ' ${gtcheck_options} ' - | \
    awk -v chrom={} "/^CN/ { print \$0 \"\t\" chrom } \$0 !~ /.*CN.*/ { print } \$0 ~ /^# \[1\]CN/ { print \$0 \"\tchrom\"}" - > ' ${output_prefix}'.{}.tsv' ::: ${contigs}

    order=`echo $contigs | tr ' ' '\n' | awk -v "prefix=${output_prefix}" '{ print prefix "." $0 ".tsv" }'`
    cat ${order} | \
    grep 'CN' | \
    awk 'NR == 1 && /Discordance/ { print } NR > 1 && $0 !~ /Discordance/ { print }' | \
    awk '{ gsub("(# |\\[[0-9]+\\])","", $0); gsub(" ","_", $0); print }' | \
    cut -f 2-7 | datamash --header-in --sort --group=Sample_i,Sample_j sum Discordance  sum Number_of_sites | \
    cat <(echo -e "sample_i\tsample_j\tdiscordance\tnumber_of_sites") - | \
    awk 'NR == 1 { print $0 "\tconcordance" } NR > 1 && $4 == 0 { print $0 "\t" } NR > 1 && $4 > 0 { print $0 "\t" ($4-$3)/$4 }'
    rm ${order}
}

Usage

The function vcf_chromosomes extracts chromosomes names from a VCF file using bcftools. Parallelization occurs across chromosomes.

`parallel_bcftools_merge`

parallel_bcftools_merge is run very similar to bcftools merge. The only difference is that you have to pipe it into bcftools to change it to the appropriate output. For example:

parallel_bcftools_merge -m all `ls *list_of_bcffiles` | bcftools view -O z > merged_vcf.vcf.gz

The parallel_bcftools_merge function will generate a temporary vcf for every chromosome. You can use all flags except for -O with this function.

`parallel_bcftools_gtcheck`

parallel_bcftools_gtcheck should not be used with --all-sites, or --plot. I recommend using this function with -H and -G 1 to calculate the absolute number of differences in terms of homozygous calls between samples. Also, this function requires datamash (on OSX, install with brew install datamash)

The output file is slightly different than what bcftools normally outputs. In general, I use this function specifically to calculate conocordance between individual fastq runs - like this:

parallel_bcftools_gtchek -H -G 1 union_samples.vcf.gz > concordance.tsv

This parallelized version generates concordances for each chromosome and then merges the results together using datamash. Output looks like this:

sample_i	sample_j	discordance	number_of_sites	concordance
BGI2-RET1-ED3049	BGI1-RET1-ED3049	927	2344043	0.999605
BGI1-RET1-CB4856	BGI1-RET1-CB4852	144484	2171694	0.933469
BGI1-RET1-CX11315	BGI1-RET1-CB4852	106964	2721950	0.960703
BGI1-RET1-CX11315	BGI1-RET1-CB4856	137200	2059983	0.933398
BGI1-RET1-DL238	BGI1-RET1-CB4852	148217	2097343	0.929331
BGI1-RET1-DL238	BGI1-RET1-CB4856	124132	1803664	0.931178
BGI1-RET1-DL238	BGI1-RET1-CX11315	146580	1996802	0.926593

An Alfred workflow for generating markdown tables from your clipboard

Fri, 30 Oct 2015 00:00:00 +0000

Generate markdown tables from clipboard content.

Download

Usage

Copy a csv or tsv. The script will attempt to intelligently guess the format. For example, if you copy the table below:

carat   cut color   clarity depth   table   price   x   y   z
0.23    Ideal   E   SI2 61.5    55  326 3.95    3.98    2.43
0.21    Premium E   SI1 59.8    61  326 3.89    3.84    2.31
0.23    Good    E   VS1 56.9    65  327 4.05    4.07    2.31
0.29    Premium I   VS2 62.4    58  334 4.2 4.23    2.63
0.31    Good    J   SI2 63.3    58  335 4.34    4.35    2.75

Then type tbl in alfred and you will see the following:

You can create a table with or without a header. It will be copied to your clipboard as this:

|   carat | cut     | color   | clarity   | depth   |   table |   price |      x |    y |    z |
|--------:|:--------|:--------|:----------|:--------|--------:|--------:|-------:|-----:|-----:|
|    0.23 | Ideal   | E       | SI2       | 61.5    |    55   |     326 |   3.95 | 3.98 | 2.43 |
|    0.21 | Premium | E       | SI1       | 59.8    |    61   |     326 |   3.89 | 3.84 | 2.31 |
|    0.23 | Good    | E       | VS1       | 56.9    |    65   |     327 |   4.05 | 4.07 | 2.31 |
|    0.29 | Premium | I       | VS2       | 62.4    |    58   |     334 |   4.2  | 4.23 | 2.63 |
|    0.31 | Good    | J       | SI2       | 63.3    |    58   |     335 |   4.34 | 4.35 | 2.75 |

And it will render like this:

carat	cut	color	clarity	depth	table	price	x	y	z
0.23	Ideal	E	SI2	61.5	55	326	3.95	3.98	2.43
0.21	Premium	E	SI1	59.8	61	326	3.89	3.84	2.31
0.23	Good	E	VS1	56.9	65	327	4.05	4.07	2.31
0.29	Premium	I	VS2	62.4	58	334	4.2	4.23	2.63
0.31	Good	J	SI2	63.3	58	335	4.34	4.35	2.75

Fetch Citations in Google Sheets using pubmed() function

Thu, 29 Oct 2015 00:00:00 +0000

If you need to fetch pubmed citations in aggregate it can be convenient to do so using pubmed identifiers. I’ve created a pubmed() function that can be added to a google sheet and used to fetch formatted html citations from pubmed. For example, entering the following into a cell:

=pubmed(23149456)

Will return an html-formatted citation:

Setup

To implement the function, you’ll need to copy and paste the function below into the script editor and save it as a new project. Then it will become available within your google sheet. The script editor is available through the Tools > Script Editor

/**
 * Returns formatted pubmed citation.
 *
 * @param {id} Pubmed identifier.
 * @return Formatted pubmed citation.
 * @customfunction
 */
function pubmed(id) {
  // Special thanks to http://www.alexhadik.com/blog/2014/6/12/create-pubmed-citations-automatically-using-pubmed-api
  var content =  UrlFetchApp.fetch("http://eutils.ncbi.nlm.nih.gov/entrez/eutils/esummary.fcgi?db=pubmed&id=" + id + "&retmode=json")
  summary = JSON.parse(content)
  
  var title = summary.result[id].title;
  var journal = summary.result[id].fulljournalname;
  var volume = summary.result[id].volume;
  var issue = summary.result[id].issue;
  var citation = "";
  var pub_date = summary.result[id].pubdate;
  var pages = summary.result[id].pages;
 
  var authors = "";
  for(author in summary.result[id].authors){
    authors+=summary.result[id].authors[author].name+', ';
  }
  
  var citation = "<p><strong>" + title + "</strong><br />" +
                 authors + "<br />" +
                 "(" + pub_date + ") <em>" + journal + "</em> " + 
                 volume + " (" + issue + ") " + pages + "</p>";
                 
  
  return citation
}

An Alfred Workflow for Wormbase

Thu, 09 Jul 2015 00:00:00 +0000

I have created an Alfred workflow for looking up gene information in wormbase. You can search by wormbase ID (e.g., WBGene00006759) You can use it to search for genes. Returned results will include:

Gene identifiers
Location
Caenorhabditis orthologs
Publications

Download the latest version

Usage

Search for Genes

Get Gene Information

An Alfred Workflow for Codebox

Thu, 25 Jun 2015 00:00:00 +0000

Codebox is a great program for storing and accessing snippets. It offers a quickbar menu item, but I thought Alfred might offer more functionality. So I wrote a workflow for it.

Download Codebox-Alfred workflow

Important!

The workflow works fairly well, but there are a few caveats. You should not do the following with your codebox libraries:

Don’t put spaces into tag, list, or folder names. Use an underscore instead.
Don’t nest folders/lists with the same name.

Usage

Set the codebox source using cb_src

Invoke the workflow by typing ff

Browse tags with ff #<search>

Search all Snippets: ff <search>

HGNC Search: Instant search of human genes with Alfred

Fri, 12 Jun 2015 00:00:00 +0000

I have put together an Alfred workflow – this one searches the HGNC database for genes! I have converted the text database from the HGNC website and configured it for full text search using sqlite. This allows you to lookup genes by their UCSC, Entrez, Vega, Ensembl, and many other identifiers very quickly.

Download the latest release

Usage

Full text search of the HGNC database

Information and links are provided for individual genes

Feedback

Please provide feedback. Specifically:

What other gene IDs should be displayed by default? (You can currently search for any)
What other sites would you like to be able to navigate to.
Is there additional information that should be folded in that would be useful?

An alfred workflow for working with sequence data

Thu, 11 Jun 2015 00:00:00 +0000

I’ve put together a simple alfred workflow with a few utilities for working with sequence data.

Download the latest release of Seq-Utilities

Usage

Generate a random dna sequence 200 base pairs long.

Generate the complement, reverse complement, RNA, and protein of a DNA sequence

Open up blast and pre-populate the search field

blast ATGTCCTCGTTCGACCGTCGTATTGAAGCTGCATGTAAA

Split a GFF File into Individual Features

Sun, 25 Jan 2015 00:00:00 +0000

The General Feature Format is a widely used format for annotating genome sequences. If indexed with tabix, gff files can be viewed in IGV or elsewhere. While features are organized in a nested manner (e.g. genes > exons > variant), you can pull out the individual types and index them, or combine only a few for viewing in your genome browser.

I was working with wormbase annotation files, which combine all the different types of features together (genes, ncRNA, mRNA, binding site, operon, G Quartets, piRNAs, etc). This results in a very dense track in IGV which makes it difficult to disentangle what role individual features (or features of interest) might have.

As a result, I wrote this very short script for splitting the individual feature types apart, sorting them, and indexing them with tabix. This way they can be selectively viewed in IGV or elsewhere.

import sys

current_feature = ""

for line in sys.stdin:
    feature = line.split("\t")[2]
    if feature != current_feature:
        f = file(feature + ".gff", "a+")
    f.write(line)

gunzip -kfc <GFF> | grep -v ^"#" | sort -k3,3 | python process_gff.py

for i in `ls *.gff`; do
    (grep ^"#" $i.gff; grep -v ^"#" $i.gff | sort -k1,1 -k4,4n) | bgzip > $i.sorted.gff.gz;
    tabix $i.sorted.gff.gz
    rm $i.gff
done

Aggregate FastQC Reports

Sun, 28 Dec 2014 00:00:00 +0000

Update: MultiQC (2019-06-21)

After I originally published this script for aggregating FASTQC reports, MultiQC was published by Phil Ewels. MultiQC aggregates quality-control and other associated data from sequencing tools into an interactive report. Instead of the script below, you can simply run:

# Run this command where your *_fastqc.zip files are
multiqc .

This will output a repor that looks like this:

Publication

MultiQC: Summarize analysis results for multiple tools and samples in a single report Philip Ewels, Måns Magnusson, Sverker Lundin and Max Käller Bioinformatics (2016) doi: 10.1093/bioinformatics/btw354 PMID: 27312411

Original Post (2014-12-28)

FastQC is a phenomenal sequence quality assessment tool for evaluating both fastq and bam files. If you are working with a large number of sequence files (fastq), you may wish to compare results across all of them by comparing the plots that fastqc produces. I’m talking about the set of plots that look like this:

FastQC can be invoked from the command line by typing fastqc <fastq/bam>, and it will produce an html report and associated zip file containing data, plots, and some ancillary files. The zip file contains an Images folder where the plots that become incorporated into the html report are stored. They are:

Adapter Content
Duplication Levels
Kmer Profiles
Per base N Content
Per Base Quality
Per Base Sequence Content
Per Sequence GC Content
Per Sequence Quality
Per Tile Quality
Sequence Length Distribution

The zipped folder also contains a file called fastqc_data.txt and summary.txt. fastqc_data.txt contains the raw data and statistics while summary.txt summarizes which tests have been passed.

To easily compare data across reports I wrote this short shell script (below) which will ‘aggregate’ images, statistics, and summaries by:

Unzipping all the avaible fastqc zip files.
Creating a fq_aggregated folder, and individual folders within for each plot type.
Move images from each unzipped fastqc report into the folder to which it belongs, and renaming it as the filename of the report (e.g. sample name).
Concatenating summary.txt files as fq_aggregated/summary.txt.
Concatenating the basic statistics from each report into fq_aggregated/statistics.txt.

Images will be reorganized as shown below:

`summary.txt`

fq_aggregated/summary.txt will produce a tab delimited file that looks like this:


PASS	Basic Statistics	SeqA.fq
PASS	Per base sequence quality	SeqA.fq
PASS	Per tile sequence quality	SeqA.fq
PASS	Per sequence quality scores	SeqA.fq
FAIL	Per base sequence content	SeqA.fq
PASS	Per sequence GC content	SeqA.fq
PASS	Per base N content	SeqA.fq
	…
PASS	Basic Statistics	SeqB.fq
PASS	Per base sequence quality	SeqB.fq
PASS	Per tile sequence quality	SeqB.fq
PASS	Per sequence quality scores	SeqB.fq
PASS	Per base sequence content	SeqB.fq
FAIL	Per sequence GC content	SeqB.fq
FAIL	Per base N content	SeqB.fq

`statistics.txt`

fq_aggregated/statistics.txt will look like this:


PASS	Basic Statistics	SeqA.fq
PASS	Per base sequence quality	SeqA.fq
PASS	Per tile sequence quality	SeqA.fq
PASS	Per sequence quality scores	SeqA.fq
FAIL	Per base sequence content	SeqA.fq
PASS	Per sequence GC content	SeqA.fq
PASS	Per base N content	SeqA.fq
d	…
PASS	Basic Statistics	SeqB.fq
PASS	Per base sequence quality	SeqB.fq
PASS	Per tile sequence quality	SeqB.fq
PASS	Per sequence quality scores	SeqB.fq
PASS	Per base sequence content	SeqB.fq
FAIL	Per sequence GC content	SeqB.fq
FAIL	Per base N content	SeqB.fq

The Code

# Run this script in a directory containing zip files from fastqc. It aggregates images of each type in individual folders
# So looking across data is quick.

zips=`ls *.zip`

for i in $zips; do
    unzip -o $i &>/dev/null;
done

fastq_folders=${zips/.zip/}

rm -rf fq_aggregated # Remove aggregate folder if present
mkdir fq_aggregated

# Rename Files within each using folder name.
for folder in $fastq_folders; do
    folder=${folder%.*}
    img_files=`ls ${folder}/Images/*png`;
    for img in $img_files; do
        img_name=$(basename "$img");
        img_name=${img_name%.*}
        new_name=${folder};
        mkdir -p fq_aggregated/${img_name};
        mv $img fq_aggregated/${img_name}/${folder/_fastqc/}.png;
    done;
done;


# Concatenate Summaries
for folder in $fastq_folders; do
    folder=${folder%.*}
    cat ${folder}/summary.txt >> fq_aggregated/summary.txt
done;

# Concatenate Statistics
for folder in $fastq_folders; do
    folder=${folder%.*}
    head -n 10 ${folder}/fastqc_data.txt | tail -n 7 | awk -v f=${folder/_fastqc/} '{ print $0 "\t" f }' >> fq_aggregated/statistics.txt
    rm -rf ${folder}
done

Downgrade a VCF for viewing in IGV (4.2 > 4.1)

Mon, 15 Dec 2014 00:00:00 +0000

Update: You probably no longer need this (2019-06-24)

If you are using up to date software then you probably do not need to worry about downgrading a VCF file.

Original Post (2014-12-15)

If you are using the new version of bcftools, and you frequently use IGV to view variants you may have run into issues loading the file in IGV. IGV currently does not support VCF version 4.2. However, I’ve been able to tweak the headers of newer VCF files to allow these variants to be viewable in IGV again.

All you have to do is revert the version number in the first line and replace a few characters IGV does not like. Below is a bash function that will do this – saving any inputted VCF as {vcf_filename}.dg.vcf.gz. The script also indexes the file making it ready for use.

# If you are trying to view VCF 4.2 files in IGV - you may run into issues. This function might help you.
# This script will:
# 1. Rename the file as version 4.1
# 2. Replace parentheses in the INFO lines (IGV doesn't like these!)

function vcf_downgrade() {
  outfile=${1/.bcf/}
  outfile=${outfile/.gz/}
  outfile=${outfile/.vcf/}
  bcftools view --max-alleles 2 -O v $1 | \
  sed "s/##fileformat=VCFv4.2/##fileformat=VCFv4.1/" | \
  sed "s/(//" | \
  sed "s/)//" | \
  sed "s/,Version=\"3\">/>/" | \
  bcftools view -O z > ${outfile}.dg.vcf.gz
  tabix ${outfile}.dg.vcf.gz
}

Rename Samples within a VCF/BCF

Fri, 05 Dec 2014 00:00:00 +0000

Update: Use bcftools (2019-06-21)

Since this post was originally written, bcftools has added a command for renaming samples called reheader which allows sample names to be easily modified.

Original Post (2014-12-05)

These two simple bash functions make it easy to rename samples within a bcf file by using the filename given (if it is a single sample file) or adding a prefix to all samples. This is useful if you want to merge bcf files where the sample names are identical in both (for comparison purposes).

function rename_to_filename {
    # Renames samples with the filename.
    tmp=`mktemp -t temp`
    echo ${1/.[vb]cf/} > $tmp
    bcftools reheader -s $tmp $1 > m.$1
    mv m.$1 $1
    bcftools index $1
}

function add_sample_prefix {
    # Adds a prefix to the samples within a bcf file.
    tmp=`mktemp -t temp`
    bcftools query -l $1 | awk -v g=$2 '{ print g $0 }'  > $tmp
    bcftools reheader -s $tmp $1 > m.$1
    mv m.$1 $1
    bcftools index $1
}

From SRA Project to FASTQ

Sat, 25 Oct 2014 00:00:00 +0000

Original Post (2014-10-25)

The Sequence Read Archive (SRA) contains sequence data from scientific studies stored in a special ‘sra’ format. Data is stored in a hierarchical format:

Project ▸ Study ▸ Sample ▸ Experiment ▸ Run

Recently, I had to use the SRA to download all of the sequence data for a given project. This required querying the SRA database for all the runs in a sequencing project and converting them to FASTQs. Here’s how I did it:

First, you’ll need entrez direct, and the sra toolkit. If you are on a mac, you can install both using homebrew.

brew install edirect # Entrez Direct
brew install sratoolkit

Once installed, the script below can be used to download all the sequence data associated with a given project. The script queries the project for all the associated sequence data, and converts to zipped FASTQs. Note that it also uses gnu parallel (to speed things up) and fastqc for quality control. These can be installed on mac using:

brew install parallel
brew install fastqc

Download_SRP_Runs() {
    SRP_IDs=`esearch -db sra -query $1 | efetch -format docsum | xtract -pattern DocumentSummary -element Run@acc | tr '\t' '\n'`
    for r in ${SRP_IDs}; do
        url="ftp://ftp-trace.ncbi.nih.gov/sra/sra-instant/reads/ByRun/sra/SRR/${r:0:6}/${r}/${r}.sra"
        wget $url
    done;
}

Download_SRP_Runs <SRP ID GOES HERE>

# Convert to fastq
parallel fastq-dump --split-files --gzip {} ::: *.sra

# Perform quality control
parallel fastqc {} ::: *.fastq.gz

Update: SRA Explorer (2019-06-20)

The SRA Explorer by Phil Ewels can be used to generate a collection of SRA datasets and direct download links for their FASTQs.

Calculate Depth and Breadth of Coverage From a bam File

Sat, 20 Sep 2014 00:00:00 +0000

Original Post

What is the difference between depth and coverage in sequencing experiments? Actually – they refer to the same thing, the average number of reads aligned to an individual base. Previously, I had thought coverage referred to the percentage of the genome with aligned reads to it; however the more appropriate term for this is breadth of coverage. This paper more precisely defines what breadth of coverage and depth of coverage mean.

If you need to calculate depth of coverage and breadth of coverage you can do so using the python script below. To use the script, feed the function coverage a bam file, and the function will return a dictionary of the depth of coverage, breadth of coverage, sum of depths (at every position), and number of bases mapped, for every contig/chromosome individually, and the entire genome as a whole.

Additionally, if you specify the optional second parameter specifying the mitochondrial chromosome, the script will calculate the parameters listed above for the nuclear genome and calculate the ratio of mitochondrial depth of coverage to nuclear depth of coverage. This can act as a proxy for mitochondrial count/content within a cell.

#
# This script calculates the depth of coverage and breadth of coverage for a given bam. 
# Outputs a dictionary containing the contig/chromosome names and the depth and breadth of coverage for each
# and for the entire genome.
#
# If you optionally specify the name of the mitochondrial chromosome (e.g. mtDNA, chrM, chrMT)
# The script will also generate breadth and depth of coverage for the nuclear genome AND the ratio
# of mtDNA:nuclearDNA; which can act as a proxy in some cases for mitochondrial count within an individual.
# 
# Author: Daniel E. Cook
# Website: Danielecook.com
#


import os
import re
from subprocess import Popen, PIPE

def get_contigs(bam):
    header, err = Popen(["samtools","view","-H",bam], stdout=PIPE, stderr=PIPE).communicate()
    if err != "":
        raise Exception(err)
    # Extract contigs from header and convert contigs to integers
    contigs = {}
    for x in re.findall("@SQ\WSN:(?P<chrom>[A-Za-z0-9_]*)\WLN:(?P<length>[0-9]+)", header):
        contigs[x[0]] = int(x[1])
    return contigs

def coverage(bam, mtchr = None):
    # Check to see if file exists
    if os.path.isfile(bam) == False:
        raise Exception("Bam file does not exist")
    contigs = get_contigs(bam)

    # Guess mitochondrial chromosome
    mtchr = [x for x in contigs if x.lower().find("m") == 0]
    if len(mtchr) != 1:
        mtchr = None
    else:
        mtchr = mtchr[0]

    coverage_dict = {}
    for c in contigs.keys():
        command = "samtools depth -r %s %s | awk '{sum+=$3;cnt++}END{print cnt \"\t\" sum}'" % (c, bam)
        coverage_dict[c] = {}
        coverage_dict[c]["Bases Mapped"], coverage_dict[c]["Sum of Depths"] = map(int,Popen(command, stdout=PIPE, shell = True).communicate()[0].strip().split("\t"))
        coverage_dict[c]["Breadth of Coverage"] = coverage_dict[c]["Bases Mapped"] / float(contigs[c])
        coverage_dict[c]["Depth of Coverage"] = coverage_dict[c]["Sum of Depths"] / float(contigs[c])
        coverage_dict[c]["Length"] = int(contigs[c])

    # Calculate Genome Wide Breadth of Coverage and Depth of Coverage
    genome_length = float(sum(contigs.values()))
    coverage_dict["genome"] = {}
    coverage_dict["genome"]["Length"] = int(genome_length)
    coverage_dict["genome"]["Bases Mapped"] = sum([x["Bases Mapped"] for k, x in coverage_dict.iteritems() if k != "genome"])
    coverage_dict["genome"]["Sum of Depths"] = sum([x["Sum of Depths"] for k, x in coverage_dict.iteritems() if k != "genome"])
    coverage_dict["genome"]["Breadth of Coverage"] = sum([x["Bases Mapped"] for k, x in coverage_dict.iteritems() if k != "genome"]) / float(genome_length)
    coverage_dict["genome"]["Depth of Coverage"] = sum([x["Sum of Depths"] for k, x in coverage_dict.iteritems() if k != "genome"]) / float(genome_length)

    if mtchr != None:
        # Calculate nuclear breadth of coverage and depth of coverage
        ignore_contigs = [mtchr, "genome", "nuclear"]
        coverage_dict["nuclear"] = {}
        coverage_dict["nuclear"]["Length"] = sum([x["Length"] for k,x in coverage_dict.iteritems() if k not in ignore_contigs ])
        coverage_dict["nuclear"]["Bases Mapped"] = sum([x["Bases Mapped"] for k, x in coverage_dict.iteritems() if k not in ignore_contigs])
        coverage_dict["nuclear"]["Sum of Depths"] = sum([x["Sum of Depths"] for k, x in coverage_dict.iteritems() if k not in ignore_contigs])
        coverage_dict["nuclear"]["Breadth of Coverage"] = sum([x["Bases Mapped"] for k, x in coverage_dict.iteritems() if k not in ignore_contigs]) / float(coverage_dict["nuclear"]["Length"])
        coverage_dict["nuclear"]["Depth of Coverage"] = sum([x["Sum of Depths"] for k, x in coverage_dict.iteritems() if k not in ignore_contigs]) / float(coverage_dict["nuclear"]["Length"])

        # Calculate the ratio of mtDNA depth to nuclear depth
        coverage_dict["genome"]["mt_ratio"] = coverage_dict[mtchr]["Depth of Coverage"] / float(coverage_dict["nuclear"]["Depth of Coverage"])

    # Flatten Dictionary 
    coverage = []
    for k,v in coverage_dict.items():
        for x in v.items():
            coverage += [(k,x[0], x[1])]
    return coverage

Requires BCFTools

Update: mosdepth (2019-06-21)

If you are looking to calculate coverage, I highly recommend mosdepth.

Generate a bedfile of masked ranges a fasta file

Mon, 15 Sep 2014 00:00:00 +0000

If you are calling variants as part of a NGS experiment, you likely are considering filters such as depth, quality, and filtering low complexity regions from the variant dataset. Programs such as repeatmasker are used to identify low complexity regions, replacing repetitive sequences with N‘s. Repetitive regions have a tendency to be aligned with inappropriate reads and results in false positives.

If you’ve been provided with or have generated a masked fasta file for a given genome, you can use the following script convert a masked fasta (left) into a bed file (right) with the masked ranges.

>CHROMOSOME_I 1 15072423
NNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNN
NNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNN
NNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNN
NNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNN
NNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNN
NNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNN
NNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNN
NNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNN
NNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNN
NNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNN
NNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNN
NNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNN
NNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNN
NNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNN
NNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNN
NNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNN
NNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNN
NNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNN
NNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNN
NNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNN
NNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNN
NNNNNNNNNNNGTTTGTTNNNNNNNNNNNNNNNNN
NNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNN
TATTAAAAACTGTTCNNNNNNNNNNNNNNNNNNNN

chrI    0   4831
chrI    4869    5146
chrI    5181    5305
chrI    5340    5677
chrI    5706    7409
chrI    7431    9549
chrI    9593    9651
chrI    9683    9979
chrI    10014   18897
chrI    18941   19432
chrI    19468   19747
chrI    19782   19877
chrI    19898   21314
chrI    21357   24849
chrI    24903   27411
chrI    27456   27535
chrI    27561   28015
chrI    28054   28505
chrI    28527   28918
chrI    28961   30659
chrI    30682   39364
chrI    39419   42234
chrI    42307   56428
chrI    56455   57860

Each range corresponds with a low complexity region within the fasta file. The resulting bed file can be used to filter variants out of a VCF file using a tool such as bcftools

Usage

python generate_masked_ranges.py <fasta_file> > output_ranges.txt

Script

#!bin/python

import gzip
import io
import sys
import os

# This file will generate a bedfile of the masked regions a fasta file.

# STDIN or arguments
if len(sys.argv) > 1:

    # Check file type
    if sys.argv[1].endswith(".fa.gz"):
        input_fasta = io.TextIOWrapper(io.BufferedReader(gzip.open(sys.argv[1])))
    elif sys.argv[1].endswith(".fa") or sys.argv[1].endswith(".txt"):
        input_fasta = file(sys.argv[1],'r')
    else:
        raise Exception("Unsupported File Type")
else:
    print """
    \tUsage:\n\t\tgenerate_masked_ranges.py <fasta file | .fa or .fa.gz> <chrome find> <chrome replace>
    
    \t\t'Chrome find' and 'chrome replace' are used to find and replace the name of a chromsome. For example,
    \t\treplacing CHROMSOME_I with chr1 can be accomplished by using the command as follows:
    \t\t\tpython generate_masked_ranges.py my_fasta.fa CHROMSOME_ chr
    \t\tOutput is to stdout
    """
    raise SystemExit


n, state = 0, 0 # line, character, state (0=Out of gap; 1=In Gap)
chrom, start, end = None, None, None

with input_fasta as f:
    for line in f:
        line = line.replace("\n","")
        if line.startswith(">"):
            # Print end range
            if state == 1:
                print '\t'.join([chrom ,str(start), str(n)])
                start, end, state  = 0, 0, 0
            n = 0 # Reset character
            chrom = line.split(" ")[0].replace(">","")
            # If user specifies, replace chromosome as well
            if len(sys.argv) > 2:
                chrom = chrom.replace(sys.argv[2],sys.argv[3])
        else:
            for char in line:
                if state == 0 and char == "N":
                    state = 1
                    start = n
                elif state == 1 and char != "N":
                    state = 0
                    end = n
                    print '\t'.join([chrom ,str(start), str(end)])
                else:
                    pass

                n += 1 # First base is 0 in bed format.

# Print mask close if on the last chromosome.
if state == 1:
            print '\t'.join([chrom ,str(start), str(n)])
            start, end, state  = 0, 0, 0

Generate fasta sequence lengths

Wed, 13 Aug 2014 00:00:00 +0000

This one liner:

cat file.fa | awk '$0 ~ ">" {if (NR > 1) {print c;} c=0;printf substr($0,2,100) "\t"; } $0 !~ ">" {c+=length($0);} END { print c; }'

Takes a fasta file as input:

>EF491733
tcagattcaaacaccgacgacgatgacgtggcaaagtctcgacgtgtgcg
caattcgtgtatgtgtccagcaggacctcccggagaacgcggaccagtag
gaccaccaggtctacggggatcgccaggatggcct
>EF491734
tcacagggaatgaaggcactgttcgacttgatcgctttgagaccaagacc
cgtggcaattctcggagggcaatgcactgaagtgaacgagccaatagcga
tggcgctcaagtattggcaaatcgtgcaattatcctatgcggagacacat
gccaa
>EF491735
gtcttgcatgacccaaaaggctcctgctcttctgtttcttcttccaatac
atccttctaaccagttggaagggttgacgtatcaagacttcctgcatcaa
aacttcttgaatttgccttcatttgtcgcaattgtgcagc
>EF491736
taaatggaaggaatcacttggcgctgaagaatttgctctccgcacagctt
aatcagactggaactccaatggttaatccaatgatggctttacaacaaca
agcggccgcagtaaacctgattcccaacacaccaatttacccaccc
>EF491737
actctcgcaatcgtctctccccaaatgatgttaacatcactagaaatgac
aaccgaacatatagcccagtcactcctcgtatcacaacaagtgagcggac
agtaacaccggaacagcggtcgccgggtcgaaaagcgttcgaaaccattc
>EF491738
tccctcgttcattcacaacaaaggaaaagcaaactatgggccattcattg
ttgaaattatgaactatcatcagtattctgcaatgacaagtcatatggtc
aaagtaatgaaacggccccaccaggttccgccaatgaaggtcgaccctga
gg
>EF491739
tccttccaactgttgccaactttccaactacaagacacactgaaccagaa
actacgcggagacctctgtcgccttcaaaaatgacaccttctcttccttc
tcctaccaccaccactttgcctgttttctttttgtcacaaatcactgacg
gcgatgaatcagaagatgaa

Outputs sequence name and length:

EF491733    135
EF491734    155
EF491735    140
EF491736    146
EF491737    150
EF491738    152
EF491739    170

I made this today when I needed a way to generate sequence lengths required for some ChIP-Seq analysis.

Visualizing Pairwise Queries in R

Sat, 02 Aug 2014 00:00:00 +0000

You can look for interesting associations between sets of search terms on PubMed by comparing how often two terms co-occur. The code below returns the number of publications where both terms are mentioned, acting as a rough estimate for how associated they are (at least, in the scholarly world).

In the example below, I show the results from organisms x diseases/disease-associated terms which is an imperfect look at how various terms estimate of how much each disease is studied in a given organism. Of course, this should all be taken with a (big) grain of salt because these organisms and diseases have many synonyms or related terms (e.g. M. Musculus is often referred to as Mouse in the literature). Additionally, the result count is based off of whether or not the terms were found together within the title and abstract of the literature only – and not the body of the text in many cases.

# install.packages("RISmed", "ggplot2")
library(RISmed)
library(ggplot2)

# Given two lists of terms, lets see how 'hot' they are together
set1 <- c("ebola","autoimmune","Diabetes","Glioblastoma","Asthma","Schizophrenia")
set2 <- c("C. elegans","D. Melanogaster", "M. Musculus","S. Cerevisiae")

# Generate all possible pairs
pairs <- expand.grid(set1, set2, stringsAsFactors=F)

# Search pubmed for each pair, and return the number of search results.
results <- lapply(1:nrow(pairs),  function(x) {
  query <- sprintf("%s %s", pairs[x,]$Var1, pairs[x,]$Var2)
  print(query)
  result <- EUtilsSummary(query, type='esearch', db='pubmed')
  c(q1=pairs[x,]$Var1, q2=pairs[x,]$Var2, count=QueryCount(result))
})

# Do some data formatting on the results.
results <- as.data.frame(do.call("rbind", results), stringsAsFactors=F)
# Turn the number of search results into numeric form.
results$count <- as.numeric(results$count)

# Plot the results using geom_tile
ggplot(results) +
  geom_tile(aes(x=q1, y=q2, fill=count)) +
  geom_text(aes(x=q1, y=q2, label=count), color = "white") + 
  labs(title="Disease Publications by Organism", x="x", y="y")

ggsave("~/Desktop/pairwise_search.png", width = 8, height = 6)

A Short tour around Lake Michigan

Sat, 12 Jul 2014 00:00:00 +0000

I’ve given bicycle touring a try. Originally I wanted to bike around Lake Michigan, but it turns out to be over 1,400 miles. So I compromised on a three day trip around a good chunk and making use of the ferry from Muskeegon, MI to Milwaukee, WI. This was my first time – so I also decided to stay in hotels. Next time I intend to camp. I learned a few valuable lessons along the way!

Pack less stuff! – I had way too much. In fact, I wound up breaking two spokes on the second day.
Shorten the days – Having never gone more than 40 miles in a single day, I decided to go 108 on the first day. Yeah. I probably should have gone more like 60-70 each day. By the time I got to my destination each day I was too tired to do anything. Part of the experience is seeing new places.
Get a proper touring bike – I didn’t use a touring bike because I don’t have one (yet). I used a Trek 7.2. My wrists hurt a lot for parts of the trip. Next time I’ll get a proper touring bike with the appropriate handle bars.

Pictures

How to plot all of your Runkeeper Data

Fri, 30 May 2014 00:00:00 +0000

If you use runkeeper and pay for a yearly subscription (runkeeper elite), you can export your data and plot all of your activities simultaneously using R. I’ve written a script for doing so (Special thanks to flowing data which has a tutorial that helped with a few key parts of this).

The script does a few unique things.

Runkeeper exports data in gpx format. If you ever pause an activity within runkeeper or you lose GPS reception briefly, the GPS path will get split into multiple paths within the same file. The script will retain all paths and plot them separately.
This script will merge in the type of activities so you can plot different types of activities by color.
Finally, cluster analysis is used to segregate different locations when plotting. If you are like me and have moved around a bit – this is necessary as plotting distant locations on the same map (e.g. Chicago and Boston) is not feasible and results in distant locations being plotted as single points.

Directions

Export your runkeeper data. The option is available for subscribers only under the settings menu.

Place the script below within a folder containing your runkeeper data. Set the num_locations variable to the number of places you have lived/run. This will be used to pull out the number of distinct running locations automatically.
Install the necessary R packages. You can run the following code within R to do so.

install.packages("fpc")
install.packages("plyr")
install.packages("dplyr")
install.packages("mapproj")

Run the script below from within R Studio or on unix based machines using RScript plot_runkeeper.R. If you are using Rstudio, be sure to set the working directory using setwd()

# Special thanks for insights from flowingdata.com regarding this.

library(plotKML)
library(plyr)
library(dplyr)
library(fpc)

num_locations <- 5

# Usage: Place this script in the directory containing your runkeeper data. You can run from terminal using 'Rscript map_runkeeper.R', or
# set your working directory to the location and run within RStudio (use setwd("~/location/of/runkeeper/data")).
# See below on how to set the number of clusters.

# GPX files downloaded from Runkeeper
files <- dir(pattern = "\\.gpx")

# Generate vectors for data frame
index <- c()
latitude <- c()
longitude <- c()
file <- c()

c <- 1 # Set up Counter

# 
for (f in 1:length(files)) {
  curr_route <- readGPX(files[f])
# Treat interrupted GPS paths as seperate routes (useful if you occasionally stop running..walk for a bit, and start again like I do.)
for (i in curr_route$tracks[[1]]) {
  c <- c + 1
  location <- i
  file <- c(file,rep(files[f], dim(location)[1])) 
  index <- c(index, rep(c, dim(location)[1]))
  latitude <- c(latitude, location$lat)
  longitude <- c(longitude, location$lon)
}
}
routes <- data.frame(cbind(index, latitude, longitude,file))

# Because the routes dataframe takes a while to generate for some folks - save it!
save(routes, file="routes.Rdata")
# Use to load as needed.
load("routes.Rdata")

# Fix data types
routes$file <- as.character(routes$file)
routes$latitude <- as.numeric(levels(routes$latitude)[routes$latitude])
routes$longitude <- as.numeric(levels(routes$longitude)[routes$longitude])
routes <- transform(routes, index = as.numeric(index))

# Load Meta Data
meta_data <- read.csv("cardioActivities.csv", stringsAsFactors=FALSE)
meta_data <- rename(meta_data, c("GPX.File" = "file"))

# Bind routes
routes <- left_join(routes, meta_data, by="file") %.%
  arrange(index)


# Use this function specify activity color if you have multiple activities.
activity_color <- function(activity) {
  if (activity=="Cycling") {
    color = "#00000060"
  } else if (activity=="Hiking") {
    color = "#00000060"
  } else {
    color = "#0080ff60"
  }
  color
}

# Identify clusters of points, which will correspond to locations you have run. For example,
# I have run in Boston, Iowa City, Chicago, and a few other cities. You will want to set the minimum krange
# to the number of cities you have run in (5 in my case).
clusters <- pamk(routes[,c("latitude", "longitude")], krange=num_locations:20, diss=T, usepam=F)$pamobject$medoids

# Plot Everything
for (r in 1:max(row(clusters))) {
  print(r)
  lat_range <- clusters[r,][1] + rnorm(20, sd=0.1)
  lon_range <-clusters[r,][2] + rnorm(20, sd=0.1)
  setroutes <- filter(routes, (latitude > min(lat_range) & latitude < max(lat_range)),
                      longitude > min(lon_range) &  longitude < max(lon_range))
  
  routeIds <- unique(setroutes$index)
  
  # Albers projection
  locProj <- mapproject(setroutes$longitude, setroutes$latitude, "rectangular", par=38)
  setroutes$latproj <- locProj$x
  setroutes$lonproj <- locProj$y
  
  
  # Map the projected points
  pdf(sprintf("%s-all.pdf", r))
  
  plot(setroutes$latproj, setroutes$lonproj, type="n", asp=1, axes=FALSE, xlab="", ylab="")
  for (i in routeIds) {
    currRoute <- subset(setroutes, index==i)
    lines(currRoute$latproj, currRoute$lonproj, col=activity_color(currRoute$Type), lwd=0.4)
  }
  dev.off()
}

Where I Run and Bike in Chicago

Sun, 25 May 2014 00:00:00 +0000

Original Post (2014-05-25)

Using runkeeper and with the help of a tutorial at flowing data, I was able to plot all of the running and biking I’ve been doing in Chicago since moving here two years ago. The blue is running and the black is biking.

Update: Strava (2019-06-21)

I have since migrated my data to Strava - which I like a lot better than runkeeper. You can migrate your data using the Tapariik service.

Strava has a really cool global heatmap - or you can visualize your own with their pro subscription.

The global heatmap reveals the locations of Ragbrai across the state of Iowa for the past couple of years.

Double Checking FASTQs

Sat, 24 May 2014 00:00:00 +0000

When you have performed a sequencing project, quality control is one of the first things you will need to do. Unfortunately, sample mix-ups and other issues can and do happen. Systematic biases can also occur by machine and lane.

This script will extracting basic information from a set of FASTQs and output it to summary file (fastq_summary.txt). This will work with demultiplexed FASTQs generated by Illumina machines that appear in the following format:

@HWI-EAS209_0006_FC706VJ:5:58:5894:21141#ATCACG/1

@HWI-EAS209_0006_FC706VJ – Machine name
5 – lane
58 – tile within flowcell lane
5894 – x coordinate of cluster within tile
21141 – y coordinate of cluster within tile
#ATCACG – index
/1 – member of pair (/1 or /2)

The script below will extract the machine name, lane, index, and pair.

#!/usr/bin/python
import re
from itertools import groupby as g
import subprocess
import sys
from collections import OrderedDict

def most_common(L):
  return max(g(sorted(L)), key=lambda(x, v):(len(list(v)),-L.index(x)))[0]

# Set this variable to ensure no quality score lines get examined.
fq_at_start = "HWI"

r = subprocess.check_output("""
for r in `ls *fastq.gz`; 
do
echo "$r" 
gunzip -cq $r | head -n 1000 | grep '^@%s' - | grep -v '^@@' |  egrep '(:.+){4}' -
echo "|" 
done;
""" % fq_at_start, shell=True)


f = open("fastq_summary.txt", "w")
 

orig_line =  OrderedDict([("file", ""),
      ("instrument", ""),
      ("flowcell_id", ""),
      ("flowcell_lane", ""),
      ("x_coord", ""),
      ("y_coord", ""),
      ("index", ""),
      ("pair", ""),
      ("run_id", ""),
      ("filtered",""),
      ("control_bits","")])
l_keys = orig_line.keys()

f.write('\t'.join(l_keys) + "\n")
 
for fq_group in [filter(len,x.split('\n')) for x in r.split("|")][:-1]:
  index_set = []
  for heading in fq_group[1:]:
    l = re.split(r'(\:|#| )',heading)
    line = orig_line
    line["file"] = fq_group[0]
                if l[0].startswith("@SRR"):
                    line["run_id"] = l[0]
                    line["instrument"] = l[2]
                    line["flowcell_lane"] = l[4]
                    index_set.append("")
    elif len(l) == 11:
      line["instrument"] = l[0]
      line["flowcell_lane"] = l[2]
      line["flowcell_tile"] = l[4]
      line["x_coord"] = l[6]
      line["y_coord"] = l[8]
      try:
        line["pair"] = l[10].split("/")[1]
        index_set.append(l[10].split("/")[0])
      except:
        break
    elif len(l) == 21:
      line["instrument"] = l[0]
      line["run_id"] = l[2]
      line["flowcell_id"] = l[4]
      line["flowcell_lane"] = l[6]
      line["flowcell_tile"] = l[8]
      line["x_coord"] = l[10]
      line["y_coord"] = l[12]
      line["pair"] = l[14]
      line["filtered"] = l[16]
      line["control_bits"] = l[16]
      line["index"] = l[20]
      index_set.append(l[20])
    else:
      print "error", l
  line["index"] = most_common(index_set)
  f.write('\t'.join([line[x] for x in l_keys]+ ["\n"]))

Use Google Sheets to identify gene-disease associations in Pubmed

Mon, 03 Mar 2014 00:00:00 +0000

Google Docs allows you to import XML. By using NCBIs esearch service, you can query pubmed for a list of genes. Stick the following code in A2, and a keyword in B2:

=importXML("http://eutils.ncbi.nlm.nih.gov/entrez/eutils/esearch.fcgi?db=pubmed&term=" & B2 ,"(//Count)[1]")

What is more valuable, however, is if given a gene list – you can query pubmed for each gene combined with a second keyword like a disease.

For example, suppose you are studying Cleft lip and Palate and are left with a set of genes identified from a gene expression analysis. Now you want to see if any of those genes have published findings on them related to cleft lip and palate.

You can use the & operator to concatenate two keywords (gene & " " & disease). In B2 below you would put the following:

= C2 & " " & D2

The result would look something like this:

Pubmed result counts are in Column A

An R Function for Opening a dataframe in Excel (Mac Only)

Tue, 18 Feb 2014 00:00:00 +0000

The dataframe viewer in Rstudio can be slow or unresponsive, and at times truncates the content within or the number of columns on large datasets. I want to be able to see the full columns and to be able to arrange and filter simultaneously. Although you can do this in R programmatically sometimes its easier and quicker to use Excel. The function below can be used to open a dataframe in Microsoft Excel.

This may be worth sticking in your .RProfile so it is always available.

excel <- function(df) {
  f <- paste0(tempdir(),
              '/',
              make.names(deparse(substitute(df))),
              '.',
              paste0(sample(letters)[1:5],collapse=""),
              '.csv')
  write.csv(df,f)
  system(sprintf("open -a 'Microsoft Excel' %s",f))
}

To use, just type:

excel(dataframe)

# Or pipe in using dplyr

df %>% excel()

Microsoft Excel will open with the dataframe that has been passed.

Alfred Workflow for Creating a Data Analysis Project

Sat, 25 Jan 2014 00:00:00 +0000

This idea I got from my brother. The idea is to keep any data analysis/bioinformatic projects I work on organized by sticking to a standard template. I wrote an Alfred Workflow for generating the template. There are a couple key features:

Directory Structure

Markdown (md) extension – is used for the readme because its simple and so that the directory is ready for github if desired.
Data Folder – This directory is used for storing raw data and scripts that are used to clean and prepare data for analysis.
analysis – This directory contains the scripts for producing statistics and visualizing data.
- report – any publications or presentations that come of the project can be stored in the report folder.
  - run.sh is a two line script that will run prepare_data.sh and analysis.sh. This allows you to reproduce the entirety of your work all at once and verify your results. What are your thoughts? How could this be improved?

Usage

Navigate to the directory where you would like to create the project template; open alfred and type

project [a name for your project]

Download

Downloading and storing bioinformatic databases locally

Mon, 20 Jan 2014 00:00:00 +0000

If you need to annotate biological data there are plenty of resources online (UCSC Genome Browser, BioMart), and plenty of programmatic tools to interact with these databases as well. But if you are going to be annotating a large dataset (like ChIP-Seq or RNA-Seq data) – you will probably not want to rely on web based services because a) It is inefficient b) You may get throttled or banned.

If you use python, it is easy to download and store data in an SQlite database. This allows you to query the database using SQL and quickly and efficiently annotate large datasets.

Below you will see that is what I have done here for HapMap allele frequency data (2010-08_phaseII+III), and it allows me to retrieve allele frequency data from 26,278,275 rows across 11 populations instantly. The database itself is 3.22 Gb. A zipped version (~1Gb) is available Here.

You will need sqlalchemy for this script to work. Install using pip install sqlalchemy.

#! /usr/local/bin/Python
import sqlite3
import os
import glob
import time
import sqlalchemy
from sqlalchemy import Table, Column, Index, Integer, String, Float, MetaData, ForeignKey
from sqlalchemy import create_engine
import datetime

os.chdir(os.path.dirname(__file__))


os.system('wget -nd -r  -A "allele*.gz" -e robots=off "http://hapmap.ncbi.nlm.nih.gov/downloads/frequencies/2010-08_phaseII+III/"')
os.system('gunzip *.gz # Unzip all the files')

if os.path.isfile('hapmap.db'):
    os.remove('hapmap.db')

engine = create_engine('sqlite:///hapmap.db')
conn = engine.connect()

metadata = MetaData()

freq = Table('freq', metadata,
    Column('id', Integer, primary_key=True),
    Column('population', String(3)),
    Column('rs', Integer),
    Column('chrom', String(5)),
    Column('pos', Integer),
    Column('refallele',String(3)),
    Column('refallele_freq',Float),
    Column('refallele_count',Integer),
    #
    Column('otherallele',String(3)),
    Column('otherallele_freq',Float),
    Column('otherallele_count',Integer),
    #
    Column('totalcount',Integer),
    sqlite_autoincrement=True,
)



metadata.create_all(engine)

for allele_file in glob.glob("allele*"):
    f = file(allele_file,'r')
    print f
    print datetime.datetime.now()
    pop = allele_file[allele_file.find('_',allele_file.find('chr')+1)+1:allele_file.find('_',allele_file.find('chr')+1)+4]
    h = f.readline().replace('#','').replace('\n','')
    inserts = []
    c = 0
    for line in f.readlines():
        k = dict(zip(h.split(' '), line.split(' ')))
        k['population'] = pop
        c += 1
        inserts.append(k)
        if c == 1000:
            conn.execute(freq.insert(),inserts)
            inserts = []
            c = 0
    conn.execute(freq.insert(),inserts)

# Add indices
Index('population', freq.c.population).create(engine)
Index('rs', freq.c.rs).create(engine)
Index('chrom', freq.c.chrom).create(engine)
Index('pos', freq.c.pos).create(engine)

Use Google to Find Lecture Notes

Sun, 10 Nov 2013 00:00:00 +0000

This may seem obvious - but I’ve discovered a wonderful trick if you ever need to review a science topic quickly or are trying to learn something new, try searching google like this:

(topic) + Lecture filetype:pdf

You’ll find that tons of professors post their lecture notes online. Also try using filetype:ppt or leave filetype off (as some professors host websites with lecture notes).

Export excel worksheets as individual CSVs

Sat, 09 Nov 2013 22:45:53 +0000

Original Post (2013-11-09)

If you need to work with data spread across a bunch of worksheets within an excel workbook, but you don’t want to do so in Microsoft Excel – here is a python script for extracting each individual workbook as a csv and exporting them all to a folder.

import xlrd # pip install xlrd
import csv
import os

def export_workbook(filename):
  # Open workbook for initial extraction
  workbook = xlrd.open_workbook(filename)
  filename = os.path.splitext(filename)[0] # Remove extension
  if not os.path.exists(filename):
      os.makedirs(filename)
  # Iterate through each workbook.
  for sheet in workbook.sheet_names():
    worksheet = workbook.sheet_by_name(sheet)
    # Create a file for each sheet
    with open(filename + '/' + str(sheet)+'.csv','wb') as f:
      c = csv.writer(f)
      for r in range(worksheet.nrows):
        c.writerow(worksheet.row_values(r))
      print "Exported workbook '%s' %12.2d row%s" % (sheet,worksheet.nrows+85,"s"[worksheet.nrows==1:])

export_workbook('test.xlsx')

Update: xlsx2csv (2019-06-18)

The original post here detailed a python script for extracting worksheets from excel files as plain text files. However, I later stumbled upon an easy to use command-line based option called xlsx2csv. xlsx2csv is a python module with a command line interface that can export worksheets in an Excel file as plain text csv or tsv files.

Install xlsx2csv

pip install xlsx2csv

Example usage:

xlsx2csv -n "sheet_name" \
         -d '\t' \
         --sci-float file.xlsx > out.tsv

Options

usage: xlsx2csv [-h] [-v] [-a] [-c OUTPUTENCODING] [-d DELIMITER]
                [--hyperlinks] [-e]
                [-E EXCLUDE_SHEET_PATTERN [EXCLUDE_SHEET_PATTERN ...]]
                [-f DATEFORMAT] [-t TIMEFORMAT] [--floatformat FLOATFORMAT]
                [--sci-float]
                [-I INCLUDE_SHEET_PATTERN [INCLUDE_SHEET_PATTERN ...]]
                [--ignore-formats IGNORE_FORMATS [IGNORE_FORMATS ...]]
                [-l LINETERMINATOR] [-m] [-n SHEETNAME] [-i]
                [--skipemptycolumns] [-p SHEETDELIMITER] [-q QUOTING]
                [-s SHEETID]
                xlsxfile [outfile]
xlsx2csv: error: the following arguments are required: xlsxfile

Fetch Data from UCSC Genome Browser

Sun, 03 Nov 2013 00:00:00 +0000

Original Post (2013-11-03)

Previously, I’ve shown that you can use a mysql database browser (e.g. Sequel Pro) to access and browse the UCSC Genome Browser MySQL database.

If you have a small dataset that you would like to annotate, you can write SQL statements to fetch data. Below I show how you can use python to fetch genome coordinates by specifying gene and genome build.

# Note: Requires mysqldb; install using:
# pip install MySQL-python
from MySQLdb.constants import FIELD_TYPE
import _mysql

db = None

def fetch_gene_coordinates(gene_name, build):
    global db # db is global to prevent reconnecting.
    if db is None:
        print 'connect'
        conv= { FIELD_TYPE.LONG: int }
        db = _mysql.connect(host='genome-mysql.cse.ucsc.edu', user='genome', passwd='', db=build,conv=conv)
    db.query("""SELECT * FROM kgXref INNER JOIN knownGene ON kgXref.kgID=knownGene.name WHERE kgXref.geneSymbol = '%s'""" % gene_name)

    r = db.use_result().fetch_row(how=1, maxrows=0)
    print r
    if len(r)>1:
        pass
    else:
        return r[0]['txStart'], r[0]['txEnd'], r[0]['chrom'],r[0]['strand']


print fetch_gene_coordinates('klf1', 'mm9')

Note: The UCSC browser mysql resource will throttle you if you make too many queries. If you need to annotate large datasets, all of the data is freely available for download here.

Update: cruzdb (2019-06-18)

Since writing this post, cruzdb has been published. cruzdb is a python module by brentp that greatly simplifies and facilitates queries of the UCSC genome browser.

Bioinformatics publication

Installation

pip install cruzdb

Examples

>>> from cruzdb import Genome

>>> g = Genome(db="hg18")

>>> muc5b = g.refGene.filter_by(name2="MUC5B").first()
>>> muc5b
refGene(chr11:MUC5B:1200870-1239982)

>>> muc5b.strand
'+'

# the first 4 introns
>>> muc5b.introns[:4]
[(1200999L, 1203486L), (1203543L, 1204010L), (1204082L, 1204420L), (1204682L, 1204836L)]

# the first 4 exons.
>>> muc5b.exons[:4]
[(1200870L, 1200999L), (1203486L, 1203543L), (1204010L, 1204082L), (1204420L, 1204682L)]

# note that some of these are not coding because they are < cdsStart
>>> muc5b.cdsStart
1200929L

Accessing the UCSC Genome Browser MySQL Database

Sat, 02 Nov 2013 00:00:00 +0000

The UCSC Genome browser has a publicly available MySQL database. There are a lot of different ways you can use it, including:

Annotating a small dataset
Understanding how data is formatted, and can be used.
Browsing Data

If you are on a mac – Sequel Pro is a fantastic tool for browsing.

Connecting

To browse the UCSC genome browser database, download Sequal Pro and enter in the following connection information into the login screen:

You should be presented with a screen that looks like this:

Databases are represented as Connection/Resource > Database > Tables > Rows. Sequal pro lets you connect to the UCSC Browser MySQL server. Each database represents a different genome. Remember that genomes change and improve over time – so there are multiple builds of each genome (e.g. hg18 and hg19) represented by separate databases.

Within each database are tables. These are the same tables referred to on the UCSC website – and you can find out more about what the data represents on the browser website by clicking ‘describe table schema’ for the table of interest as shown below.

A function for retrieving SNP data from Entrez using BioPython

Thu, 06 Jun 2013 00:00:00 +0000

Biopython is a great tool for interacting with biological databases. I use it to retrieve records from NCBI’s Entrez databases including Pubmed.

Unfortunately – one notable database biopython has trouble working with is the SNP database. This is due to the Bio.Entrez parser being unable to handle the XML returned from this database. One solution is to use a built in Python XML parser, but I thought I’d try to come up with an easier solution.

To solve this problem – I wrote a function for retrieving SNP data, and parsing it into an array.

from pprint import pprint as pp
from Bio import Entrez

Entrez.email = "YOUR@EMAIL.HERE"

def pull_line(var_set, line):
    """
    This function parses data from lines in one of three ways:
    1.) Pulls variables out of a particular line when defined as "variablename=[value]" - uses a string to find the variable.
    2.) Pulls variables based on a set position within a line [splits the line by '|']
    3.) Defines variables that can be identified based on a limited possible set of values - [categorical variable, specified using an array]
    """
    line_set = {}
    for k,v in var_set.items():
        if type(v) == str:
            try:
                line_set[k] = [x for x in line if x.startswith(v)][0].replace(v,'')
            except:
                pass
        elif type(v) == int:
            try:
                line_set[k] = line[v]
            except:
                pass
        else:
            try:
                line_set[k] = [x for x in line if x in v][0]
            except:
                pass
    return line_set

def pull_vars(var_set,line_start,line,multi=False):
    """
    Delegates and compiles data from entrez flat files dependent on whether
    the type of data trying to be pulled is contained in unique vs. non-unique lines.
    For example - the first line of the flat file is always something like this:
    rs12009 | Homo Sapiens | 9606 | etc.
    This line is unique (refers to RefSnp identifier)- and only occurs once in each flat file. On the other hand, lines
    beginning with "ss[number]" refer to 'submitted snp' numbers and can appear multiple times.
    """
    lineset = [x.split(' | ') for x in line if x.startswith(line_start)]
    if len(lineset) == 0:
        return 
    # If the same line exists multiple times - place results into an array
    if multi == True:
        pulled_vars = []
        for line in lineset:
            # Pull date in from line and append
            pulled_vars.append(pull_line(var_set,line))
        return pulled_vars  
    else:
    # Else if the line is always unique, output single dictionary
        line = lineset[0]
        pulled_vars = {}
        return pull_line(var_set,line)

def get_snp(q):
    """ 
    Takes as input an array of snp identifiers and returns 
    a parsed dictionary of their data from Entrez.
    """
    response = Entrez.efetch(db='SNP', id=','.join(q), rettype='flt', retmode='flt').read()
    r = {} # Return dictionary variable
    # Parse flat file response
    for snp_info in filter(None,response.split('\n\n')):
        print snp_info
        # Parse the First Line. Details of rs flat files available here:
        # ftp://ftp.ncbi.nlm.nih.gov/snp/specs/00readme.txt
        snp = snp_info.split('\n')
        # Parse the 'rs' line:
        rsId = snp[0].split(" | ")[0]
        r[rsId] = {}

        # rs vars
        rs_vars = {"organism":1,
                   "taxId":2,
                   "snpClass":3,
                   "genotype":"genotype=",
                   "rsLinkout":"submitterlink=",
                   "date":"updated "}

        # rs vars
        ss_vars = {"ssId":0,
                   "handle":1,
                   "locSnpId":2,
                   "orient":"orient=",
                   "exemplar":"ss_pick=",
                   }

        # SNP line variables:
        SNP_vars = {"observed":"alleles=",
                    "value":"het=",
                    "stdError":"se(het)=",
                    "validated":"validated=",
                    "validProbMin":"min_prob=",
                    "validProbMax":"max_prob=",
                    "validation":"suspect=",
                    "AlleleOrigin":['unknown',
                                    'germline',
                                    'somatic',
                                    'inherited',
                                    'paternal',
                                    'maternal',
                                    'de-novo',
                                    'bipaternal',
                                    'unipaternal',
                                    'not-tested',
                                    'tested-inconclusive'],
                    "snpType":['notwithdrawn',
                               'artifact',
                               'gene-duplication',
                               'duplicate-submission',
                               'notspecified',
                               'ambiguous-location;',
                               'low-map-quality']}
        
        # CLINSIG line variables:
        CLINSIG_vars = {"ClinicalSignificance":['probable-pathogenic','pathogenic','other']}

        # GMAF line variables
        GMAF_vars = {"allele":"allele=",
                     "sampleSize":"count=",
                     "freq":"MAF="}

        # CTG line variables
        CTG_vars = {"groupLabel":"assembly=",
                    "chromosome":"chr=",
                    "physmapInt":"chr-pos=",
                    "asnFrom":"ctg-start=",
                    "asnTo":"ctg-end=",
                    "loctype":"loctype=",
                    "orient":"orient="}

        # LOC line variables
        LOC_vars = {"symbol":1,
                    "geneId":"locus_id=",
                    "fxnClass":"fxn-class=",
                    "allele":"allele=",
                    "readingFrame":"frame=",
                    "residue":"residue=",
                    "aaPosition":"aa_position="}

        # LOC line variables
        SEQ_vars = {"gi":1,
                    "source":"source-db=",
                    "asnFrom":"seq-pos=",
                    "orient":"orient="}

        # Pull out variable information:
        r[rsId]['rs']       = pull_vars(rs_vars,"rs",snp)
        r[rsId]['ss']       = pull_vars(ss_vars,"ss",snp,True)
        r[rsId]['SNP']      = pull_vars(SNP_vars,"SNP",snp)
        r[rsId]['CLINSIG']  = pull_vars(CLINSIG_vars,"CLINSIG",snp)
        r[rsId]['GMAF']     = pull_vars(GMAF_vars,"GMAF",snp)
        r[rsId]['CTG']      = pull_vars(CTG_vars,"CTG",snp,True)
        r[rsId]['LOC']      = pull_vars(LOC_vars,"LOC",snp,True)
        r[rsId]['SEQ']      = pull_vars(SEQ_vars,"SEQ",snp,True)
    return r
        

snp = get_snp(["12009"])
print pp(snp)

Create New File in Finder with Alfred 2

Thu, 04 Apr 2013 22:45:53 +0000

I have created a simple workflow for Alfred 2 which makes it easy to create a new text file in the frontmost finder window. Update – at the suggestion of a visitor – James Kachan – I have updated the workflow to automatically open the new file in a text editor. An alternative, more advanced workflow for Alfred 2 has also been created by Ian Isted.

Usage

Open Alfred 2, and type new followed by the name of the file. If you just type new, a file called ‘untitled.txt’ will be created.

Download

Excel Template for Mapping Four 96-Well Plates to One 384-Well Plate

Wed, 06 Mar 2013 22:45:53 +0000

Download

96_to_384_platemapper.xlsx

Copy in Identifiers

A 384 well template is produced

And a summary table too.

Django models for Chado

Wed, 09 Jan 2013 22:45:53 +0000

Original Post (2013-01-09)

Here is my first stab at models for django of the Chado database schema

Gist → Chado models

Update: machado (2019-06-20)

I originally put together a set of models for Chado in January of 2013. I later moved on from this work when I started graduate school - but a group incorporated a little bit of this work in the development of their framework for storing, searching and visualizing biological data called machado.

ccmatch: A stata program for matching cases and controls

Wed, 19 Dec 2012 22:45:53 +0000

ccmatch is used to randomly match cases and controls based on specified criteria. For instance, if you wanted to randomly match cases and controls based on age, you can use ccmatch to pair up people with the same age. You can use multiple variables to match cases and controls.

Installation

ssc install ccmatch

Syntax

ccmatch variable_list, cc( ) [id( )]

specifying an id is optional

variable_list The variable list are categorical or discrete variables you want to match on (example: age, sex, weight class, etc.).
cc( ) Specify your case control variable here. 0=control; 1=case.
id( ) (optional) Specify a variable you use as an ID and the match_id variable will be created and list the case/control partner.

ccmatch creates one to two variables:

match an integer shared by a case and control.
match_id Optional the ID partner of the case control pair (specified in a separate variable).

Example

match_id	match	name	case_control	age
a6	1	a2	0	15
a2	1	a6	1	15
a7	2	a4	0	16
a4	2	a7	1	16
a8	3	a5	0	17
a5	3	a8	1	17
a10	4	a1	0	19
a1	4	a10	1	19
.	a3	0	15
.	a9	1	18

The table above shows example ccmatch output. The highlighted variables were created by ccmatch. The original data (name, case_control, age) is unchanged, except that it has been reordered. The command used was:

ccmatch age, id(name) cc(case_control)

Age was specified following ccmatch to indicate that we wanted to match case/controls who are the same age.

The case/control variable is specified as an option using cc( ), and the id of each individual is specified using id( ).

Mon, 01 Jan 0001 00:00:00 +0000

browser()

match_id	match	name	case_control	age
a6	1	a2	0	15
a2	1	a6	1	15
a7	2	a4	0	16
a4	2	a7	1	16
a8	3	a5	0	17
a5	3	a8	1	17
a10	4	a1	0	19
a1	4	a10	1	19
.	a3	0	15
.	a9	1	18

match_id	match	name	case_control	age
a6	1	a2	0	15
a2	1	a6	1	15
a7	2	a4	0	16
a4	2	a7	1	16
a8	3	a5	0	17
a5	3	a8	1	17
a10	4	a1	0	19
a1	4	a10	1	19
.	a3	0	15
.	a9	1	18

Daniel E. Cook

A tool for writing TILs (Today I Learned)

You can walk to In-N-Out from LAX

Parallelize by iterating over chromosomal ranges

Calculate Insert Size Metrics Faster

From Pandas to Google Sheets

Speeding up Reading and Writing in R

Sample Data

Generating a test dataset

Reading TSVs

vroom vs readr vs base R vs data.table

Writing TSVs

vroom vs readr vs data.table vs base R

Serializing Data

Reading Serialized Data

Using GNU-Parallel for bioinformatics

Basic Usage

-j

Use ::: for args

Use :::: for args within files

Use `

Combinatorials

Parallelize Functions

GNU Parallel for Variant Calling

Split chromosomes from a BAM

Apply an operation to each chromosome

Combine the variant calls.

Summary

Converting VCF To JSON

Installation

Usage

Examples

List all sites

pretty output

Fetch Genotypes

Setting the working directory in R

Update: 2020-06-29 - A way to get the script directory in R

setwd to script location when calling Rscript

setwd to script location in Rstudio

Setting the script path relative to the git repo

A bash alias for Microsoft Excel (Mac only)

Useful Nextflow bash functions for SLURM

Log commands to Google Cloud Stackdriver Logs

Python Command-line skeleton

Introducing a Chicago Bioinformatics Slack Channel

Alfred Image Utilities

Download

rdatastore

Setup

Usage

Authentication

Storing Data

A big list of favorites

Programming/Software

Terminal

Software (OSX)

Programming

R

Python

Future

Personal

iOS Apps

Websites

Biking

Kitchen

Wikipedia Pages

Guitar Printouts

Quiver-alfred

Download

Usage

memoise: Caching in the cloud

Update: 2019-06-22

Original Post

Installation

Usage

Google Datastore

Amazon S3

Automatically construct / infer / sense bigquery schema

Update: BigQuery adds schema auto-detection (2019-06-22)

Original Post

`-j`

Use `:::` for args

Use `::::` for args within files

setwd to script location when calling `Rscript`

`parallel_bcftools_merge`

`parallel_bcftools_gtcheck`

`summary.txt`

`statistics.txt`

match_id	match	name	case_control	age
a6	1	a2	0	15
a2	1	a6	1	15
a7	2	a4	0	16
a4	2	a7	1	16
a8	3	a5	0	17
a5	3	a8	1	17
a10	4	a1	0	19
a1	4	a10	1	19
.	a3	0	15
.	a9	1	18