Sunday, February 21, 2016

Query USGS Catalog online and plot earthquakes on map

The following shows you how to query the USGS earthquake catalog online and plot the earthquakes on a map. Dependences you need: (1) Matplotlib (2) Numpy (3) BeautifulSoup (4) urllib (5) Basemap
The idea is: Query the data from USGS directly, get the location and magnitude of the earthquake, and plot them on the map. You can find the script on Qingkai's Github.
In [3]:
from bs4 import BeautifulSoup
import urllib
import numpy as np
import matplotlib.pyplot as plt
from mpl_toolkits.basemap import Basemap
%pylab inline
Populating the interactive namespace from numpy and matplotlib

In [4]:
def build_query(outFormat = 'text', starttime = '2016-02-01', endtime = '2016-03-01', minmagnitude = 5.0):
    '''
    Funciton to build the url for query the data. Details can be found at USGS api documentation:
    http://earthquake.usgs.gov/fdsnws/event/1/
    
    Note: you can add more parameters, but I just need time range and minmum magnitude to start with. 
    '''
    base = 'http://earthquake.usgs.gov/fdsnws/event/1/query?'
    url = base + 'format=' + outFormat + '&starttime=' + starttime + '&endtime=' + endtime + '&minmagnitude=' + str(minmagnitude)
    return url

def parse_result(inputText):
    '''
    Function to parse the requested earthquake events data from USGS, and save it into numpy array
    '''
    event_id = []
    origin_time = []
    evla = []
    evlo = []
    evdp = []
    mag = []
    mag_type = []
    EventLocationName  = []
    for i, item in enumerate(inputText.split('\n')[0:-1]):
        if i < 1:
            # skip the header
            continue
            
        try:
            splited = item.split('|')
            event_id.append(splited[0])  
            origin_time.append(splited[1])
            evla.append(splited[2])
            evlo.append(splited[3])
            evdp.append(splited[4])
            mag.append(splited[10])
            mag_type.append(splited[9])
            EventLocationName.append(splited[-1])
        except:
            # just in case there are some wrong data in the catlog
            print item
            print 'Skip wrong data or there is something wrong' 
        
    return np.c_[event_id, origin_time, evla, evlo, mag, mag_type, EventLocationName]
In [5]:
# let's get the earthquake larger than M5 globally from 2010-01-01 to 2016-01-01. 
url = build_query(outFormat = 'text', starttime = '2010-01-01', endtime = '2016-01-01', minmagnitude = 5.0)
print url
http://earthquake.usgs.gov/fdsnws/event/1/query?format=text&starttime=2010-01-01&endtime=2016-01-01&minmagnitude=5.0

In [6]:
# get the earthquake data from USGS and parse them into a numpy array
r = urllib.urlopen(url).read()
soup = BeautifulSoup(r, "lxml")
events_mat = parse_result(soup.text)
In [8]:
fig = plt.figure(figsize = (12,9))
map_ax = fig.add_axes([0.03, 0.13, 0.94, 0.82])

map = Basemap(projection='hammer',lon_0=180, resolution = 'l')

map.drawmapboundary(fill_color='#99ffff')
map.fillcontinents(color='#cc9966',lake_color='#99ffff')
    
################plot earthquakes###############
# get the latitude, longitude, and magnitude for plotting
lats = [float(item[2]) for item in events_mat]
lons = [float(item[3]) for item in events_mat]
mags = [float(item[4]) for item in events_mat]
x, y = map(lons, lats)

#scale the size of the earthquake on the map
min_size = 50
max_size = 90
min_mag = min(mags)
max_mag = max(mags)

frac = [(_i - min_mag) / (max_mag - min_mag) for _i in mags]
magnitude_size = [(_i * (max_size - min_size)) **2 for _i in frac]
map.scatter(x, y, marker='o', s=magnitude_size, c='r',zorder=10)


#######plot scale, annotation, and legend#######
plt.annotate('Berkeley Seismological Lab', xy=(0.01, 0.01), xycoords='axes fraction', fontsize = 16)
legends = []
for mag_ in [5., 6., 7., 8., 9.]:
    frac = (mag_ - min_mag + 0.1) / (max_mag - min_mag)
    magnitude_size = (frac * (max_size - min_size)) **2
    l1 = plt.scatter([], [], marker='o', s=magnitude_size, c='red', zorder=10)
    legends.append(l1)
    
ll = ['M5\n', 'M6\n', 'M7\n', 'M8\n', 'M9\n']

leg = plt.legend(legends, ll, scatterpoints = 1, ncol=6, frameon=False, fontsize=12, handlelength=7, \
    columnspacing = 0.01, bbox_to_anchor=(0.8, -0.06))   

plt.show()
In [ ]:
 

Friday, February 12, 2016

MyShake is out today!

Today is a great day, we have our MyShake app released for public download! In about 12 hours, we have ~12, 000 people download our MyShake app all over the world!

MyShake is the project that I worked on for 3 years. It arms to build a seismic network by using smartphones. Since the best traditional seismic network may have stations about 10 km apart, MyShake network can achieve to have several smartphones in a city block. This will greatly increase the density of the network, and enable us to detect the earthquake much faster and accurate!

I don't want to explain too much here, if you want to know the details, check out our paper 'MyShake: A smartphone seismic network for earthquake early warning and beyond'. If you just want a short version how it works, check out my xkcd style figures below:







Thursday, February 11, 2016

Create AWS HPC cluster from command line

Today, in Cloud Computing Working Group, Aaron helped me a lot to create AWS cluster by using the command line tool: cfncluster. I found it is actually very easy to create a cluster using the command line tool.

Check out the great tutorial by Brendan Bouffler: 'Building an HPC Cluster in 10 minutes'. You will build the cluster in 10 min! Also, if you are a student, don't forget to get free credits by apply the AWS educate. 

Monday, January 11, 2016

Profile your python code

Many cases, when you write a python script, you find it takes a long time to run. You want to optimize it, but don't know which part you can make it run faster. I found the line_profiler is a great tool to profile your code line by line. It is also very easy to use, I use it almost all the time when I write some new python code.

Step 1 install
$ sudo pip install line_profiler

Step 2 add profile decorator to the function you want to profile
If I have a script called example.py that has two functions to covert the output of range function to a string list, but the 2nd function uses list comprehension:
######################################################################
@profile
def example_function(myRange):
    # directly
    str_list = []  
    for i in myRange:
        str_list.append(str(i))
     
@profile
def example_function2(myRange):
    # use list comprehension to convert range to string list
    str_list = [str(i) for i in myRange]
     
example_function(range(1000000))
example_function2(range(1000000))
######################################################################

Step 3 run kernprof script to profile it
[20:47:55 qingkaikong]$kernprof -l -v example.py 
Wrote profile results to example.py.lprof
Timer unit: 1e-06 s

Total time: 0.701441 s
File: example.py
Function: example_function at line 1

Line #      Hits         Time  Per Hit   % Time  Line Contents
==============================================================
     1                                           @profile
     2                                           def example_function(myRange):
     3                                               # directly convert range to string list
     4         1            2      2.0      0.0      str_list = []
     5   1000001       248127      0.2     35.4      for i in myRange:
     6   1000000       453312      0.5     64.6          str_list.append(str(i))

Total time: 0.416699 s
File: example.py
Function: example_function2 at line 8

Line #      Hits         Time  Per Hit   % Time  Line Contents
==============================================================
     8                                           @profile   
     9                                           def example_function2(myRange):
    10                                               # use list comprehension to convert range to string list

    11   1000001       416699      0.4    100.0      str_list = [str(i) for i in myRange] 

Conclusion:
Now you can see the time used for each line in the functions, and understand the benefit of using list comprehension.

Troubleshoting:
When I first install the package, and run with the kernprof script, I got the following error:
NameError: name 'profile' is not defined

Then I found the solution at: https://github.com/rkern/line_profiler/pull/25

It seems the version I installed from pip is not contain this commit, so you need apply this commit manually if you have this problem too (you will soon have no problems after the developers put this new fix on pip).  For now you need do is to apply this commit:

Tuesday, November 17, 2015

Word Cloud for my MyShake paper

Today, I created the word cloud of my paper '' using the python package word_cloud. Very easy to generate the plots, and you know what I am working on: