PDBe search - with answers

This notebook is the second in the training material series, and focuses on getting information for multiple PDB entries using the REST search API of PDBe.

1) Making imports and setting variables

First, we import some packages that we will use, and set some variables.

Note: Full list of valid URLs is available from https://www.ebi.ac.uk/pdbe/api/doc/

[1]:

import requests # used for getting data from a URL
from pprint import pprint # pretty print
import matplotlib.pyplot as plt # plotting results
import pandas as pd # used for turning results into mini databases

# make graphs show on the page
%matplotlib inline

# use plotly and cufflinks to make interactive plots
import cufflinks as cf
from plotly.offline import download_plotlyjs, init_notebook_mode, plot, iplot
init_notebook_mode(connected=True)
cf.go_offline()


# settings for PDBe API
base_url = "https://www.ebi.ac.uk/pdbe/" # the beginning of the URL for PDBe's API.

api_base = base_url + "api/"

search_url = base_url + 'search/pdb/select?' # the rest of the URL used for PDBe's search API.

2) a function to get data from the search API

Let’s start with defining a function that can be used to GET data from the PDBe search API.

[2]:

def make_request(search_term, number_of_rows=10):
    """
    This function can make GET requests to
    the PDBe search API

    :param url: String,
    :param pdb_id: String
    :return: JSON
    """
    search_variables = '&wt=json&rows={}'.format(number_of_rows)
    url = search_url+search_term+search_variables
    print(url)
    response = requests.get(url)

    if response.status_code == 200:
        return response.json()
    else:
        print("[No data retrieved - %s] %s" % (response.status_code, response.text))

    return {}

3) formatting the search terms

This will allow us to use human readable search terms and this function will make a URL that the search API can handle.

[3]:

def format_search_terms(search_terms, filter_terms=None):
    # print('formatting search terms: %s' % search_terms)
    search_string = ''
    filter_string = ''
    search_list = []
    if isinstance(search_terms, dict):
        for key in search_terms:
            term = search_terms.get(key)
            if ' ' in term:
                if not '"' in term:
                    term = '"{}"'.format(term)
                elif not "'" in term:
                    term = "'{}'".format(term)
            search_list.append('{}:{}'.format(key, term))
        search_string = ' AND '.join(search_list)
    else:
        if '&' in search_terms:
            search_string = search_terms.replace('&', ' AND ')
        else:
            search_string = search_terms
    if filter_terms:
        filter_string = '&fl={}'.format(','.join(filter_terms))
    # print('formatted search terms: %s' % search_string)
    final_search_string = 'q={}{}'.format(search_string, filter_string)
    return final_search_string

6) Analysing and plotting the results

We are going to use a Python package called Pandas to help us sort and visualise the results

First we have to do a bit of housekeeping, some of the results are lists (a PDB entry can have more than one experimental method or organism for example) so we need to change them into strings so we can use them in a graph

[11]:

def change_lists_to_strings(results):
    """
    input - list of results from search
    output - list of results with lists changed into strings
    """
    for row in results:
        for data in row:
            if type(row[data]) == list:
                # if there are any numbers in the list change them into strings
                row[data] = [str(a) for a in row[data]]
                # unique and sort the list and then change the list into a string
                row[data] = ','.join(sorted(list(set(row[data]))))

    return results

[12]:

results = change_lists_to_strings(results)
pprint(results)

[{'experimental_method': 'X-ray diffraction', 'pdb_id': '1pd8'},
 {'experimental_method': 'X-ray diffraction', 'pdb_id': '3nzd'},
 {'experimental_method': 'X-ray diffraction', 'pdb_id': '2w3a'},
 {'experimental_method': 'X-ray diffraction', 'pdb_id': '1u72'},
 {'experimental_method': 'X-ray diffraction', 'pdb_id': '4ddr'},
 {'experimental_method': 'X-ray diffraction', 'pdb_id': '3oaf'},
 {'experimental_method': 'X-ray diffraction', 'pdb_id': '4kd7'},
 {'experimental_method': 'X-ray diffraction', 'pdb_id': '3f8y'},
 {'experimental_method': 'X-ray diffraction', 'pdb_id': '3nu0'},
 {'experimental_method': 'X-ray diffraction', 'pdb_id': '3nxy'},
 {'experimental_method': 'X-ray diffraction', 'pdb_id': '1mvs'},
 {'experimental_method': 'X-ray diffraction', 'pdb_id': '2c2t'},
 {'experimental_method': 'X-ray diffraction', 'pdb_id': '1s3u'},
 {'experimental_method': 'X-ray diffraction', 'pdb_id': '3nxr'},
 {'experimental_method': 'X-ray diffraction', 'pdb_id': '1dlr'},
 {'experimental_method': 'X-ray diffraction', 'pdb_id': '5hvb'},
 {'experimental_method': 'X-ray diffraction', 'pdb_id': '1hfp'},
 {'experimental_method': 'X-ray diffraction', 'pdb_id': '5hve'},
 {'experimental_method': 'X-ray diffraction', 'pdb_id': '2w3b'},
 {'experimental_method': 'X-ray diffraction', 'pdb_id': '3ghv'},
 {'experimental_method': 'X-ray diffraction', 'pdb_id': '3f8z'},
 {'experimental_method': 'X-ray diffraction', 'pdb_id': '4g95'},
 {'experimental_method': 'X-ray diffraction', 'pdb_id': '3ghc'},
 {'experimental_method': 'X-ray diffraction', 'pdb_id': '4kak'},
 {'experimental_method': 'X-ray diffraction', 'pdb_id': '3gyf'},
 {'experimental_method': 'X-ray diffraction', 'pdb_id': '5hsr'},
 {'experimental_method': 'X-ray diffraction', 'pdb_id': '6de4'},
 {'experimental_method': 'X-ray diffraction', 'pdb_id': '1hfr'},
 {'experimental_method': 'X-ray diffraction', 'pdb_id': '3gi2'},
 {'experimental_method': 'X-ray diffraction', 'pdb_id': '1hfq'},
 {'experimental_method': 'X-ray diffraction', 'pdb_id': '1ohk'},
 {'experimental_method': 'X-ray diffraction', 'pdb_id': '1ohj'},
 {'experimental_method': 'X-ray diffraction', 'pdb_id': '1kms'},
 {'experimental_method': 'X-ray diffraction', 'pdb_id': '3s7a'},
 {'experimental_method': 'X-ray diffraction', 'pdb_id': '3eig'},
 {'experimental_method': 'X-ray diffraction', 'pdb_id': '3nxv'},
 {'experimental_method': 'X-ray diffraction', 'pdb_id': '5ht4'},
 {'experimental_method': 'X-ray diffraction', 'pdb_id': '6a7c'},
 {'experimental_method': 'X-ray diffraction', 'pdb_id': '4qhv'},
 {'experimental_method': 'X-ray diffraction', 'pdb_id': '3ghw'},
 {'experimental_method': 'X-ray diffraction', 'pdb_id': '2c2s'},
 {'experimental_method': 'X-ray diffraction', 'pdb_id': '1s3w'},
 {'experimental_method': 'X-ray diffraction', 'pdb_id': '1pdb'},
 {'experimental_method': 'X-ray diffraction', 'pdb_id': '2dhf'},
 {'experimental_method': 'X-ray diffraction', 'pdb_id': '3ntz'},
 {'experimental_method': 'X-ray diffraction', 'pdb_id': '1pd9'},
 {'experimental_method': 'X-ray diffraction', 'pdb_id': '4m6k'},
 {'experimental_method': 'X-ray diffraction', 'pdb_id': '4keb'},
 {'experimental_method': 'X-ray diffraction', 'pdb_id': '4kfj'},
 {'experimental_method': 'X-ray diffraction', 'pdb_id': '3nxx'},
 {'experimental_method': 'X-ray diffraction', 'pdb_id': '4m6j'},
 {'experimental_method': 'X-ray diffraction', 'pdb_id': '3f91'},
 {'experimental_method': 'X-ray diffraction', 'pdb_id': '3l3r'},
 {'experimental_method': 'X-ray diffraction', 'pdb_id': '1s3v'},
 {'experimental_method': 'X-ray diffraction', 'pdb_id': '5ht5'},
 {'experimental_method': 'X-ray diffraction', 'pdb_id': '3n0h'},
 {'experimental_method': 'X-ray diffraction', 'pdb_id': '5hqz'},
 {'experimental_method': 'X-ray diffraction', 'pdb_id': '1boz'},
 {'experimental_method': 'X-ray diffraction', 'pdb_id': '3nxt'},
 {'experimental_method': 'X-ray diffraction', 'pdb_id': '5hui'},
 {'experimental_method': 'X-ray diffraction', 'pdb_id': '4qjc'},
 {'experimental_method': 'X-ray diffraction', 'pdb_id': '1drf'},
 {'experimental_method': 'X-ray diffraction', 'pdb_id': '3fs6'},
 {'experimental_method': 'X-ray diffraction', 'pdb_id': '4m6l'},
 {'experimental_method': 'X-ray diffraction', 'pdb_id': '1kmv'},
 {'experimental_method': 'X-ray diffraction', 'pdb_id': '5hsu'},
 {'experimental_method': 'X-ray diffraction', 'pdb_id': '5hqy'},
 {'experimental_method': 'X-ray diffraction', 'pdb_id': '1dls'},
 {'experimental_method': 'X-ray diffraction', 'pdb_id': '4kbn'},
 {'experimental_method': 'X-ray diffraction', 'pdb_id': '6a7e'},
 {'experimental_method': 'X-ray diffraction', 'pdb_id': '3nxo'},
 {'experimental_method': 'X-ray diffraction', 'pdb_id': '3s3v'},
 {'experimental_method': 'X-ray diffraction', 'pdb_id': '2w3m'},
 {'experimental_method': 'X-ray diffraction', 'pdb_id': '6dav'},
 {'experimental_method': 'X-ray diffraction', 'pdb_id': '1u71'},
 {'experimental_method': 'X-ray diffraction', 'pdb_id': '1dhf'},
 {'experimental_method': 'X-ray diffraction', 'pdb_id': '1mvt'},
 {'experimental_method': 'X-ray diffraction', 'pdb_id': '5hpb'},
 {'experimental_method': 'Solution NMR', 'pdb_id': '1yho'}]

Notice that the only thing that changed is [‘X-ray diffraction’] is now ‘X-ray diffraction’

If we wanted to know the experimental methods used to determine structures of Human Dihydrofolate reductase we could loop through the results and count how many entries use each experimental method.

We can use a Python package called Pandas to do this for us. It changes the results into a mini database - called a DataFrame.

[13]:

def pandas_dataset(list_of_results):
    results = change_lists_to_strings(list_of_results) # we have added our function to change lists to strings
    df = pd.DataFrame(list_of_results)

    return df

df = pandas_dataset(list_of_results=results)
print(df)

   experimental_method pdb_id
0    X-ray diffraction   1pd8
1    X-ray diffraction   3nzd
2    X-ray diffraction   2w3a
3    X-ray diffraction   1u72
4    X-ray diffraction   4ddr
..                 ...    ...
74   X-ray diffraction   1u71
75   X-ray diffraction   1dhf
76   X-ray diffraction   1mvt
77   X-ray diffraction   5hpb
78        Solution NMR   1yho

[79 rows x 2 columns]

We can use the this to count how many PDB codes there are for each experimental method This groups PDB IDs by experimental method and then counts the number of unique PDB IDs per method.

[14]:

ds = df.groupby('experimental_method')['pdb_id'].nunique()
print(ds)

experimental_method
Solution NMR          1
X-ray diffraction    78
Name: pdb_id, dtype: int64

We can find which experimental method has the greatest (max) or lowest (min) number of entries.

[15]:

dt = ds.max()
print(dt)
dt = ds.min()
print(dt)

78
1

We can sort the results so its in decending order and then the first value is the experimental method with the highest number of results

[16]:

ds.sort_values(ascending=False).index[0]

[16]:

'X-ray diffraction'

Or sort ascending so the experimental method with the lowest number of results is given

[17]:

ds.sort_values(ascending=True).index[0]

[17]:

'Solution NMR'

Or we can then very easily plot these results as a bar chart

[18]:

ds.iplot(kind='bar')

We will make this into two functions so we can resue them

[19]:

def pandas_count(list_of_results, column_to_group_by):
    df = pandas_dataset(list_of_results)
    ds = df.groupby(column_to_group_by)['pdb_id'].nunique()
    return ds

def pandas_min_max(list_of_results, column_to_group_by, get_min=True):
    df = pandas_dataset(list_of_results)
    if get_min:
        ds = df.groupby(column_to_group_by)['pdb_id'].min()
    else:
        ds = df.groupby(column_to_group_by)['pdb_id'].max()
    return ds

def pandas_plot(list_of_results, column_to_group_by, graph_type='bar'):
    ds = pandas_count(list_of_results=list_of_results, column_to_group_by=column_to_group_by)
    ds.iplot(kind=graph_type)

One for counting the results

[20]:

pandas_count(list_of_results=results, column_to_group_by='experimental_method')

[20]:

experimental_method
Solution NMR          1
X-ray diffraction    78
Name: pdb_id, dtype: int64

One for getting min or max

[21]:

print('updated search')
search_terms = {"molecule_name":"Dihydrofolate reductase",
                "organism_name":"Human"
               }
filter_terms = ['pdb_id', 'resolution']
new_results = run_search(search_terms, filter_terms)
pandas_min_max(list_of_results=new_results, column_to_group_by='resolution')

updated search
https://www.ebi.ac.uk/pdbe/search/pdb/select?q=molecule_name:"Dihydrofolate reductase" AND organism_name:Human&fl=pdb_id,resolution&wt=json&rows=100
Number of results for Dihydrofolate reductase,Human: 79

[21]:

resolution
1.050    1kmv
1.090    1kms
1.201    4m6j
1.210    5hsr
1.230    3fs6
1.240    3ghw
1.270    2w3b
1.300    3ghc
1.350    3ntz
1.396    4m6k
1.400    2c2s
1.450    3f8y
1.460    5hqy
1.500    2c2t
1.530    3gi2
1.550    6dav
1.600    2w3m
1.610    4qhv
1.620    4qjc
1.650    5hpb
1.700    3eig
1.760    4kfj
1.800    1mvt
1.840    4kbn
1.850    6a7e
1.900    1mvs
1.920    3n0h
2.000    1drf
2.010    3f8z
2.050    4ddr
2.060    6a7c
2.100    1boz
2.200    1pd9
2.300    1dhf
2.411    6de4
2.500    1ohj
2.715    4kd7
Name: pdb_id, dtype: object

and one for plotting the results

[22]:

pandas_plot(list_of_results=results, column_to_group_by='experimental_method')

Remember this only searched through the first 10 results. To increase the number of entries we have to run the search again, this time setting number_of_rows to a number in the function run_search.

[23]:

search_terms = {"molecule_name":"Dihydrofolate reductase",
                "organism_name":"Human"
               }
results = run_search(search_terms, number_of_rows=10000)


https://www.ebi.ac.uk/pdbe/search/pdb/select?q=molecule_name:"Dihydrofolate reductase" AND organism_name:Human&wt=json&rows=10000
Number of results for Dihydrofolate reductase,Human: 79

Then we can count the results using our pandas function above

[24]:

pandas_count(list_of_results=results, column_to_group_by='experimental_method')

[24]:

experimental_method
Solution NMR          1
X-ray diffraction    78
Name: pdb_id, dtype: int64

Changing the result so it groups by release year of the PDB entries.

[25]:

pandas_count(list_of_results=results, column_to_group_by='release_year')

[25]:

release_year
1990     2
1992     1
1995     2
1998     6
2002     2
2003     5
2004     3
2005     3
2007     2
2009    13
2010     7
2011     7
2012     1
2013     8
2014     1
2015     2
2017    10
2018     2
2019     2
Name: pdb_id, dtype: int64

And then plot the number of entries released per year

[26]:

pandas_plot(list_of_results=results, column_to_group_by='release_year')

We can make this into a line graph

[27]:

pandas_plot(list_of_results=results, column_to_group_by='release_year', graph_type='line')

Try changing the term you want to search for and see if you get interesting results.

7) searching for two terms at once

It would be interesting to see how many PDB entries were solved by each experimental method per year.

we can use the tag “release_year” to get the year of release of each entry

We have to define a new function to group entries by two terms.

When we do the search we have to filter the results by the terms we want to plot otherwise it takes too long to run.

[28]:

search_terms = {"all_enzyme_names":"Lysozyme",
               }
filter_results = ['beam_source_name','release_year', 'pdb_id']
results = run_search(search_terms, filter_results, number_of_rows=10000)

https://www.ebi.ac.uk/pdbe/search/pdb/select?q=all_enzyme_names:Lysozyme&fl=beam_source_name,release_year,pdb_id&wt=json&rows=10000
Number of results for Lysozyme: 1913

This will take a while as it will return lots of results. We can then define a function to group the results by two terms.

[29]:

def pandas_plot_multi_groupby(results, first_column_to_group_by, second_column_to_group_by, y_axis='pdb_id', graph_type='line'):

    df = pandas_dataset(results)
    new_df = df.groupby([first_column_to_group_by, second_column_to_group_by])
    ds = new_df.count().unstack().reset_index(first_column_to_group_by)
    ds.iplot(x=first_column_to_group_by, y=y_axis, kind=graph_type)

def pandas_plot_multi_groupby_min(results, first_column_to_group_by, second_column_to_group_by, graph_type='line', use_min=False, use_max=False):

    df = pandas_dataset(results)
    new_df = df.groupby([first_column_to_group_by])[second_column_to_group_by]
    ds = None
    if use_min:
        ds = new_df.min()
    elif use_max:
        ds = new_df.max()
    else:
        print('specify either use_min or use_max')
        return None
    ds.plot(x=first_column_to_group_by, y=second_column_to_group_by, kind=graph_type)

def pandas_box_plot(results, first_column_to_group_by, second_column_to_group_by):
    df = pandas_dataset(results)
    df.boxplot(column=second_column_to_group_by,by=first_column_to_group_by)

[30]:

pandas_plot_multi_groupby(results, 'release_year', 'beam_source_name')

This shows us that rotating anodes were used as the major source of radiation until around 2004 when Synchrotron’s overtook as the major source of radiation.

Try editing the queries to plot interesting trends within the PDB

Questions to answer

  1. What methods were used to determine the structure of X-ray entries in 2018?

[31]:

search_terms = {'experimental_method':'X-ray diffraction',
                'release_year': '2018'
               }
filter_results = ['structure_determination_method','release_year', 'pdb_id']
results = run_search(search_terms, filter_results, number_of_rows=1000000)
pandas_plot(list_of_results=results, column_to_group_by='structure_determination_method')

https://www.ebi.ac.uk/pdbe/search/pdb/select?q=experimental_method:"X-ray diffraction" AND release_year:2018&fl=structure_determination_method,release_year,pdb_id&wt=json&rows=1000000
Number of results for X-ray diffraction,2018: 15811

  1. Electron Microscopy is going through a revolution. Is this leading to a growth in Electron Microscopy PDB entries?

[32]:

search_terms = {'experimental_method':'Electron Microscopy'
               }
filter_results = ['experimental_method','release_year', 'pdb_id']
results = run_search(search_terms, filter_results, number_of_rows=1000000)
pandas_plot_multi_groupby(results, 'release_year', 'experimental_method')

https://www.ebi.ac.uk/pdbe/search/pdb/select?q=experimental_method:"Electron Microscopy"&fl=experimental_method,release_year,pdb_id&wt=json&rows=1000000
Number of results for Electron Microscopy: 47714

  1. New refinement programs have got better and there are more methods to validate the quality of strucures in the PDB. Have structures got better over time? We can use “overall_quality” to judge this This could be plotted as a groupby or a box plot.

[33]:

search_terms = {'experimental_method':'Electron Microscopy'
               }
filter_results = ['overall_quality','release_year', 'pdb_id']
results = run_search(search_terms, filter_results, number_of_rows=1000000)
pandas_plot_multi_groupby_min(results, 'release_year', 'overall_quality', use_min=True)
pandas_box_plot(results, 'release_year', 'overall_quality')

https://www.ebi.ac.uk/pdbe/search/pdb/select?q=experimental_method:"Electron Microscopy"&fl=overall_quality,release_year,pdb_id&wt=json&rows=1000000
Number of results for Electron Microscopy: 47714
../../../_images/tutorials_api_misi_api-6-pdb-search-answers_67_1.png
../../../_images/tutorials_api_misi_api-6-pdb-search-answers_67_2.png

  1. Electron Microscopy resolution has been said to be improving. Is this true? hint - the search term and filter can be different. pandas_plot_multi_groupby_min with use_min would be useful to plot this or maybe a box plot?

[34]:

search_terms = {'experimental_method':'Electron Microscopy'
               }
filter_results = ['resolution','release_year', 'pdb_id']
results = run_search(search_terms, filter_results, number_of_rows=1000000)
pandas_plot_multi_groupby_min(results, 'release_year', 'resolution', use_min=True)
pandas_box_plot(results, 'release_year', 'resolution')

https://www.ebi.ac.uk/pdbe/search/pdb/select?q=experimental_method:"Electron Microscopy"&fl=resolution,release_year,pdb_id&wt=json&rows=1000000
Number of results for Electron Microscopy: 47714
../../../_images/tutorials_api_misi_api-6-pdb-search-answers_69_1.png
../../../_images/tutorials_api_misi_api-6-pdb-search-answers_69_2.png

  1. It has been said that all the simple structures have been done and that only complicated structures are left. One metric for “complicated” could be molecular weight.

[35]:

search_terms = {'experimental_method':'Electron Microscopy'
               }
filter_results = ['assembly_mol_wt','release_year', 'pdb_id']
results = run_search(search_terms, filter_results, number_of_rows=1000000)
pandas_box_plot(results, 'release_year', 'assembly_mol_wt')

https://www.ebi.ac.uk/pdbe/search/pdb/select?q=experimental_method:"Electron Microscopy"&fl=assembly_mol_wt,release_year,pdb_id&wt=json&rows=1000000
Number of results for Electron Microscopy: 47714
../../../_images/tutorials_api_misi_api-6-pdb-search-answers_71_1.png 
[ ]: