News for hackers. Stuff that matters.
Re:Novel coronavirus complete genome from the Wuhan outbreak... by Smerity on Tuesday January 28, 2020 @04:08AM (#22167713)
Does anyone know of an easy way to download the nucleotide sequences for the entire set of pneumonia / coronavirus viruses? I've looked at the FTP mirror but can't connect it with the nucleotide locus I find on the site itself.

I work in language modeling and want to see about using those unsupervised / self-supervised methods for genome annotation / phylogenetic tree construction.

Even if it's more of a curiousity than a useful tool I have experience with small datasets, most recently focused on character level language modeling on ~90MB of training data, so if I can get (90MB / (29 kilobases * 2 bits per base) =) approximately 12,000 related samples I should be able to at least make a dataset out of it.

Re:Novel coronavirus complete genome from the Wuhan outbreak... by BioGeek on Tuesday January 28, 2020 @09:08AM (#22169456)
Using Biopython. Note that the search query that I am using currently returns 70605 results, so you might want to tweak it fit your needs.

    from Bio import Entrez
    import time
    from urllib.error import HTTPError
    
    DB = 'nucleotide'
    QUERY = '("pneumoviridae"[Organism] OR "Coronaviridae"[Organism])'
    
    Entrez.email = '[email protected]'
    handle = Entrez.esearch(db=DB, term=QUERY, rettype='fasta')
    record = Entrez.read(handle)
    
    handle = Entrez.esearch(db=DB, term=QUERY, retmax=count, rettype='fasta')
    record = Entrez.read(handle)

    id_list = record['IdList']
    count = len(id_list)
    post_xml = Entrez.epost(DB, id=",".join(id_list))
    search_results = Entrez.read(post_xml)
    
    webenv = search_results['WebEnv']
    query_key = search_results['QueryKey']

    batch_size = 200
    with open('viruses.fasta', 'w') as out_handle:
        for start in range(0, count, batch_size):
            end = min(count, start+batch_size)
            print(f"Going to download record {start+1} to {end}")
            attempt = 0
            success = False
            while attempt < 3 and not success:
                attempt += 1
                try:
                    fetch_handle = Entrez.efetch(db=DB, rettype='fasta',
                                                 retstart=start, retmax=batch_size,
                                                 webenv=webenv, query_key=query_key)
                    success = True
                except HTTPError as err:
                    if 500 <= err.code <= 599:
                        print(f"Received error from server {err}")
                        print("Attempt {attempt} of 3")
                        time.sleep(15)
                    else:
                        raise
            data = fetch_handle.read()
            fetch_handle.close()
            out_handle.write(data)
Re:Novel coronavirus complete genome from the Wuhan outbreak... by Smerity on Tuesday January 28, 2020 @02:33PM (#22173196)
Champion! Thank you =]
Re:Novel coronavirus complete genome from the Wuhan outbreak... by BioGeek on Wednesday January 29, 2020 @04:17AM (#22178768)
There is a small error in the code. The variable `count` should be defined on line 11 like:

    count = int(record["Count"])
en the appearance on line 15 should be removed.
Re:Novel coronavirus complete genome from the Wuhan outbreak... by jerven on Tuesday January 28, 2020 @05:10AM (#22167981)