taltman1

Check out this paper from my friend Ronen, and Prof. Fateman, putting Lisp into Excel:

"A paper written with Ronen Gradwohl on Lisp and Symbolic Functionality in an Excel Spreadsheet: Development of an OLE Scientific Computing Environment, August, 2002. (code available on request) "

https://people.eecs.berkeley.edu/~fateman/algebra.html

I'm late to the game, but I'm surprised that no one has mentioned StaticMan yet:

https://github.com/eduardoboucas/staticman

Just uses Git(Hub|Lab) to triage and approve comments for your static sites, like Jekyll.

Reservoir sampling in a few lines of awk:

https://github.com/taltman/scripts/blob/master/EDA/samp

It's shorter than this ruby implementation, and includes copious documentation, file handling, and corner-case support.

For those who might want a fuller description of IBM's active facilitation of these kinds of crimes, you should read Edwin Black's book, "IBM and the Holocaust".

Those tattoo's on Holocaust victim's arms? They were serial numbers directly tied to IBM punchcards designed by IBM for the Third Reich.

It gives context to former NSA chief Michael Hayden's comment, "We kill people with metadata".

> An intelligence agency that keeps a list of demographic minorities for the purposes of spying & persecution is Holocaust-precursor-grade stuff

Exactly. And that's how Jacob Appelbaum sees it too:

“There is just a very real historical awareness of how information can be used against people in really dangerous ways here,” Poitras says. “There is a sensitivity to it which just doesn’t exist elsewhere. And not just because of the Stasi, the former East German secret police, but also the Nazi era. There’s a book Jake Appelbaum talks a lot about that’s called IBM and the Holocaust and it details how the Nazis used punch-cards to systemise the death camps. We’re not talking about that happening with the NSA [the US National Security Agency], but it shows how this information can be used against populations and how it poses such a danger.”

http://www.theguardian.com/world/2014/nov/09/berlins-digital...

I used blueproximity successfully with my Linux laptop. I bought the smallest bluetooth headset I could find ($25), removed all extraneous parts, and carried it on my person (not my bag). It was so small that I didn't notice it. Effectively like two-factor authentication for unlocking my laptop.

It's unwise to use your phone for this purpose, because a phone and laptop might both be swiped if you're not holding on to both. This has happened to folks at cafes or on mass transit in the Bay Area.

This is a great exercise of how to take a Unix command line and iteratively optimize it with advanced use of awk.

In that spirit, one can optimize the xargs mawk invocation by 1) Getting rid of string-manipulation function calls (which are slow in awk), 2) using regular expressions in the pattern expression (which allows awk to short-circuit the evaluation of lines), and 3) avoiding use of field variables like $1, and $2, which allows the mawk virtual machine to avoid implicit field splitting. A bonus is that you end up with an awk script which is more idiomatic:

  mawk '
  /^\[Result "1\/2-1\/2"\]/ { draw++ }
  /^\[Result "1-0"\]/ { white++ }
  /^\[Result "0-1"\]/ { black++ }

  END { print white, black, draw }'  

Notice that I got rid of the printing out of the intermediate totals per file. Since we are only tabulating the final total, we can modify the 'reduce' mawk invocation to be as follows:

  mawk '
  {games += ($1+$2+$3); white += $1; black += $2; draw += $3}
  END { print games, white, black, draw }'

Making the bottle-neck data stream thinner always helps with overall throughput.

KEGG filters at the MAP or MODULE level, but it merely highlights the enzymes of the mosaic MAP/MODULE that it inferred is present in the organism, rather than display the subset of the network that is actually present.

I agree with you regarding their mutual incompleteness. There's plenty more literature to curate! See my systematic omparison of KEGG and MetaCyc for more details:

http://www.ncbi.nlm.nih.gov/pmc/articles/PMC3665663/

A problem with these wall charts and the KEGG diagrams is that not only do you not know which enzymes or pathways are active in which tissues within a single multi-cellular organism, you don't know which organisms they come from. These pathway charts are actually mosaic pathways, trying to show a summary of metabolism across all domains of life. This can be misleading, in that no one organism contains all parts as displayed.

Using Pathway Tools, the pathways are automatically laid out for you in a rational manner, organizing pathways and super-pathways based on a manually-curated ontology. For example, all biosynthetic pathways are displayed side by side. Wall charts dazzle with their complexity, but it isn't the best layout for organizing information.

The Roche Biochemical Pathways charts were the inspiration for the MetaCyc family of pathway/genome databases, all enabled by the Pathway Tools software (itself written with 400,000+ lines of Common Lisp):

Example pathway: Super-Pathway of glycolysis, pyruvate dehydrogenase, TCA, and glyoxylate bypass

http://www.metacyc.org/META/NEW-IMAGE?type=PATHWAY&object=GL...

MetaCyc has an order of magnitude more information integrated into it than most wall charts. Plus, you can get true interactive metabolic overview maps, where you can click on the enzymes and compounds (example from the E. coli database):

http://www.ecocyc.org/overviewsWeb/celOv.shtml (be sure to zoom in and click on things!)

The Pathway Tools software, and the metabolic networks, are freely available for academic use, and you can use it to construct your own metabolic networks.

Full disclosure: I'm a former full-time bioinformaticist on the project.

It's not exactly macrology, but here's a page on functional AWK: http://awk.info/?Funky

I love using gawk for slicing and dicing huge data files. The performance and conciseness of an AWK script is hard to beat. For those interested in general resources related to AWK programming, http://awk.info/ is a great place to start.

Notice how they cherry-pick among genomes to get their fitted line. A common fallacy is to associate genome size with the "complexity" of an organism. Even assuming that there is a simplistic linear ordering of organismal life (i.e., "lower" and "higher" organisms), a "lower" organism like the poplar tree has more than two times the number of genes as the human genome.

Whether vanilla TeX supports this out of the box or not, the TeX technology stack (including MetaFont and friends) has intimate knowledge of the size and shape of all glyphs. It is also not random as to which types of letters accentuate or obfuscate the border of a river in text. It is hard for me to believe that one cannot use dynamic programming to minimize the occurrence of rivers in the final layout, in the same way that TeX can minimize ugly borders using sophisticated hyphenation algorithms.

TeX uses a dynamic programming algorithm to perform its advanced hyphenation, which allows the text to fill the page "beautifully": http://en.wikipedia.org/wiki/TeX#Hyphenation_and_justificati...

If TeX is already using dynamic programming in order to improve the visual appearance of the words, I imagine that the same can be done for the space between the words without having to resort to image processing of the TeX document rendered as PDF.

You can think of Eshell as a Unix shell that is 100% customizable, and those customizations are done in a very high level language like Elisp, not in C. Sometimes it's great to use macros in a shell script.

Yes, it can be used for running scripts: https://github.com/emk/eshell/blob/master/em-script.el

In case anyone is interested, I've extended this code and put it up on GitHub as a script called 'samp':

https://github.com/taltman/scripts

Looks like a valid sample to me. Are you bothered by the ordering of the sample members? Then I'd continue the pipeline to include a call to shuf.

As minimax stated, your awk code won't provide exactly k samples. Here's a bit of awk code that implements reservoir sampling (apologies in advance for any bugs) and prints the current sample to stdout, without needing to first process the entire stream. It simply prints the set of samples to stdout whenever the sample updates, with sets of samples separated by a distinct string. It is called as follows (of course, replace dmesg with any stream of your choosing):

  dmesg | gawk -f reservoir-sample.awk k=5 record_separator='==='


  #!/usr/bin/gawk -f

  ### reservoir-sample.awk
  ###
  ### Sample k random lines from a stream, without knowing the size of the stream.
  ###
  ### (Tomer Altman)

  ### Parameters: (set from command-line)
  ##
  ## k: number of lines to sample
  ## record_separator: string used to separate iterations of the sample in stdout.

  ## Define a function for returning a number between 1 and n:
  function random_int (n) { return 1 + int(rand() * n) }

  ## Initialize the PRNG seed:
  BEGIN { srand() }

  ## For the first k lines, initialize the sample array:

  NR <= k { sample[NR] = $0 }

  ## If we've initialized the sample array, and we pick an integer between one and the current record number that is less than or equal to k, update the sample array and print it to stdout:
  NR > k && (current_random_int = random_int(NR)) <= k {

    sample[current_random_int] = $0

    for (i in sample) print sample[i]
    
    print (record_separator != "") ? record_separator : "--"
  }

I don't understand why the data is unstructured, and thus SQL is not an option. Isn't the data coming in from calls to the company's own API?