How expensive is SF rent? Use Unix commands to find out
How expensive is SF rent? Here's a coding challenge that uses Unix command line tools to find out.
Hacker friends suggested I post the result and code here because I'm writing open source statistics software and this is a practical example of results and code.
The results for a one bedroom in SF: mean $3348, median $3475, interquartile range $1514, and standard deviation $1130.
How to use Unix commands to find out: use `curl` to fetch Craigslist apartment listings, then `sed`, `grep`, and `tr` to filter and parse the data, then `num` to calculate the statistics.
curl -s http://sfbay.craigslist.org/search/sfc/apa\?min_price\=1\&bedrooms\=1 |
sed 's/\(<p class="row"\)/\n\1/g' |
grep ' 1br ' |
tr ">" "\n" |
sed -n 's/^\$\([0-9]\+\).*/\1/p' |
num mean median interquartile-range standard-deviation OFMT="%.0d"
(
10 comments
| 4 points
| on HN
)
For this script, the fetch returns at most 100 rows, so won't overflow.
More broadly, the Num command runs using awk, and number handling depends on your system's awk implementation and C libraries. Current awk implementations use an internal representation of all numbers, including integers, that uses double-precision floating-point numbers. I know GNU awk supports arbitrary precision arithmetic, and also can use --bignum, GNU MPFR and GNU MP (GMP); these will be great additions to the Num command for the next major version upgrade.