spulec

Yes, we've been improving our scraping technology for the past 8 years as we've worked on YipitData (the #1 provider of web data to wall street).

Python is the only supported language right now. Scrapy is an awesome project, but we have a very different approach. We strive to be Flask instead of Django.

If you want to, you can use ReadyPipe entirely in the browser through jupyter notebooks instead of needing to setup a local environment. This is especially helpful with more complicated systems using Selenium and Puppeteer. We discuss a lot more of the features that differentiate us on our homepage: https://readypipe.com/ and the docs https://docs.readypipe.io/

Feel free to reach out (email in profile) with additional questions.

YipitData, NYC, Fulltime. VISAs welcome.

Data Systems Engineer: http://yipitdata.com/jobs/#job-engineer

YipitData develops clever ways to learn about businesses from online data. We specialize in acquiring this difficult to obtain but extremely valuable information through modern, large-scale technology.

We are profitable, VC backed, and our clients include the largest funds on Wall Street. Our goal is to predict quarterly financial performance and identify long-term inflection points for every public internet company. There are over 100 such companies today that create meaningful data, and 15 new ones go public every year.

We are building the premier destination for understanding data about important companies. We think this is the future (http://avc.com/2012/08/it-is-hard-to-hide-from-the-web/) of investment research.

YipitData:

- Python + Redis + MySQL

- Continous integration with Travis CI and Buildbot

- Continuous deployment

- Cloud hosting in AWS with Cloudformation and Chef

- "Automate the boring stuff"

You:

- Conscientious about assumptions baked into a system

- Critical eye for small things that "seem off"

- Disdain for inefficient processes (you often find yourself complaining about elevator algorithms)

- Curiosity to dig deep into issues that you don't fully understand

Your first day will consist of:

- Fresh bagels (is there any other way to start a new job in NYC?)

- Unwrapping your new MacBook

- Your first commit (adding yourself to our about page)

- Your first production rollout (this is automatic since we practice continuous deployment)

- Introduction meetings with team members from across the company

- Learn about our different products by going through the codebase with another engineer

- Welcome drinks!

Your first week will involve training for the following:

- Overview of our infrastructure

- Introduction to Finance

- Web Extraction

- Introduction to Sell Side Research

- Introduction to Outsourcing

- Overview of our internal libraries that allow us to quickly develop new products

Within three months of joining, you will:

- Investigate and evaluate the potential for new data products

- Help write the technical spec for a new data product

- Take primary ownership for the development and maintenance of that product

- Configure the servers in the production environment for this product

- Work with data analysts to refine the resulting data from this product

- Start to contribute back to the shared libraries we use across products

- Pair program with data analysts on smaller projects

- Help in recruiting more data engineers

We offer highly competitive salary, equity, and excellent benefits. YipitData promotes strong engineering and company cultures; candidates should be excited about being a part of a fast growing start-up.

Send me an email if you are interested: [email protected]

Agreed. We've had success using a very base ubuntu ami and then having the user data install chef and do a chef run.

This results in slower startups for new machines, but the flexibility seems to be worth it. Eventually we'll get to a point where the system will just build a new ami anytime a cookbook change is committed.

I'm curious about 2012-08 and 2012-11. Pet feature, hackathon, large release with hotfixes?

Correct. I've gone with his latter solution for now and added a warning about import order. I'm not very happy with this solution through and will be spending some time with ctypes in the next few days to come up with something better.

Hey Timothy, thanks for the response. You are correct. I'll add a warning to the library and work on a solution. Unfortunately, I think it will need to involve ctypes.

Absolutely. Stashing seemed like the cleaner choice to me, but either would work.

Cool. We had looked at doing it that way, but decided that we wanted to have the flexibility of accessing the entire repo so that we can run things like "./manage.py validate" and quick unit tests.

A leftover pdb.set_trace() that snuck into production is the exact reason we first added this :)

If you're still interested in NYC startups, check out yipit.com/jobs and email me([email protected]). We're hiring and have sponsored visas before.

One of my favorite quora answers:

http://www.quora.com/Dropbox/Why-is-Dropbox-more-popular-tha...

New York, NY - Yipit

Just off raising $6 million, we are looking for the 12th member of the team(when we posted this last month, we were looking for our 8th).

Come join us on the ground floor of one of the best startups in New York. Right now, great companies like 10Gen, FourSquare, Hunch, SeatGeek, and YCharts are all here growing together. Silicon Alley is going through a renaissance and you can be part of it.

-UI Lead Architect: Our interface sits on top of over 350 daily deal services and is used by hundreds of thousands of people. We need you to own that interface.

-HTML5/CSS3/jQuery Developer: All user-facing activity relies on these technologies. We will commit the full resources of the team to supporting you.

-Python(Django) Developers: We work with the latest technology including: Amazon Web Services, RabbitMQ, Gunicorn, Nginx, and Git. This should excite you. Go to http://yipit.com/about/jobs/ to apply. Email [email protected] with any questions.

New York, NY - Yipit

Just off raising $6 million, we are looking for our 8th member of the team. Come join us on the ground floor of one of the best startups in New York. Right now, great companies like 10Gen, FourSquare, Hunch, SeatGeek, and YCharts are all here growing together. Silicon Alley is going through a renaissance and you can be part of it.

-UI Lead Architect: Our interface sits on top of over 350 daily deal services and is used by hundreds of thousands of people. We need you to own that interface.

-HTML5/CSS3/jQuery Developer: All user-facing activity relies on these technologies. We will commit the full resources of the team to supporting you.

-Python(Django) Developers: We work with the latest technology including: Amazon Web Services, RabbitMQ, Gunicorn, Nginx, and Git. This should excite you.

Go to http://yipit.com/about/jobs/ to apply. Email [email protected] with any questions.

Yipit is actually in Toronto, Montreal, and Vancouver.

Originally it was all done manually, but we've been automating more and more.

Feel free to send me any feedback. [email protected]

Yipit.com aggregates Groupon, LivingSocial, and hundreds of other daily deal sites(including non-location specific ones like Woot). It will filter the deals based on your location and specific interests you have.

Disclosure: I recently started working for Yipit.

As someone who just finished an internship, I can provide some insight into the good and bad.

As far as a main project, I would choose something new and independent. Think of that crazy idea you've been playing around with for a while, but haven't had time to implement.

Another thing that often gets overlooked is a side project. My internship work level tended to be very cyclic. Whenever my main project was stalled for one reason or another, I had little to do(I ended up roaming through the code base). Find an incremental project for this. Bugs and small features may suffice.

Finally, recruiting. Posting on sites like this is a good start. The type of students you want will find it. Also contact the career centers or CS departments of the specific schools you like. Most have some sort of system in place. Recruiting begins in September in a lot of places.

Send me an email if you want to pick my brain.

I can confirm that the security hole was not limited to only friends. I was able to insert non-friend's IDs for the "viewas" parameter.

Yes, the title was designed to be somewhat tongue-in-cheek.

Thank you, corrected.

I agree that Alice has not provided a substantial argument, but this does not make Bob's response any more legitimate.

This is certainly not meant to be comprehensive. More of a 'top hits' for those less willing to look over the entire Wikipedia article.

The on-demand approach is not completely eliminated. Users can still watch episodes of previously broadcast shows. I'm hoping that the show owners will be out there encouraging voters. I also thought that some statistics displayed to the user could encourage voting(consecutive weeks voting, total shows voted for, etc.)

The long-term plan would be to have multiple channels, with at least one that would presumably fit your interests.