News for hackers. Stuff that matters.
Re:Whoosh – Fast, full-text indexing and searching library i... by mattdeboard on Tuesday November 19, 2013 @07:55AM (#6760800)
Solr is just so easy to get up and running, so powerful with so many options for scaling, I don't see the benefit of "pure Python" here. (To be fair, despite living and breathing Python most days, I've never thought "pure Python" is a selling point.)
Re:Whoosh – Fast, full-text indexing and searching library i... by randlet on Tuesday November 19, 2013 @08:29AM (#6760992)
Pure Python is definitely a selling point for those of us who regularly deploy to multiple platforms. Knowing you can 'pip install pure-python-lib' on any platform and have it work every single time without regards to what c compilers and c libraries you have available is a major boon.
Re:Whoosh – Fast, full-text indexing and searching library i... by davb on Tuesday November 19, 2013 @10:43AM (#6761873)
Or, more importantly (in my case), very easily bundle pure Python code with your application. I understand the arguments against it, but in any project that might produce deliverables that will be in play for more than a few weeks the first thing I do is create a "lib" folder where any third party dependencies live and get imported from.

With Python this is usually very easy.

I've maintained my fair share of legacy code and more often than not one of the biggest issues is tracking down ancient libraries which work with my code. There are very few packages I trust to maintain fairly stable interfaces and be available years into the future (things like Apache HTTPd, PostgreSQL and memcached); everything else lives in the project lib folder. I can "hg clone" my app at any time and get the exact version of the third party packages I need.

Every so often I update the packages and test the integration.

Re:Whoosh – Fast, full-text indexing and searching library i... by lotyrin on Tuesday November 19, 2013 @06:02PM (#6765126)
This is why good dependency management systems support both specifying compatible semver ranges but also pinning/locking specific releases. You get nice one-step install/update/add/remove for your deps with the tool, but a fresh checkout can get the last known working versions and deploys can be guaranteed to use the same versions as were run in CI, etc.

Looking now, it seems that pip can finally do this with pip-compile.

Re:Whoosh – Fast, full-text indexing and searching library i... by zzzeek on Tuesday November 19, 2013 @02:46PM (#6763867)
Solr and its backend Lucene are written in Java. so it runs on any platform arguably even more easily than Python.
Re:Whoosh – Fast, full-text indexing and searching library i... by danudey on Tuesday November 19, 2013 @07:58PM (#6765703)
If you're assuming a known-good Python stack, then a pure-python solution will just work. Using Solr/Lucene means having a known-good Python stack and a known-good JVM. Python is installed with most (all?) Linux distributions by default, but Java is almost never part of the base install.

Also, a pure-python solution allows for an in-process index/search, rather than adding another external process dependency which needs to be monitored and maintained.

Re:Whoosh – Fast, full-text indexing and searching library i... by ris on Tuesday November 19, 2013 @04:25PM (#6764605)
> so it runs on any platform arguably even more easily than Python

I think that would be a very difficult argument to make.

Re:Whoosh – Fast, full-text indexing and searching library i... by randlet on Tuesday November 19, 2013 @02:55PM (#6763958)
Understood but I was addressing whether 'pure Python' is a selling feature which it undeniably is.

edit: I don't mean to say that it is a selling feature for everyone (i.e. people who aren't already using Python) but for those who are already using Python it definitely is.

Re:Whoosh – Fast, full-text indexing and searching library i... by nknighthb on Tuesday November 19, 2013 @08:04AM (#6760854)
Having had nothing but extremely negative experiences with production JVM deployments, anything being not-JVM-based is a huge selling point for me. Furthermore, "pure python" itself means that I can run it in any Python implementation I want, including pypy, without worrying about any C modules.
Re:Whoosh – Fast, full-text indexing and searching library i... by swasheck on Tuesday November 19, 2013 @12:18PM (#6762635)
All of the troubles that I've had have been relative to compiling the JVM Python code and linking it correctly to my Java binary at compile time. Once I've gotten that working, I've found Lucene to be quite simple - and faster than Whoosh!
Re:Whoosh – Fast, full-text indexing and searching library i... by koolkao on Tuesday November 19, 2013 @10:28AM (#6761755)
what kinds of JVM issues did you run into in deployment? for personal projects I have always used C++, Python, RoR, MEAN, etc. but at work where my role is non-dev, our software dev team likes Java and the heavy weighted GWT... frankly shutting me out. I'd love to know how to convince them to switch out of Java.
Re:Whoosh – Fast, full-text indexing and searching library i... by nknighthb on Tuesday November 19, 2013 @10:45AM (#6761887)
Memory issues out the ass. Which are obviously all in my head because there are no memory leaks in Java. I must be doing it wrong. Am I sure I know how to multiply? Of course you shouldn't be using version 1.2.3.4.5, our code only works with 1.2.3.4.4, everybody knows that. Well, except for that code, which of course you have to run on 1.2.3.4.4c. And this third-party daemon over here, that only runs on an obscure 6-year-old JVM and you have to put it in /var/deadchicken/ and insert three squirrels before executing.

You won't convince them, they've already jumped off the cliff. It's all perfectly normal to them. If you want to do development, find a company that hasn't been contaminated yet.

Re:Whoosh – Fast, full-text indexing and searching library i... by diek on Tuesday November 19, 2013 @12:52PM (#6762943)
> Memory issues out the ass.

As others have posted, this is indicative of bad code. Java can certainly leak memory in the form of retained live references.

> Of course you shouldn't be using version 1.2.3.4.5, our code only works with 1.2.3.4.4, everybody knows that.

That's completely independent of the language or runtime being used, and is purely a project management issue (which breaking changes go into which version).

> that only runs on an obscure 6-year-old JVM

If there is code that imports classes specific to a JVM (com.sun, etc), then that code is doing something pretty much universally agreed on in the Java community to be a 'bad thing'. Otherwise, bytecode from Java 1.0 still runs on the latest JVM without issue.

You can use any language or runtime badly. There is a lot of code out there written in Java, and a lot of 'commodity' Java developers writing it. Of course there is going to be a greater volume of bad code, that's not an indictment of the platform itself.

Re:Whoosh – Fast, full-text indexing and searching library i... by nknighthb on Tuesday November 19, 2013 @01:13PM (#6763112)
> That's completely independent of the language or runtime being used, and is purely a project management issue (which breaking changes go into which version).

That passage was about the runtime. At one point, I had to deal with four different JVMs in the same server farm.

> You can use any language or runtime badly.

No other ecosystem has so consistently fucked me over. Not even PHP. I actually prefer Java to PHP for just writing code, but PHP to absolutely anything running on the JVM for operations.

Know when I stopped regularly getting woken up in the middle of the night? 2011, when I stopped supporting anything on a JVM. Ironically, it would have been two years earlier, but I foolishly self-inflicted Cassandra on myself. Never again.

Re:Whoosh – Fast, full-text indexing and searching library i... by robrenaud on Tuesday November 19, 2013 @11:54AM (#6762451)
I don't actually write Java professionally.

But it sounds like your problem isn't Java/JVM, but rather really shitty code. If it breaks when you change minor versions of the JVM, the code is broken.

Re:Whoosh – Fast, full-text indexing and searching library i... by nknighthb on Tuesday November 19, 2013 @12:12PM (#6762571)
Just from my own personal experience, you've just indicted code from three well-known SV companies, at least two more non-SV fortune-500s, a mess of B2B vendors, the Apache foundation, and a few minor, independent open-source projects. I've heard stories from friends in a few other companies about both internal and third-party code.

At some point you reach the conclusion that the problem with an ecosystem is more fundamental than a few lazy developers. But even if not, your statement isn't actionable. If all the code in a certain set happens to be really shitty, the only rational choice is to not use that code. Any other argument ends up being one only of semantics.

Re:Whoosh – Fast, full-text indexing and searching library i... by ris on Tuesday November 19, 2013 @04:35PM (#6764655)
Culture. It is all about culture.

You'll have all seen the pattern. The sort of people who use language X tend to be the same people who use VCS Y tend to be the same people who think doing Z is a good idea tend to be the same people who don't see any problem with IDE W and protocol V.

Culture (by definition) is something that propagates amongst a community, and sometimes it's just poisonous. Or at least seems poisonous from an outsider's point of view. Except if it's java, in which case there's no two ways about it.

Re:Whoosh – Fast, full-text indexing and searching library i... by jsmeaton on Tuesday November 19, 2013 @04:22PM (#6764575)
I think I've deployed only one java web app (puppet-db) that doesn't have huge memory leak issues. With the lack of process recycling in tomcat to mitigate "bad code", I cringe every time I have to deploy another one.
Re:Whoosh – Fast, full-text indexing and searching library i... by jimbokun on Tuesday November 19, 2013 @11:28AM (#6762237)
I am very happy with Maven for tracking dependencies and building and deploying artifacts.

Please educate me, what am I missing out on?

Re:Whoosh – Fast, full-text indexing and searching library i... by nknighthb on Tuesday November 19, 2013 @12:00PM (#6762497)
> I am very happy with Maven

I have two responses to that, which may seem contradictory but are both heartfelt:

1) Your company uses Maven? You're lucky!

2) Maven? The prosecution rests.

> Please educate me, what am I missing out on?

You're happy. I'm obviously just another annoying, delusional ops guy who doesn't know what he's talking about. Why would you think you're missing out on anything? Go about your business. Everything is just fine.

Re:Whoosh – Fast, full-text indexing and searching library i... by yid on Tuesday November 19, 2013 @01:01PM (#6763017)
Don't be so bitter. Your post above was informative to me, and it's telling that you alluded to the community blaming ops troubles on "bad code", and the two people who replied to you did just that.
Re:Whoosh – Fast, full-text indexing and searching library i... by ra on Tuesday November 19, 2013 @08:03AM (#6760847)
Solr, Sphinx and ElasticSearch are all reliable search-at-scale options for those of us without petabyte problems.

Whoosh is fine for very small datasets (megabytes) and low loads, just watch out for file permission issues.

Re:Whoosh – Fast, full-text indexing and searching library i... by Torn on Tuesday November 19, 2013 @09:03AM (#6761178)
ElasticSearch is actually designed to scale to petabytes and beyond:

http://elasticsearch.com/product/ ("petabytes of data with ease")

Re:Whoosh – Fast, full-text indexing and searching library i... by SEJeff on Tuesday November 19, 2013 @09:42AM (#6761404)
Solr is pretty good stuff, but I've had a bit more luck clustering and scaling elasticsearch horizontally with many hundreds of terabytes of data. From what I've read however, the newer solr versions took a lot of ideas from Elasticsearch and they are somewhat comparable at the moment.
Re:Whoosh – Fast, full-text indexing and searching library i... by jMyles on Tuesday November 19, 2013 @03:39PM (#6764281)
Not the most popular, but test suite veracity is my favorite "pure Python" selling point.
Re:Whoosh – Fast, full-text indexing and searching library i... by mattdeboard on Tuesday November 19, 2013 @08:45PM (#6765855)
You'll have to explain this. Most tests we write in Python are redundant with static typing.