It was hard. There are so many scams out there. If you are not hand curating listings, you have to rely on somewhat novel approaches to filter out all the bad data. For example, any mention of 'Jesus' or 'God' automatically blocked the listing. Sure there might have been legitamate listings, but anytime I would go in and check 99% of the time it was a scam. You also have scams of people listing units they don't own or have any relation too. They ask for a security deposit up front and just pocket it, leaving the new tenant to have an awkward conversation with the real resident.
Data is often unformatted too. Scraping out bedrooms and bathrooms can even be a challenge on websites like craigslist where the listing is just one big paragraph. (Not anymore, craigslist has come a long way, but that's how it was 10 years ago). Often times you had to just search for the closest number around the word "bed" or "bath". Don't even think about getting features like "driveway" or "laundry" out of them.
In the end, we ended up utilizing some pretty intense ETL pipelines to collate historical data, census information, property assessment data, and other things to try to get a more accurate picture on our listings.
But that didn't win out. What won out are the sites like Apartments.com or Zillow, where legitimate property owners can post their listings in a formatted searchable way. We could scrape them and post the same listings on our site, but at that point we were just pushing our customers to another platform that honestly worked better than our own.
We couldn't have the most up to date data, that was determined by how fast we could go back to scrape a listing. And often times we were knee deep in a battle to avoid being blocked by these companies. Often times, after we had exhausted our proxies, the only thing left to use was Tor.
[...]
> We couldn't have the most up to date data, that was determined by how fast we could go back to scrape a listing. And often times we were knee deep in a battle to avoid being blocked by these companies. Often times, after we had exhausted our proxies, the only thing left to use was Tor.
The flip side is if you are looking at listing from small property management companies, they are probably low on resources. The websites aren't very well optimized to serve thousands of concurrent users. Every request to search inventory hits SQL Server which is probably one small box with no automatic failover. I don't like it but unless we somehow help these people better optimize their website (how?),they will continue using heavy handed tactics like blocking scrapers.
Source: got to the "send me a viewing fee in itunes gift cards" stage of one of these scams, once. Googled the initial email and it's a common scan template.
From the agent's perspective, it's good to drive people to their company's website rather than one of the big aggregate ones. Having more people visit it means they're more likely to do business with that company specifically. Getting people to visit your company's site is pretty straightforward: they put up ads there earlier, often so much earlier that by the time you see an ad on an aggregate site, it's already fully booked for viewings.
And that's how we end up with dozens of different company sites, each crappy in their own way, and in the middle of it all, people trying to find a home pulling their hair out, refreshing a bunch of different websites every few hours.
I find it much preferable to North American sites like https://realtor.ca that think that number of baths is more important than floorspace (which to my utter amazement is often not even available).
There are sometimes rare jewels published directly by individuals on immoweb.be but the bulk of the offers is trash.
Selling though is a completely different story it's almost impossible without an agent. And they do provide a valuable service; like photo session, publishing on funda and others, taking care of house-viewing etc.,
Even houses can be difficult in many cases.
This is something I noticed last time I was apartment hunting in Montréal. You'd see a sign or listing, call, and the number would be out of service. As if landlords were using virtual numbers/call forwarding services and just shutting it off once the unit was rented.
I don't call phone numbers on listing sites anymore, even if there's a face and an agency next to it. Better off looking up the agency, going to their website an calling that number (which in all three cases of the above, were different than the one on Zillow).
---
* by "interesting set of experiences" I mean calling the number on a listing took me to an automated menu and a robotic voice where I had to enter the street number, and then first five letters of a street name so it could "look up" the property I was inquiring about, and connect me to the right seller.
Hanging up on that nonsense immediately and calling the leasing agency's number via their website got me a human being in two rings, and the leasing agent for the unit I was interested in after about 45 seconds of hold music.
This is strange, wasn't my experience at all. Just about every house I viewed through funda was advertised by agents. And when I put up my house for sale the first thing the agent did was to publish on funda. All the agents care about is selling a house they get a hefty sum (0.3% if I remember correctly) and are totally fine advertising on funda.
As a house seeker I dealt with a bunch of agents and the experience wasn't all that bad to be honest. Mind you all this was in early 2019; from what I hear late 2020 onwards the market has gone bananas in the Netherlands and Amsterdam in particular.
- paid searches (even at other agents)
- unpaid searches at their own agency
- social media
- agency website
- funda
each step would be on the next day, so if you're only watching funda, you're already four days behind
Think it’s all related apartments and houses not really being fungible.
ESIT: Also, more directly to GP’s experience, when I was in the market in NL I found the biggest aggregator to be very useful, which is apparently different from their experience: funda.nl
https://www.statista.com/statistics/257344/top-lobbying-spen...
This is how capitalism works when there is no government helping monopolies to form in the name of security / justice / anti-money laundering / insert-politician-buzzword-of-the-day.
The natural end condition of an unregulated market is a monopoly. That is why we need regulation.
So, something is operating to provide implicit regulation. In this case it seems like chaos serves that role. Monopoly relies on a self-ordering property common to most market conditions.
The market for rentals is never efficient and will never be! People who post to Zillow with a perfect description know they can extract more money for it.
Today, there would be a lot more information available. However, as you suggest, I assume all that available information probably increases the efficiency of connecting a property with potential buyers/renters so you're less likely to luck into something that others just haven't found yet. (The funny thing is that the house I ended up buying turned out be being sold by someone I knew quite well at work.)
Various things cost me a fair bit of money, effort, and angst that I'm glad I didn't know about at the time. But, at the end of the day, I got a ridiculously good deal--especially by today's standards--for a semi-rural place only about an hour out of Boston.
Everyone that attempts to address the housing market just winds up realizing how lucrative it is to not fix it
Once they get a little data and hire some agents its just a reversion to the mean
Would a pad mapper intentionally make some listings bad in order to reduce effective supply?
call them with the bdrm count you're looking for, and if they know exactly what building you're talking about and units they have available, sometimes they'll give you the code to walk right in and check the place out.
you can use this convo to push the rate down a bit or ask for a rent concession, and since you're likely talking to the landlord/agency it'll be no-fee.
that's how i've found my last 3 rentals :) gl!
I was going to say the same:
> walk around the neighborhood you're thinking of renting in...call them...
It works far better in Belltown (Seattle) than the free "apartment shopper weekly" printed ones (which have glossy pictures and outrageous prices).
Your entire post is spot-on. Thank You.
The places you get are amazing. Clear views of the SF harbor, tucked away in PacHeights, full parking, cheaper rent etc. After doing this for 15 years I knew all the managers of those 10 buildings by name. My friends that "wing it" get sooo screwed. They go through listings, trying to piece if it is a nice place, always getting their initial deposit stolen, application fees swiped, going to open apartments 20 people deep, dealing with large corporate landlords, on and on. Make friends!
I hesitate to say it, but there is already a solution to this, and it has been the one that New York has had for years: you hire an agent. A good agent will sort through all of that for you, and streamline the process to the point where you can go from search to signing in under a week (if not quicker). Of course, there's the problem of bad agents, but that's true of anything. As you note, online services are often bad, as well.
In other words, I don't see this as a problem without a solution; I see it as a problem without a software automated solution, which is true for a lot of problems in life, particularly ones that involve huge numbers of messy, opinionated people with different motivations and interests.
It seems it's in their job description to try to convince me to pay more for less. They listen carefully to my requirements and then take me to places that don't match at all.
I guess they figure since I've taken my entire afternoon with them, and since they've been friendly, I might give up on what I'm looking for and rent whatever it is they decided to shill that day.
Of all the countries I've rented in and had occasion to interact with agents, ones in Turkey behaved the most like cartoon caricatures: they use every sitcom used car salesman trick in the book. It would be funny to watch, except that they're perpetually wasting their time and mine.
She was pretty much useless. Most of the places she sent us didn't match our criteria very well, and quite a few were above where we wanted to spend. As each day went by and we nixxed place after place she clearly de-prioritized us in her work (I'm assuming she was working with multiple other people who were looking for places) and became less responsive.
At the end of the day I had to do my own hunt and fly up a second time to look at places over a weekend to find something.
And that's a paid agent from a large moving corp. I do wonder if a (more expensive) boutique agent would have been more effective but honestly having worked in real estate I consider them all at the same level as used car salesmen anyway.
As an aside I also tried to automate my apartment hunt. The main thing that matters in NYC is time to respond. Unfortunately Zillow, StreetEasy, etc are not very easy to automate on the messaging side due to bot countermeasures. It was an incredibly time consuming, manual process. Happily found a great place though.
In NYC specifically, there are agencies that make most of their business from renter-side representation. You can find the good ones easily on the usual review sites.
In Melbourne Australia, what I have seen is that rental agents are different and the real estate agents are different although they work for the same company.
As someone else in this thread noted, the landlord often hires an agent because they don't want to deal with the tidal wave of crap that comes from dealing with applicants. That person gets paid, obviously, and the landlord isn't the one paying them. That's the rate you're talking about here. These people are not incentivized to provide good service to the renter, and typically don't. They suck.
Higher-end buildings often have in-house agents (again: you're paying for this, whether you realize it or not), and these are called "no fee" buildings. But an increasing number of places don't have either, and just provide access to professional renter's agents instead of listing publicly. Or they do list publicly, and barely respond to the tidal wave of yahoos, knowing that motivated renters hire an agent. YMMV.
You know how we know this is a solvable problem? Listings for places to buy don’t have nearly as many problems.
> You know how we know this is a solvable problem? Listings for places to buy don’t have nearly as many problems.
Just as OP said, there are tons of sites out there that claim to "solve" this, but don't. So no, I don't know that this is a computer-solvable problem.
There are agencies that make most of their business in this area, and generally have better rate structures (typically a month or two of rent, flat rate). Others want to charge you on a percentage basis, which you can negotiate, and obviously changes the incentive structure. All of these are "expensive" (relative to free), but worth it depending on how you value your time.
Other people are (rightly) noting that there is risk of getting a scammy real estate agent, but that's true of any service. You have to look at reviews and trust your gut. Good real estate agents are motivated to do the right thing for you, because they see it as a marketing channel into larger sales in the future. Bad real estate agents are in it only for the immediate cash. These people behave in such obviously different ways that they're pretty easy to discriminate, in practice.
The only way they might be working for you is if you are paying them an hourly rate or per showing or something like that.
If you're saying that someone doesn't "work for you" unless you pay them a flat or hourly rate, then I disagree completely. All incentive structures have problems. Someone who works on contingency is incentivized to get you to sign. If they take a percentage cut, of course, they want you to sign the most expensive thing they can. Someone who works on a flat-rate basis is incentivized to do as little custom work as possible. Someone who works on an hourly basis is incentivized to drag out the proceedings. There is no perfect system.
Most of these agreements are structured where the fee is paid by you, after signing a lease, as a function of the monthly/annual rental price. The agent works for you, you just don't pay up front or a fixed cost or on an hourly basis. Does this change the incentive structure? Yes. Does it mean that they don't provide you a benefit? No. You just have to be aware of the incentives.
Just like when buying a house, an apartment broker works on behalf of the landlord or the tenant. They charge a fee which is a percentage of a year’s rent. It is negotiable.
Dwellsy.com has the largest inventory of any rental property site out there, is aiming for zero fraud (and has quite a few features implemented to prevent against it,) parses descriptions to more accurately represent the actual amenities (it's decent, but not perfect) and doesn't clump units in apartment buildings into one ranged price listing, as we prefer to have the units represented as individual listings.
Hopefully it'll help, but feel free to reach out directly if you run into issues.
In the ancient "before times" there were also actual rental agencies where a human would look for an appropriate apartment for you. I didn't know about them, but a local NorCal person turned me on and it was the only way to find things. They followed all of the different local listings daily and knew the area fairly well (e.g. how parking, transport, different commutes would be). It was $25/mo and ~$100/yr for them to go through the listings call the renter, if necessary. They usually also collected $100 fee, if they hooked you up (though I'm sure you could work around it) with one of their (usually several times a week email of 5-10) available recommendations and usually a $50+ fee for running a "credit check". The reason it seemed to work was that the renters trusted the rental agency to only send them real referrals, and the agency knew all about scammers and fake listings, and daily follow-up. It only worked due to the relatively small area and and high volume. Each agent was looking for ~50-100 people during a month and making several $k/mo themselves, filling them typically in a few months.
Like travel agents used to be an actual thing... but now you either do it yourself on-line, or you're going on some giant family cruise thing.
I think there was a fee in the $50 to $75 range, but it included the credit check and access to their listings.
If multiple candidates compete over the same apt, no searching method in the world will resolve the issue. Either you have high supply of apts, or a super intelligent city software that matches people to available apartments based on their socioeconomic and professional profiles.
Tried the same thing with NYC and things went nowhere fast.
There were a few outright lying apartment management companies, but I managed to look at some nice places being rented out directly by their owners and settled on a good one
"Hooray, sounds great!"
"Ok, which element of that list would you like?"
Conclusion: it hasn't changed much at all.
It's still necessary to search on a myriad of sites. Most of the time, by the time you inquire, a place is gone. Or the ad was misleading and that "$2000 2BR" is actually "$2000 for the cheapest Studio we have, but we also have 2BR for $3.5K".
Finding places that match my criteria still requires careful inspection of the description. Checking a filter box for "Parking" excludes places that only mention parking in the description.
With all of the modernization and streamlining that has happened in other parts of our lives, apartment hunting remains unstructured, messy, frustrating, stressful.
Aggregators like Padmapper helped for awhile, but these days, it seems like most listings are posted by scammy middlemen just trying to make a commission on the listing, and now the site actively makes the process even harder.
Is hunting for an apartment doomed to be this messy forever? It seems like an area that is ripe for disruption or a better option, but what would it take to actually solve this?
---
With all of that said, this post is exactly what I needed right now. I've been toying with a very similar idea to improve the efficiency of my search process and this is good inspiration.