News for hackers. Stuff that matters.
Re:EU Open Web Search project kicked off by marginalia_nu on Tuesday September 20, 2022 @04:48PM (#32917596)
I'm a bit skeptical EU-funding a bunch of professors is the way a search engine will be built.

The primary goal for academics is to publish new findings, while what you need to build a search engine is rock solid CS and information retrieval basics. Academically, it's not very exciting. Most of it was hashed out in the 1980s or earlier.

Re:EU Open Web Search project kicked off by jjulius on Tuesday September 20, 2022 @05:25PM (#32918004)
>I'm a bit skeptical EU-funding a bunch of professors is the way a search engine will be built.

Heh, so, funny story...

>A second grant—the DARPA-NSF grant most closely associated with Google’s origin—was part of a coordinated effort to build a massive digital library using the internet as its backbone. Both grants funded research by two graduate students who were making rapid advances in web-page ranking, as well as tracking (and making sense of) user queries: future Google cofounders Sergey Brin and Larry Page.

>The research by Brin and Page under these grants became the heart of Google: people using search functions to find precisely what they wanted inside a very large data set.

https://qz.com/1145669/googles-true-origin-partly-lies-in-ci...

Re:EU Open Web Search project kicked off by dizhn on Tuesday September 20, 2022 @07:51PM (#32919312)
Splitting when the project looks like it's gonna make money is the American way. (Thanks for the public funds. khaaa chingg)
Re:EU Open Web Search project kicked off by google234123 on Tuesday September 20, 2022 @08:04PM (#32919407)
They did a lot more good by making a company rather than sticking around in academia and publishing a few extra papers.
Re:EU Open Web Search project kicked off by pantalaimon on Wednesday September 21, 2022 @04:26AM (#32922178)
I see this a lot in a project I'm involved with where the majority of contributors are from academia. The incentive is to mostly push novel things, often with questionable practicability, in order to write a paper about it.

But nobody will implement the 'boring' features needed to make the thing generally useful.

Re:EU Open Web Search project kicked off by Xeoncross on Tuesday September 20, 2022 @09:18PM (#32919872)
username checks out
Re:EU Open Web Search project kicked off by permo-w on Wednesday September 21, 2022 @09:41AM (#32925022)
not that I support this approach, but the return on public investment there is clearly huge
Re:EU Open Web Search project kicked off by ddorian43 on Wednesday September 21, 2022 @03:21AM (#32921764)
The best remote+paying job that I've ever seen online was from a Darpa project (memex project, about search engines). $180K - $250K+. This was ~5 years ago.

Curious what the salaries will be on this one.

Re:EU Open Web Search project kicked off by mellavora on Wednesday September 21, 2022 @05:01AM (#32922420)
Marginalia, I know you are working on a (fantastic!) search engine of your own https://search.marginalia.nu/

I salute your efforts and endorse your search engine. I also recognize that you know what it takes to build a search engine.

I don't think you have deep familiarity with EU academia. - The primary goal for academia is to influence society. Publishing is a route to that. - Being head of the EU search engine project would give high academic status - There are hundreds of articles which you could publish on this project - Rock solid CS. Would someone like Knuth count as "rock-solid"? Who is better at CS, the person who can implement quicksort because they practiced leetcode, or the person who invented quicksort? - information retrieval basics. Again, these basics were probably developed in academia.

The skills you say are basic to this endevour are more prevalent in top-quality professors and post-docs than they are in industry.

Re:EU Open Web Search project kicked off by marginalia_nu on Wednesday September 21, 2022 @07:23AM (#32923498)
On top of these skills, you actually need most of all software engineering and architecture experience. I don't think this is common in academia at all. Not in professors, not in PhD-students. You need practical experience building complex software at a large scale. Across that, you need to implement these CS fundamentals.

This is requires far more CS than you'll find in your usual software development effort, for sure, and many CS professors absolutely fit that bill. However, to the same degree it also demands far more on the software engineering side. People out of academia in general, from every time I've seen them build software, have not been all too impressive on that side of things.

Web search has an incredible demand for being well rounded, beyond anything else I've encountered. CS isn't the hard part bottle-necking everything else, it's just one of the many hard parts.

Re:EU Open Web Search project kicked off by margarina72 on Tuesday September 20, 2022 @09:25PM (#32919917)
One could have been skeptical about US-funding a bunch of university students to build a search engine a few years ago.
Re:EU Open Web Search project kicked off by harha on Tuesday September 20, 2022 @09:29PM (#32919932)
Search was quite broken back then. It got reasonably good at some point, now I’d argue it’s the content as a whole that’s gotten worse.

I’m just skeptical if the EU bureaucrats will put the money in the right place, and if this is even the right approach.

Re:EU Open Web Search project kicked off by margarina72 on Tuesday September 20, 2022 @10:19PM (#32920179)
The project can fail, you are correct, but it does not take anything from any other projects, it is just an other initiative trying to contribute in the space.

Parent commenter own search project Marginalia Search [1] could even benefit from it, or even maybe collaborate with it.

It is not a winner-take-all situation, and we need various open initiative in this space to get out of the current conundrum we are in with Google stronghold on search.

1: https://search.marginalia.nu

Re:EU Open Web Search project kicked off by harha on Wednesday September 21, 2022 @01:21AM (#32921078)
Taxes are taking from funds for personal projects, and universities already get a lot of funding in Europe to do research, maybe they should focus on creating a better environment for students to become researchers or entrepreneurs instead.

Overall I think there are better ways to improve search from an EU perspective by doing what they are supposed to be doing:

- create a fair environment for companies to compete in, e.g., take a look how Google, Apple or Meta's assets are set up to make it harder for competitors, break that up

- improve standards in eduction – it doesn't really make sense for all member countries to think of and maintain a good CS curriculum and they all seem to be pretty bad at it

- make it easier to build something and get funded, and reward creating prosperity, don't tax it to death

Just tested marginalia's random mode btw. Pretty cool, reminds me of the internet when I was a kid

(edit: formatting)

Re:EU Open Web Search project kicked off by olalonde on Tuesday September 20, 2022 @10:58PM (#32920363)
I don't think the US funded Google?
Re:EU Open Web Search project kicked off by arjenpdevries on Wednesday September 21, 2022 @02:46AM (#32921559)
Google started out of a CIA funded Stanford project.
Re:EU Open Web Search project kicked off by ben_w on Wednesday September 21, 2022 @03:20AM (#32921761)
CIA? Other people are saying DARPA and NASA got together to fund the NSF which funded the PhDs of the founders of Google, but even that's a bit too indirect IMO. Where does the CIA fit in?
Re:EU Open Web Search project kicked off by arjenpdevries on Wednesday September 21, 2022 @11:51AM (#32926912)
https://qz.com/1145669/googles-true-origin-partly-lies-in-ci...

Enjoy! It's a great story.

(Plus: for who might not know, DARPA is US defense research, and heavily influenced by the intelligence services needs. Which is not necessarily bad! Just good to understand where and how Google originated. And wrt DARPA, they funded the creation of the internet itself, for whatever matters.

In Europe, things often go slightly different. The Web is a result of CERN, who are also a project partner of OpenWebSearch.EU. Why? Well, better search can also be beneficial for better science, not just for end users wanting to find their way or buying something.)

Re:EU Open Web Search project kicked off by ben_w on Wednesday September 21, 2022 @02:11PM (#32928955)
Thanks :)
Re:EU Open Web Search project kicked off by hkt on Tuesday September 20, 2022 @05:08PM (#32917809)
..correct me if I'm wrong, but Google was started by a couple of postdoctoral researchers, no?
Re:EU Open Web Search project kicked off by DannyBee on Tuesday September 20, 2022 @05:24PM (#32917987)
Who deliberately did not stay in academia to do it. More to the point, a successful team building a product like a search engine requires roles that academia doesn't really have.

Who is doing product management?

Who is doing product marketing?

etc

This is all applied engineering at this point, not R&D. How does it at all fit into academia's strong suit?

Re:EU Open Web Search project kicked off by FridgeSeal on Tuesday September 20, 2022 @07:15PM (#32919027)
I think that maybe the point is this is not being tackled as a purely economic endeavour (or if it is, it’s in the “indirect” manner), as such, I suspect roles like “product marketing” are probably unnecessary, at least for now.

Also, tell me you wouldn’t love to work on a large project that wouldn’t be subject to the arbitrary whims and promises of the marketing department.

Re:EU Open Web Search project kicked off by JumpCrisscross on Tuesday September 20, 2022 @08:13PM (#32919472)
> the point is this is not being tackled as a purely economic endeavour (or if it is, it’s in the “indirect” manner), as such, I suspect roles like “product marketing” are probably unnecessary, at least for now

Which results in an interesting engine nobody uses. Products that start with the tech and then think of selling it fall on their faces for a reason.

Re:EU Open Web Search project kicked off by viraptor on Wednesday September 21, 2022 @12:37AM (#32920844)
Google started with tech and disdain for online advertising. They didn't start with the ads.
Re:EU Open Web Search project kicked off by DannyBee on Wednesday September 21, 2022 @07:56AM (#32923754)
and were forced to pivot because it didn't work to do it that way, empirically disproving your thesis?
Re:EU Open Web Search project kicked off by viraptor on Wednesday September 21, 2022 @05:36PM (#32931830)
They weren't forced to pivot. Gates explained to they how much money is to be made and they changed their mind. For some time they were very excited how unobtrusive and helpful the ads were. Then they realised there's even more money and the rest is history.

Is Google proves anything, it's that greed is real.

Re:EU Open Web Search project kicked off by mkl95 on Tuesday September 20, 2022 @04:57PM (#32917698)
> 14 European research and computing centers

> 7 countries.

> 25+ people.

There are literally dozens of them!

https://openwebsearch.eu/partners/

Re:EU Open Web Search project kicked off by marginalia_nu on Tuesday September 20, 2022 @05:10PM (#32917833)
I don't think the number of people or even the size of the budget is wrong. A small team can be incredibly powerful and productive if you have the right people. In fact, I think far more often search engines fail from trying to start too big than too small.

The problem is that you need people who actually know how to architect complex software systems much more than you need revolutionary new algorithms. For that, professors are the wrong people. A professor on the team, sure, that might be helpful. Not half a Manhattan project's worth.

Re:EU Open Web Search project kicked off by thwayunion on Tuesday September 20, 2022 @09:11PM (#32919836)
> For that, professors are the wrong people.

Have no fear; all of the actual work will be done by PhD students straight out of undergrad, and most of the actual leadership will be done by a string of recent PhD grads who need results in 6 months because they'll be full time job marketing for the 6 months after that ;-)

Re:EU Open Web Search project kicked off by mkl95 on Tuesday September 20, 2022 @05:29PM (#32918050)
It happens all the time in Europe. Collaboration between public and private companies is pretty much a pipe dream in the EU. Some company that actually works on building search technology would achieve way more than a bunch of professors.

I disagree on the budget though. It is basically pocket change.

Re:EU Open Web Search project kicked off by marginalia_nu on Tuesday September 20, 2022 @05:50PM (#32918276)
Arguably the biggest most unsolved problem in search is how to make a profit (or even break even). This can be approached in two ways: You can either try to find some way of making search more profitable, or you can find a way to make search cheaper. I think the latter is a lot more plausible than the former.

A shoestring budget keeps the costs down by design and by necessity. A large budget virtually ensures the search engine becomes so expensive to operate it will never break even.

Re:EU Open Web Search project kicked off by qualudeheart on Wednesday September 21, 2022 @02:30AM (#32921463)
Why not offer a paid tier? Seems to work for Kagi. Information elites will soon flock to paid search engines, which won’t be much more expensive than a streaming subscription. I pay for Netflix and am willing to pay for a search engine that offers as good a search service as the video streaming offered by Netflix.
Re:EU Open Web Search project kicked off by mellavora on Wednesday September 21, 2022 @05:06AM (#32922449)
> Arguably the biggest most unsolved problem in search is how to make a profit

And the EU just solved that problem.

Re:EU Open Web Search project kicked off by marginalia_nu on Wednesday September 21, 2022 @10:43AM (#32925880)
Did they though?
Re:EU Open Web Search project kicked off by warinukraine on Wednesday September 21, 2022 @07:38AM (#32923608)
> Im a bit skeptical EU-funding a bunch of professors is the way a search engine will be built.

Worked for Google.