News for hackers. Stuff that matters.
Re:Library-managed 'arXiv' spreads scientific advances rapid... by CJefferson on Tuesday December 27, 2016 @03:35PM (#13265969)
Can I just make a general plea?

You should upload your paper to arXiv. When you do, please upload your source (tex, or word I imagine), as well as a PDF.

For the blind, PDF is the worst possible format, and tex and word are the best formats. Don't hide, or lose, the blind-accessible version of your paper.

Re:Library-managed 'arXiv' spreads scientific advances rapid... by apathy on Tuesday December 27, 2016 @04:11PM (#13266213)
When I worked at the Cornell Theory Center in the late 90s, I didn't know about the arXiv, but I sure as hell knew about screen readers. One of the best FORTRAN programmers (numerical analysis/applied math research faculty I think) was blind as a bat.

I learned a lot about not underestimating people there.

It's a strange university but it always makes me happy to think that it (specifically Ginsparg, and to some degree strogatz and the law scholars) pushed forward what the Web was supposed to be.

Not a place to buy stuff, but a place to learn stuff, without waiting for it to make its way through a hyper politicized review process so that it could be printed in a 17th century fashion and mailed to some corners of the world, eventually perhaps reaching a fraction of the people who could use it. Rather, everyone everywhere with a connection.

Anyone who doesn't deposit preprints (arXiv, biorxiv, or wherever) or who doesn't agitate for their coauthors to do so is not really in it for the science. It's fine to be competitive -- deposit yours first. Make the fucking discovery instead of parasitically piggybacking on those who do the work.

But that last part, that's hard. Very hard. As long as there is enough money left in academia to encourage lazy shits, those of us who care about scholarship will have to push, hard, to remove the last refuge of these scoundrels.

You're either on the side of justice -- open data, open formats, open scholarship -- or you are tacitly endorsing Elsevier & Springer, who haven't the slightest problem using crap like incremental JavaScript & mangled PDFs to deny access to scholarship even to those who have paid.

David (blocking on his last name) at CTC made that choice for me. He set an example that forced me to admit what was right. I hope others will do the same. It's the right thing to do.

Re:Library-managed 'arXiv' spreads scientific advances rapid... by frozenport on Tuesday December 27, 2016 @11:47PM (#13268459)
The OpenAcess and self-publication has removed an important check on research papers. If nobody cared about your subject, or if a journal's peer review process resulted in boring articles, it would not be financially viable.

I think advocates of sharing science should also understand that, at present, the majority of the content is asinine or incorrect. Removing checks and balances is not helping.

Re:Library-managed 'arXiv' spreads scientific advances rapid... by apathy on Wednesday December 28, 2016 @12:57AM (#13268735)
The majority of content in most journals is asinine or incorrect. About the only thing that can fix this is quicker turnaround (cf STAP, NgAgo, the godawful GWAS paper in Neuron recently, etc).

It's not at all clear to me that the checks or balances were helping beforehand. Hiring a shit ton of MBA types to administer research probably didn't help much either.

Also, out of curiosity, is this open access / fast review problem what led to machine learning falling into irrelevance?

Because we wouldn't want that to happen to other fields. Can you imagine if raw clinical trial data was out there, for example? (The Horror! Best keep these appallingly expensive results locked up!)

Re:Library-managed 'arXiv' spreads scientific advances rapid... by frozenport on Wednesday December 28, 2016 @11:44PM (#13275670)
>>fix this is quicker turnaround

Why? As the first author of several publications, more than half the reviews I received were not well thought out. This part of the process doesn't happen, Open Acess and self publishing websites exploited this fact for greater throughput.

>>machine learning

I would argue that the machine learning bloom in recent years is because of an availability of new technology from industry, supported by industry. This is also why they were able to tolerate less conventional venues for publication - as these were not viewed as mile markers for progress. Yet, when I look at more "settled" machine learning fields like speech, I see the same plethora of academic papers filled with useless crap.

Perhaps the growth of ML has nothing to do with Open Acess. There is although certainly there is an underlying profit motive for successful technologies, Google ain't going to use a technique that doesn't work.

>>raw clinical trial data

It would be great if it were published, and indeed many people would pay for quality trial data. I see OA and self-publishing of research results having no relation to authors withholding clinical data.

Re:Library-managed 'arXiv' spreads scientific advances rapid... by apathy on Thursday December 29, 2016 @01:33AM (#13276006)
Re: 1: no shit. The turnaround is for rebuttals. I try to do thorough reviews (I have published plenty as well) but realistically I recognize that no one has the sort of free cycles required to catch everything in advance. What's worse is that an awful lot of reviews are pure political bullshit.

In the end it's the people who may or may not try to build on your work (or mine!) that are the best judges of whether it's BS.

Re: 2: probably, but diffusion of ideas benefits massively from readily accessible documentation. One of the things that will cause me to immediately reject a paper is if its method implementation does not work as described. I will always reject in such cases. They are alarmingly common and this is one of the few easy cases in reviewing. Lack of an implementation is also cause to reject.

Re: 3: it has everything to do with it. Unless you're running and publishing trials please don't lecture me on this. See opentrials for a particularly frightening take on why published trials so poorly represent those submitted to the FDA or registered with CTEP. Meanwhile actual RCTs get buried by shitbags demanding more subjects than there are recorded cases in the past 40 years (no joke).

If you're not doing trials, you may have to take my word for it (or get involved in RCTs and find out for yourself). A lot of the enterprise is actively harmful to science by omission.

Re:Library-managed 'arXiv' spreads scientific advances rapid... by nileshtrivedi on Wednesday December 28, 2016 @10:37AM (#13271136)
This argument applies to all speech that has now become easier and cheaper:

"If nobody cared about your opinion, it would not be financially viable to publish it. But now we have blogs/tweets."

Effectively, cost of publishing was working as a quality filter. Now, we need to actually implement proper filters instead of bad proxies for quality.

Re:Library-managed 'arXiv' spreads scientific advances rapid... by tripzilch on Wednesday December 28, 2016 @10:03AM (#13270879)
Could you try and make that argument without the term "financially viable"? We're talking about science here. Capitalism is a tiny thing that really isn't required to rule everything, and particularly in this case I don't think it has any authority based on which to implement that "check".

If I just leave that out you seem to imply science publications should be selected by whether enough people care about, or whether they are boring or not. Both (especially the latter) are rather bad measuring sticks for the quality and importance of research.

Adding "financially viable" to "less boring" produces "EXCITING!", adding it to "enough people care" yields "POPULAR!". These are definitely not forces that should be pulling scientific research.

Not saying there shouldn't be checks. Just that market forces are probably too stupid for it.

Re:Library-managed 'arXiv' spreads scientific advances rapid... by apathy on Wednesday December 28, 2016 @02:26PM (#13272921)
CNS (Cell, Nature, Science) == Buzzfeed science

Although that's not giving BuzzFeed enough credit, they tend to check their sources & verify results/reports. Also if a piece of fake news is simply resubmitted, they don't pretend like they never checked on it before.

There are plenty of checks (mostly from NIH, mostly for incremental garbage work on dead models like cell lines) and very few balances against the tyranny of CNS...

Re:Library-managed 'arXiv' spreads scientific advances rapid... by grigjd3 on Wednesday December 28, 2016 @10:05AM (#13270899)
arXiv isn't meant to replace peer review. When I submitted to phys. rev. d, a submission to arXiv to allow people to see the work ahead of review was expected.
Re:Library-managed 'arXiv' spreads scientific advances rapid... by mixedmath on Tuesday December 27, 2016 @05:03PM (#13266584)
I'm a (PhD student) mathematician and every paper I've uploaded has included the TeX. I looked at about 10 papers in math at random, and 9/10 of them also included the TeX.

But this isn't something I've looked for in the past. So I wonder: in your experience, about what percentage of papers on the arXiv have included the source?

Re:Library-managed 'arXiv' spreads scientific advances rapid... by johncolanduoni on Tuesday December 27, 2016 @10:40PM (#13268269)
It's far less common in physics :/
Re:Library-managed 'arXiv' spreads scientific advances rapid... by frozenport on Tuesday December 27, 2016 @11:49PM (#13268470)
Almost nobody in experimental life sciences uses Tex. My advisor refused to read anything that wasn't in MS Word.
Re:Library-managed 'arXiv' spreads scientific advances rapid... by jmnicholson on Wednesday December 28, 2016 @09:52AM (#13270791)
We're working to provide a Word-like interface on git repositories for researchers to write their manuscripts in at authorea.com. Our editor allows researchers to write in markdown, LaTeX, or rich text all within one article.
Re:Library-managed 'arXiv' spreads scientific advances rapid... by hackuser on Tuesday December 27, 2016 @04:33PM (#13266388)
> For the blind, PDF is the worst possible format

I'm surprised. Nobody has created an accessibility solution for PDFs after all these years of ubiquity? What's the story?

Re:Library-managed 'arXiv' spreads scientific advances rapid... by contravariant on Tuesday December 27, 2016 @07:39PM (#13267544)
Well, I don't know about PDF, but PostScript, which it's based on is basically just a programming language for drawing symbols in specific positions on a page. Depending on how it was written (or more likely, generated) this could be readable, or incredibly unreadable.

As an example, in post script the following example from Wikipedia would simply show the text Hello World:

     %!PS  
     /Courier             % name the desired font  
     20 selectfont        % choose the size in points and establish   
                          % the font as the current one  
     72 500 moveto        % position the current point at   
                          % coordinates 72, 500 (the origin is at the 
                          % lower-left corner of the page)  
     (Hello world!) show  % stroke the text in parentheses  
     showpage             % print all on the page
Now if you're lucky you can just extract all quoted text and read those in order, but that's unlikely to work for all documents.
Re:Library-managed 'arXiv' spreads scientific advances rapid... by hackuser on Tuesday December 27, 2016 @08:22PM (#13267746)
Almost all formats would similarly require content to be parsed from code; for example, consider HTML, Word, or Excel.

It makes me wonder how screen readers work. Thinking out loud, it seems that the screen readers should let the applications (e.g., Word) handle their own parsing and presentation and obtain the data after that. Otherwise, the screen reader would have to reinvent many wheels, interpreting the code for all applications including all their versions, features, quirks, and platform integration issues - such a daunting and difficult task that it seems unlikely. But where do screen readers hook into the content? After it's output from the application but before it's an image for the screen (which could require OCR)? Ironically, I suppose PostScript or PDF could provide common interfaces.

Re:Library-managed 'arXiv' spreads scientific advances rapid... by contravariant on Tuesday December 27, 2016 @08:45PM (#13267836)
Sure all formats require some parsing to get to the content. However Tex, HTML, Word, Excel and languages like that were designed to allow people to format text and other data into a document. Therefore they generally make text and other information appear sequentially, and separate display logic from content.

PostScript and PDF have no such separation, the are the display logic. They are fully fledged programs that list the position, size, font, colour, of every symbol on every page, if you're lucky in a vaguely logical order, but there's no reason it should be. If the files were written by a human you may have some hope of extracting some of the content, but almost nobody writes PDF or PostScript by hand any-more.

Re:Library-managed 'arXiv' spreads scientific advances rapid... by swiley on Tuesday December 27, 2016 @08:42PM (#13267823)
It's not that there's code, it's that word and HTML are document formats while PostScript/PDF is a vector graphics format. Generally, if you remove the formatting tags from HTML or word (leaving just the text) the characters are extremely likely to be in the same order as the rendered document (sans things like running head and page numbers). Furthermore, HTML and word both explicitly delimit things like paragraphs while PostScript just changes the drawing position.
Re:Library-managed 'arXiv' spreads scientific advances rapid... by CJefferson on Wednesday December 28, 2016 @08:13AM (#13270236)
I think the difference with word, excel and Tex is that because they are editable, the content must be stored in a way where the flow is expressed, and each part can be clearly broken into its parts.

With PDFs, all you have is the position of lines and characters on the page. There is no flow, ordering, or semantics.

Re:Library-managed 'arXiv' spreads scientific advances rapid... by witty_username on Wednesday December 28, 2016 @01:35AM (#13268856)
Well, less can extract the text; so I don't see why that's an issue.
Re:Library-managed 'arXiv' spreads scientific advances rapid... by vsl on Wednesday December 28, 2016 @03:32AM (#13269265)
It’s only an issue with crap PDFs (it’s possible to omit text information or obfuscate it to the point you copy & paste garbage out of them; pdfTeX-created PDFs are of course fine).
Re:Library-managed 'arXiv' spreads scientific advances rapid... by CJefferson on Wednesday December 28, 2016 @08:14AM (#13270244)
It doesn't work often for multi column pdfs or tables, ligatures are usually misparsed, and maths is just destroyed.
Re:Library-managed 'arXiv' spreads scientific advances rapid... by noamyoungerm on Wednesday December 28, 2016 @07:03AM (#13269982)
It's totally possible (and a relatively frequent occurrence) to have pdfs where the order of characters in the code has no relationship at all to how those same characters are laid out visually on the page. Anything marginally more complex than a series of paragraphs with no formatting at all basically requires you to render out the whole pdf and figure out the order that you are actually supposed to read the characters in.
Re:Library-managed 'arXiv' spreads scientific advances rapid... by tripzilch on Wednesday December 28, 2016 @10:10AM (#13270933)
Yup. For instance, Word's PDF output, has an absolutely positioned textbox for every word (and sometimes sub-word). This is for kerning purposes. If you want your original text back, you're going to need some OCR-like preprocessing and heuristics to guess what textboxes belong to the same line. If you have multiple columns, good luck distinguishing them from accidental rivers.

It's not impossible, but I wouldn't know immediately what tools get this most right. And it's always a lossy operation going back and forth.

Re:Library-managed 'arXiv' spreads scientific advances rapid... by robin-berjon on Tuesday December 27, 2016 @05:37PM (#13266829)
PDF is only accessible if it is specifically crafted to be so (and then again, I don't think that goes very far). To the best of my knowledge only Adobe's tools and Word actually output that (and at least in the latter they tend to look like train-wrecks, which may or may not be PDF's fault more than Word's).
Re:Library-managed 'arXiv' spreads scientific advances rapid... by cygx on Tuesday December 27, 2016 @05:12PM (#13266650)
They probably can get at the text, but I have doubts about any formulae...
Re:Library-managed 'arXiv' spreads scientific advances rapid... by chime on Tuesday December 27, 2016 @04:02PM (#13266155)
It would be great if arXiv upload screen said this. Not sure if it does or not.
Re:Library-managed 'arXiv' spreads scientific advances rapid... by sibrahim on Tuesday December 27, 2016 @05:38PM (#13266847)
That's certainly been my experience.

I've had arXiv automatically block bare uploads of TeX-derived PDFs (presumably identified through PDF metadata). For these, it required that the source be uploaded and compiled on their server.

On balance, it's probably best to require source as arXiv does, but this can create interesting issues from time to time. Since the source is downloadable, researchers can inadvertently end up sharing partial results or snark that was commented out in the TeX source.

Re:Library-managed 'arXiv' spreads scientific advances rapid... by kkylin on Wednesday December 28, 2016 @09:59AM (#13270848)
I haven't uploaded a paper in a while (coauthors have been doing it more recently), but I'm pretty sure it encourages uploading document sources somewhere during the process. The issue that sibrahim mentions, about stuff in comments, is real: on a multi-author tex the file tends to end up being pretty messy, with lots of informal remarks and old versions in the comments. I always strip out all the comments before submitting.
Re:Library-managed 'arXiv' spreads scientific advances rapid... by ASipos on Tuesday December 27, 2016 @05:29PM (#13266766)
For the record, if you upload the TEX, arXiv autogenerates the PDF.
Re:Library-managed 'arXiv' spreads scientific advances rapid... by ctchocula on Tuesday December 27, 2016 @05:57PM (#13266979)
The problem I face is that my papers is in multiple tex files in several folders connected to a main.tex using \includes. Is there a convenient option that will take care of this issue?
Re:Library-managed 'arXiv' spreads scientific advances rapid... by rhaps0dy on Tuesday December 27, 2016 @07:02PM (#13267393)
The latexpand perl script, the flatten and flatex programs. Please upload your papers in the future! :)

https://tex.stackexchange.com/questions/21838/replace-inputf...

Re:Library-managed 'arXiv' spreads scientific advances rapid... by xvilka on Wednesday December 28, 2016 @02:05AM (#13268945)
Setting up something like webui for storing those files for every paper, something like gitlab or gogs, could really help. It could add more transparency, ability to add some feedback in the issues, and so on.
Re:Library-managed 'arXiv' spreads scientific advances rapid... by sibrahim on Tuesday December 27, 2016 @07:07PM (#13267417)
I've had success using \import{subdir/}{BasenameInSubdir} from the import package.

The basic (and unfortunate in a xkcd/1479 way) issue appears to be that the arXiv compile server doesn't allow writing to subdirectories. A subdir \include implicitly requires write access for the aux file.

Re:Library-managed 'arXiv' spreads scientific advances rapid... by robin-berjon on Tuesday December 27, 2016 @05:39PM (#13266851)
I know that Word has decent accessibility built in (because Microsoft actually cares about this) but I'm surprised that you're getting mileage out of TeX which is a very visual format. Do you basically screen-read the source? Or is there a non-visual output for it that works?
Re:Library-managed 'arXiv' spreads scientific advances rapid... by CJefferson on Tuesday December 27, 2016 @05:58PM (#13266985)
Yes, you do have to learn Tex to read it, but I don't know of any other format any blind mathematician uses. It is common to teach maths with latex -- while it isn't perfect by any means, it is better than any alternative.
Re:Library-managed 'arXiv' spreads scientific advances rapid... by agumonkey on Tuesday December 27, 2016 @04:33PM (#13266384)
PDF readers manage to parse text back somehow effectively, maybe not on formula / formatting heavy PDFs. Anyway good call, accessibility is not only for mainstream websites. I'm sure the blind dude that aced math classes in college would agree.
Re:Library-managed 'arXiv' spreads scientific advances rapid... by CJefferson on Tuesday December 27, 2016 @06:00PM (#13266995)
Formulas and many tables do very badly.

Would be nice if latex embedded the source of equations into pdfs. Wonder how hard that would be to add?

Re:Library-managed 'arXiv' spreads scientific advances rapid... by agumonkey on Tuesday December 27, 2016 @06:02PM (#13267019)
Embedding anything in PDF doesn't seem hard (considering what a few security talks said).

How large are latex files ? probably not much a few 100KBs. It's easy to add a compressed stream in a PDF. It's a great idea.