This is true, but there have been a lot of browser bugs in the past with Origin/Referer headers. Relying on those is not as foolproof as a CSRF token, which would require a more severe UXSS-type issue to leak. I wouldn’t advise it.