Information below is about the system used in Library of Alexandria but other libraries of that time, e.g. the one in Pergamum, was probably organized similarly: Each roll had a tab on the end (the circular end of the scroll) that gave the author's name and some other information, but often no title. Problem was that many rolls contained more than one work and many works took many rolls. Famous librarians have invented techniques to make the job of finding a work easier, since there were 490,000 rolls in the main library! The main library was reserved for the scholars of the Museum, while its sister library, with about 43k books, was open for public use.

Zenodotus (https://en.wikipedia.org/wiki/Zenodotus) the Library's first director, invented alphabetical order as a mode of organization. Rolls were also arranged in different rooms, by topic, e.g. poetry, history. Another important device was bibliographic work, such as the Pinakes (Tables) of Callimachus (https://en.wikipedia.org/wiki/Pinakes), which filled no less than 120 books (it has not survived but was widely quoted).

If you want to delve deeper, Libraries in the Ancient World by Lionel Casson, from which I took the informatio above, is a very informative and readable book. It even has the only known depiction of how the rolls were arranged on shelves.*