They could have taken off and accelerated to record-breaking speeds if that's all they wanted to do, but the typical mission profile was to fly to far away locations to conduct surveillance, so in-air refueling was always part of the mission profile.
TLDR: They needed full tanks (more or less) or an inert layer of nitrogen over the gas in the tanks before hitting Mach 3 or else the fumes in the tank might ignite. At the end you'll see that they had a system for fueling the tanks on the ground in order to hit mach 3 right after takeoff but it was a maintenance nightmare.
If I remember right, the SR71 could fly at Mach 3 for up to 90 minutes between refuelings. That's significantly longer than any jet ever made.
Except the XB-70. Data is rare, but it was to cruise at 3.0 for somewhere between one and two hours.
There are many faster objects, including the X-15 and Shuttle. Both are faster than the Blackbird but do not count. Shuttle is a spacecraft but even during reentry, shuttle could not hit such speeds unaided by other aircraft that departed from it along the way (the boosters etx). Nor could the X-15 which was dropped from a larger carrier aircraft. Both could not get to their records speed unaided.
Neither could the SR-71/YF-12. They burnt so much fuel getting off the ground that, to preform a run to top speed and get home, they needed to refuel. So how is that fundamentally different than Shuttle or the X-15? For an aircraft to qualify as an aircraft it should be able to get up and down without help. Therefore it is the Mig-25/31 that deserves the praise of "fastest aircraft".