Even better (with Python 3.5 or newer):

    from contextlib import suppress

    with suppress(OSError):
        os.remove(temp_file)

More generally, context managers are the right way to refactor repeated try/except/finally clauses.