"He that loveth father or mother more than me is not worthy of me: and he that loveth son or daughter more than me is not worthy of me." Would you look your son in the eye as you lower the knife?
If you believe that God is more loving than you are, it is not at odds at all. In fact, if you want what's best for those around you, you are responsible to obey the Lord first and foremost, as he loves those around you more than you do. He also loves you more than you love yourself.
If, however, you don't trust in God's love, then you're totally right. You'd be obeying someone who you're not sure has your neighbors' best interest at heart, and thus abdicating your duty of care.
God ultimately saved Isaac so I'm not sure why Abraham's sacrifice would be a counter-example to God's love here. We can talk about all the people who actually died, seemingly by God's orders, in the Old Testament, but we don't know what happened to these people ultimately, given redemption and salvation by Christ. Death itself being conquered and eternal happiness being granted makes death a poor example of injustice.
For me, responsibility is irrevocably linked to the idea of care - caring for those other than myself makes me responsible for my actions, inasmuch as they impact those around me. That ranges from those I love (my partner, family, friends) to strangers on the street that I nonetheless "care" for, by recognizing our shared humanity. Care itself comes from the idea of finitude - specifically, the idea of fragility of life in the face of our mortal existence. If I'm being poetic, "living rightly" means a life that is spent caring for others, strengthening the bonds that link us in face of death. The consequences of our actions are felt precisely because of the finitude of our lives - all we've got is each other, and our responsibility is towards one another.
Religious morality of the other hand is best expressed in the story of Abraham being asked to sacrifice his son (it's no wonder the three major religions are called "Abrahamic"). For a deeper dive into the idea, I recommend reading Kierkegaard's retelling of this story in Fear and Trembling - to recap however, God asks Abraham to sacrifice Isaac, they travel to mountain where Abraham binds Isaac and lifts up the knife to kill him. At the last moment, God interrupts Abraham, and a ram is offered as sacrifice instead. This story, perhaps one of the most well-known stories from the Old Testament of the Christian Bible, goes directly against the idea of responsibility in moral judgments - Abraham does not falter, for he fully trusts his Lord, giving up his son in devotion to God's command. The idea of care or responsibility for ones actions is superseded by the belief that the will of God is, in fact, more important than caring for those around you. A true belief in an everlasting life presupposes a detachment from the world (as any cursory reading of writers such as Saint Augustine shows), in a way protecting and insulating the believer from truly coming to terms with the consequence of their actions (in this case, murdering their only son).
To sum it up, I just can't get behind the idea that a final judge is somehow required for having responsibility in moral judgments. In fact, the opposite is true - only through the secular faith that there is no better everlasting world can we form meaningful relationships and take full responsibility for our actions.