The argument fundamentally stems from the Protestant belief in "Sola Scripture": that all Christian doctrine and belief derives from scripture. Since many protestants prefer a literal interpretation of scripture, they insist on a literal interpretation of Genesis 1 for consistency.
You can see this backed up if you watch any of the material from Answers in Genesis, as I watched at my church from elementary school onward.
You can also see this reinforced by the fact that young earth creationism literally did not exist as a phenomenon in the early church, or the medieval church, or in modern Catholic and Orthodox churches.
Many early church fathers (and before that, Jewish scholars) long wondered whether Genesis 1 recorded literal history, or if it was a literary story. In particular, they took note of the chronology and found it strange that God created day and night (the first day) before he created the sun (the fourth day).
The strongest argument against literal interpretation, I'm, is that the literary structure of Genesis 1 is so clear: God performance three acts of separation (day from night, waters above from waters below, and waters below from themselves with land) followed by three acts of decoration, where he decorates the previously separated things: sun and moon and stars in the sky, birds in the air and fish in water, and then land animals and man.
Yeah, so unfortunate. And then, every Sunday at Bible study, a new teacher will give a different interpretation of the same word.
Genesis reads that God created the heavens and the earth in 'six defined periods of time'. It's unfortunate that the original translators back in the 1600s used the word 'day', and even more unfortunate that so many cling so strongly to what was probably a single person's translation.