Are you sure it was the xc3000 that was documented and not the xc6200 series (e.g. xc6264)? I remember the xc3000 being documented the usual amount for their parts (pinout, architectural description, call for info about licensing the software you need to have to use the thing), but the xc6200 datasheet definitely included details about the bits and is still floating around: https://duckduckgo.com/?q=xc6200.pdf

Can you find the "programming and layout specs docs," online or physical copies? I can't find any mention of them anywhere. I'm very curious to see them.

But in how far didn't it do well for Xilinx?

Well the guy I know who used to be in charge of strategic marketing for Xilinx shared that it had not met sales projections and support costs were higher (which cut into margin) when people were trying to program the things with their own tools. So sharing all that stuff about the chip alienated some big customers, reduced profit margins, and didn't help volumes. That pretty much defines 'fail' in the chip business.

This is an interesting story. An unfortunate end. Most of these things sound like short term issues that get resolved as the tools are made. Alternately Xilinx could have just put the docs out there and said they only support ISE. That way if an open source developer makes a tool they have no responsibility, but can reconsider in a few years in case it gets wide acceptance. I am not a market strategist, but supporting everyone's custom tool sounds like a quick way to waste money and the wrong way to try and open source something. Similar to respecting every coupon people bring in, even hand written ones.