The main problem with this idea is that high skilled labor is conflated with h1B labor. The H1B system is completely abused by big outsourcing companies like Infosys, HCL,TCS and Wipro. The proof of this abuse is that the average salary from these workers is much lower from non-outsourced H1Bs e.g. microsoft workers on H1Bs.

For example, a few years Disney theme parks laid off all of their IT staff and immediately replaced them with HCL H1B workers for lower cost. Clearly in this case H1Bs were abused to lower salaries of American workers by bringing in low skill (defined by salary) labor.

Immigration isn't inherently bad but the H1B program is deeply flawed (allocations by country, dominance by outsourcing companies, workers being tied to employers)