Create-react-app is pretty heavy, because it includes support for things you may not need or want (css modules, typescript, etc).

Putting together your own light-weight system isnt hard if you are familiar with the problems these tools solve, but unless you have a site so simple that it shouldn't use react anyway, you'll want the power and features that these tools can provide (automatic source splitting for lazy-loading code on route changes, sane management of language files for internationalization, extensive linting rules for even non-js things like accessibility features in html / jsx, automatic image optimizatuon, etc).