News for hackers. Stuff that matters.
Re:Ask HN: Has anyone successfully pivoted from web dev to A... by akudha on Wednesday July 03, 2024 @01:05PM (#40867890)
Could you please elaborate a bit more? I am your typical web-dev/frontend/backend developer, don't really know much about what is involved in data engineering.

Do you mean collecting, cleaning data? Or setting up databases (if yes, how is it different from me managing my employer's databases, except for size)?

In other words, what does a data engineer do all day?

Re:Ask HN: Has anyone successfully pivoted from web dev to A... by sgaur on Wednesday July 03, 2024 @01:47PM (#40868313)
Several data engineers that I know in this space (AI for business applications) are doing the usual ETL + mapping pipeline related work. Nowadays a lot of them are also having to deal with unstructured data such as textual reviews of products, service quality feedback, policy documents. Data engineers are developing the pipelines for chunking, vectorization, ingestion into RAG pipeline and for LLM training fine tuning. So it's still collecting, cleaning data, but quite different at the next level of detail, in my opinion.
Re:Ask HN: Has anyone successfully pivoted from web dev to A... by akudha on Wednesday July 03, 2024 @02:12PM (#40868569)
Thank you. How does one get into this field? Web dev isn’t that exciting anymore
Re:Ask HN: Has anyone successfully pivoted from web dev to A... by sevensor on Wednesday July 03, 2024 @02:46PM (#40868886)
The data engineers I know started out as data scientists. They discovered that their ability to make models that did anything useful was hampered by low-quality or absent data. Then they discovered that there was a lot more job satisfaction (not to mention much better pay) in making useful data available than there was in clicking the "train" button over and over in slightly different configurations.
Re:Ask HN: Has anyone successfully pivoted from web dev to A... by gaws on Wednesday July 03, 2024 @04:24PM (#40869641)
> How does one get into this field?

Network and meet someone who will give you, a web developer (a profession that is heavily stigmatized in the industry), a chance in a very competitive field.

Re:Ask HN: Has anyone successfully pivoted from web dev to A... by songbird23 on Wednesday July 03, 2024 @04:47PM (#40869821)
I didnt realize web devs are stigmatized in the industry, as a web developer
Re:Ask HN: Has anyone successfully pivoted from web dev to A... by shepherdjerred on Wednesday July 03, 2024 @08:14PM (#40871279)
Web developers are, front end engineers are not.
Re:Ask HN: Has anyone successfully pivoted from web dev to A... by mdeeks on Wednesday July 03, 2024 @05:17PM (#40870100)
They are not. There is a stigma associated with the ecosystem I think.

People who can create beautiful and user-friendly interfaces are extremely valuable and respected.

But my experiences trying to set up a typescript project and the unbelievable number of frameworks and crazy things to navigate (mjs/cjs) made me yearn for the pains of the Python ecosystem again.