Re:We Don’t Use Docker
by SatvikBeri on Tuesday March 16, 2021 @12:07PM
(#26478659)
My solution started out as a 10-line Python script where I would manually clean the data we received, then process it.
CEO: "Will this scale?"
Me: "No, absolutely not, at some point we'll need to hire someone who knows what they're doing."
As time passed and we got more data, I significantly improved the data cleaning portions so that most of it was automated, and the parts that weren't automated would be brought up as suggestions I could quickly handle. I learned the very basics of performance and why `eval` is bad, set up my script so I didn't need to hard-code the number of files to process each day, started storing data on a network drive and then eventually a db...
I still don't know what I'm doing, but by the time I left it took maybe 5 minutes of manual data cleaning to handle thousands of jobs a day, and then the remainder could be done on a single machine.
Me: "No, absolutely not, at some point we'll need to hire someone who knows what they're doing."
As time passed and we got more data, I significantly improved the data cleaning portions so that most of it was automated, and the parts that weren't automated would be brought up as suggestions I could quickly handle. I learned the very basics of performance and why `eval` is bad, set up my script so I didn't need to hard-code the number of files to process each day, started storing data on a network drive and then eventually a db...
I still don't know what I'm doing, but by the time I left it took maybe 5 minutes of manual data cleaning to handle thousands of jobs a day, and then the remainder could be done on a single machine.