nith3n
Re:How to be a -10x Engineer
on Tuesday April 04, 2023 @10:12PM (#35448421)
As a platform team leader that caters to other engineers within the organisation, extensive documentation is what enabled me and my team to be productive.
I answer almost all questions posed to me with a simple have you read this (confluence link )
Re:Ask HN: Should we see the scorecards from our interviews after gett...
on Saturday November 12, 2022 @01:32AM (#33570609)
I saw my interviewers notes in our teams SharePoint.
Mixed emotions after knowing that i was almost not hired for being younger / having less experience than expected for the role. 7 years now in the same company and the same person has claimed that I hired him multiple times over the years.
Re:Ask HN: Big Data 10x tips and tricks
on Tuesday November 08, 2022 @03:38AM (#33517166)
Counter point.
CSV is a great format for humans to comprehend.
For big data systems, CSV is a arguably worse format
Row wise storage doesn't provide compression benefits like columnar storage which can significantly reduce storage needs.
Data could contain the same delimiters ( comma, newline ) as used by the parsers and introduce errors in computations. binary formats( parquet/ORC) can eliminate this issue while providing other benefits as well.
A data quality framework like deequ should help catch errors in data before you introduce the same to downstream applications in ELT.(not infalliable )
For ELT processes, always filter first before other processing.
Plan your partitions according to expected querying patterns. If reports are run country wise , then country is a good partition key or if date wise then date is a good parition key.
Be aware of data skews , you might introduce skew during partitions ( ex : partitioning by country and a few countries have large number of records compared to others.
Random thoughts
Re:How to run a small social network site for your friends
on Thursday December 09, 2021 @12:31AM (#29494162)
I used to run a "for friends only" social network on ning.com a decade ago. Was fun till Ning changed direction.
Re:TikTok owner ByteDance sees its earnings double in 2020
on Friday June 18, 2021 @11:46AM (#27551569)
You should also see the pathetic copycat feature YouTube has created called the shorts. Facebook's attempt with short videos and reels is not much better either.
Why does every company feel the urge to copy this and also fail so badly at this
Re:The rice of the sea: how a tiny grain could change the way humanity...
on Saturday April 10, 2021 @09:58AM (#26761522)
https://en.m.wikipedia.org/wiki/Pokkali_Rice
There are some rice varieties which can tolerate salinity , but not upto sea water levels.