News for hackers. Stuff that matters.

rishiloyola

No I am not using Kafka. It is just basic python server. I want to calculate what is the nth percentile of size of my incoming request object over past one hour.

I don't want to store size of each request in memory. It will eat so much of my RAM.

Incoming traffic:

- 1st batch

--> 60 requests

---> size of 1st request is 10kb

---> size of 2nd request is 2kb

...

...

- 2nd batch

--> 10 requests

---> size of 1st request is 5kb

---> size of 2nd request is 8kb

...

- 100th batch

I am talking about percentile(10th, 50th, 95th) size of request.

Ask HN: What is the best way to calculate percentile of streaming data?

Hello,I need to code python function which will iterate through incoming requests and calculate percentile of size of body dynamically. Which lib or algo do you guy recommend?Example: Requests are coming in batches. Let's say first batch has 50 requests, next one has 80 etc. I need to calculate percentile of size of body that each request has.

( Read More... | 2 comments | on HN )
How do you handle back pressure?

Do you have specific module for it?

Well logstash is now supporting good persistent queue. https://www.elastic.co/guide/en/logstash/current/persistent-...

I don't know why author didn't put correctness tick mark on it.

yeah not even their blogs.

Ask HN: Which system books developers must read?

Hello,I have interest in low level systems and Distributed Systems. I want to go deep into these fields. I started reading following books1. Linux Kernel Development by Robert Love2. Designing data-intensive applicationsCan someone recommend me more books to increase my knowledge?

( Read More... | 1 comment | 2 points | on HN )
Re:Ask HN: What do you do with your Raspberry Pi? on Monday June 24, 2019 @03:34PM (#20267535)
So finally did you impress that girl or not? :D
Re:Show HN: A LaTeX-based resume building app on Wednesday June 12, 2019 @08:26AM (#20163814)
Not able to access it. Nginx error - 502 Bad Gateway

Ask HN: How did you implement GDPR compliant Global Log Collection system?

Hello Guys, I am running hundreds of servers across the globe. Right now all these servers are pushing the data to centralized ELK cluster. I want to make a regional log collection system. Like all EU servers should send all logs to the EU cluster likewise for Japan, US, India servers.Maintaining ELK clusters per region can be a pain and paying a huge amount to AWS Elastic Search Service is not...

( Read More... | 1 comment | 2 points | on HN )

Ask HN: How can I become a good Site Reliability Engineer?

Hello,I just started my career as an SRE. I am really passionate about this field. I do watch videos and read blogs regularly related to different architectures to increase my knowledge.I want guidance from the HN community to become better SRE. What should I do in my free time? Which blogs, articles, books, videos you recommend me to watch? Please consider me as a beginner in this field. I am...

( Read More... | 1 comment | 2 points | on HN )

Ask HN: What is a cool thing to do in Paris?

Hello,I just moved to Paris. I am having a hard time over here because I don't have any friends and family over here. I would like to ask the HN community about what is a cool thing to do in Paris? What do you do usually on weekends in Paris? Is there any techie group, club or museum?Of course, I can google the same question or find events on meetup.com but I want to hear it from HN...

( Read More... | 5 comments | on HN )
Re:The Emotional Engineer on Monday May 20, 2019 @05:32PM (#19965448)
No, you are not the only one here :-)

Yeah, sometimes I am really pissed off because many people just try to find issues in my PR. They don't care for the rest of the work. I would suggest you talk to your manager about this topic. You need a better environment and which is fine.

Ask HN: How did you implement GDPR compliant Global Log Collection system?

Hello Guys,I am running hundreds of servers across the globe. Right now all these servers are pushing the data to centralized ELK cluster. I want to make a regional log collection system. Like all EU servers should send all logs to the EU cluster likewise for Japan servers.Maintaining ELK clusters per region can be a pain and paying a huge amount to AWS Elastic Search Service is not a good...

( Read More... | no comments | 3 points | on HN )
Yeah, the harsh truth.

Ask HN: Hire Us?

We are a small group of engineers out of Paris. We are expert at cloud costing and handling the large infrastructure. We are about to start our consultancy service. I would like to take help of the hackernews community to get the first few clients. We can help with hosting, cloud costing, web development, search engines, scaling issues or normal devops stuff. We will soon launch our website as...

( Read More... | 1 comment | 2 points | on HN )

Ask HN: How to start your own software/ML/Web consultancy company?

I am 22 years old. I have good knowledge of systems, web development, and backend development. One of my friends is good at ML and AI. We are thinking to start our own consultancy company to help other companies to build their products. Any advice I can get from HN community? How to get the first client? How to make new connections for our business?

( Read More... | 2 comments | 5 points | on HN )
Re:Ask HN: Who is hiring? (March 2019) on Saturday March 02, 2019 @06:57AM (#19288846)
Hey I am interested in working with you remotely part-time(20-25 hours). How to contact you? Mention your email or website.

Older »