Running Large-Scale Graph Analytics with Memgraph and Nvidia CuGraph Algorithms

Re:Running Large-Scale Graph Analytics with Memgraph and Nvi... by JacobiX on Wednesday August 17, 2022 @10:23AM (#32496810)

This is the first time I hear about memgraph, its really interesting because I'm beginning a project that involves the analysis of some large graphs. Sometimes, before using a lib, I dig into the GitHub repo. So here, I viewed their implementation of a ring buffer (memgraph/src/data_structures/ring_buffer.hpp).

I was surprised with what I found, especially the sleep after adding an element and the loop just before a spin-lock. Not sure, if this is a good implementation, I hope if a more knowledgable person could explain to me this piece of code.

```

  template <typename... TArgs>
  void emplace(TArgs &&...args) {

    while (true) {
      {
        std::lock_guard<memgraph::utils::SpinLock> guard(lock_);
        if (size_ < capacity_) {
          buffer_[write_pos_++] = TElement(std::forward<TArgs>(args)...);
          write_pos_ %= capacity_;
          size_++;
          return;
        }
      }

      SPDLOG_WARN("RingBuffer full: worker waiting");

      // Sleep time determined using tests/benchmark/ring_buffer.cpp

      std::this_thread::sleep_for(std::chrono::microseconds(250));

    }
  }

```

[ Parent | on HN ]

Re:Running Large-Scale Graph Analytics with Memgraph and Nvi... by mbuda on Wednesday August 17, 2022 @10:53AM (#32497229)

Nice observation! That sleep is there in the case the queue is full, which means the caller can't put an additional element into it. The only option is to wait and see if the queue becomes less full... The hard question is for how long? Here we just added a fixed amount which was/is reasonable for our use-cases. Maybe/probably a better would be to implement exponential backoff https://en.wikipedia.org/wiki/Exponential_backoff, but that's a bit more complex (not too much). We never hit any limitation with the current implementation, if we do, we'll improve further!

[ Parent | on HN ]