sighingnow
Re:GraphScope Journey: Visualizing Progress&Lessons in Graph Computing...
on Wednesday July 31, 2024 @10:32AM (#41119519)
We have unveiled a new webpage that tells the story of GraphScope, charting its course from inception to its future prospects in the field of graph computing. The page explores how Alibaba Cloud has developed GraphScope from its early stages, overcoming challenges and achieving significant milestones to become a crucial tool in graph data processing. This journey highlights not only technological advancements but also practical lessons in scaling, performance, and adaptability.
Re:Why async gradient update doesn't get popular in LLM community?
on Tuesday October 10, 2023 @08:43AM (#37831331)
The pipedream-2bw paper[1] and the zero-offload paper[2] both show that 1-step delayed asynchronous gradient update doesn't affect the convergence (and perplexity) while improve the training efficiency (by fully utilize the bubbles in pipeline parallelism) at a large margin.
However, both the Megatron-LM[3] and the DeepSpeed[4] don't use pipedream-2bw scheduling. Could anyone share me some insights or ideas about why such an efficient scheduling scheme doesn't get popular in the LLM pretraining community? Does it suffer convergence/accuracy issue in practice? Or are there any other concerns that blocking it become the default / most popular pipeline parallelism scheduling?
[1]: https://arxiv.org/abs/2006.09503
[2]: https://arxiv.org/abs/2101.06840
Why async gradient update doesn't get popular in LLM community?
Show HN: Visualizing pipeline parallel algorithms: GPipe, 1F1B (& -2bw, & eager)
https://gist.github.com/sighingnow/ce6aef69aaa379517ac4b68bc2873a9c
Visualizing how pipeline mode parallel works and how batches are scheduled under various scheduling scheme: GPipe (F-then-B), Pipedream (1F1b), Pipedrea-2BW (no flushes between iterations) and Eager-1F1B (better computation-communication overlapping).
Show HN: Static_str_ops: &'static str and non-const operations in Rust
https://github.com/unsafecoerce/static_str_ops
It is often asked by Rust programmer that how to create `&'static str` in Rust with non-const operations at runtime, e.g., returns the result of `format!()` as `&'static str`, rather than `String`.The crate static_str_ops addressed this issue, by allocating a hash set under the hood, and return the reference as the result. Along with this crate, a set of utilities are provided,...
Show HN: Efficient intermediate data sharing for Kedro pipelines
Data processing pipelines are becoming increasingly complex, and intermediate data sharing is becoming the bottleneck, especially for data-intensive analytics and data preprocessing in machine learning and AI.This blog shows the possibility of efficient data sharing in data science pipelines, which naturally fits the settings of Kubernetes. It demonstrates how existing codebases can benefit...
Alibaba Doubles LDBC SNB Interactive Benchmark Record
Re:Show HN: Graphlearn-for-PyTorch, distributed graph learning on PyTorch
on Monday May 15, 2023 @11:07PM (#35957347)
Optimizing distributed sampling and feature lookup looks really attractive. It's really challenging to deploy GNN training at an industrial-scale for a large graph.
Will GLT be part of graphscope[1] and replacing the current graphscope-for-learning implementation?
Show HN: GraphAr – Open-source file format for archiving/exchanging graph data
https://github.com/alibaba/GraphAr
Hi, Hacker News! We're excited to announce the release of GraphAr, an open-source file format for archiving and exchanging graph data.The landscape of graph processing systems is fragmented, with various types of systems, including graph databases, graph computation systems, and GNN systems. However, currently, there is no common file format for efficiently storing and exchanging graph data...
Tencent TDSQL breaks TPC-C record
Re:Tencent TDSQL breaks TPC-C record
on Sunday April 02, 2023 @11:31PM (#35418240)
On March 30th, the Transaction Processing Performance Council (TPC) released its latest rankings: Tencent Cloud's TDSQL set a new record with a performance of 814 million tpmC and a cost of 1.27 RMB per tpmC. This achievement placed TDSQL at the top spot for both tpmC performance and cost-effectiveness in the world.
Re:Show HN: Libobjectfs – zero-dependency, single-artifact library for...
on Sunday April 02, 2023 @05:10AM (#35408620)
More language bindings, e.g., Python and Rust, will be added later. Don't hesitate to open issues for feature request if you are struggling with handle various different I/O and its dependency hell trouble and find this library might be helpful in your application!
Show HN: Libobjectfs – zero-dependency, single-artifact library for many storage
https://github.com/unsafecoerce/objectfs
Introducing libobjectfs - a revolutionary library for interacting with any object storage. With zero system environment dependency and a single artifact of ".a/.so/.dylib/.dll", ObjectFS makes it incredibly easy to integrate into your system and easy to distribute along with your application.Within the single header and single static/shared library artifact, ObjectFS supports a wide range of...
Show HN: GraphAr – Open-source file format for archiving/exchanging graph data
https://github.com/alibaba/GraphAr
Hi, Hacker News! We're excited to announce the release of GraphAr, an open-source file format for archiving and exchanging graph data.The landscape of graph processing systems is fragmented, with diverse types of systems, including graph databases, graph computation systems, and GNN systems. However, there is currently no common file format for efficiently storing and exchanging graph data that...
Show HN: Yet another) (better) ChatGPT action for pull request code view
https://github.com/marketplace/actions/chatgpt-action
chatgpt-action is designed as an AI assistant backed by chatgpt for interesting things like scoring a pull request, reviewing the patch hunks, assign labels to pull request and issues on Github.Compare with existing such actions, this one has the following unique features:- focus on "review", rather than "summarization". It doesn't just talk about what the PR is doing, rather, it tells users...
Ask HN: Is there any catalog engines that is designed for ML and AI?
I would like to know if there's any open source catalog engines is design for ML and use ML datas (datasets and ML models) as the first-class citizen?I have read the "awesome-data-catalogs" list [1] but found none of them has the "ML 1st citizen" property.[1]: https://github.com/opendatadiscovery/awesome-data-catalogs
Ask HN: Is there any data catalog that targets ML as the first citizen?
Hi, I would like to know is there any opensource data catalog systems that targets machine learning applications (datasets (unstructral, e.g., text, image, and video) and models) as the citizen?I have read the awesome-data-catalogs ([1]) list but found none of them is treating ML as 1st cizten and the support for datasets and models are not specific enough.[1]:...
Re:GitHub “allows” unauthorized users “merging” PRs, bypass write perm...
on Thursday August 25, 2022 @10:59PM (#32602884)
Allowing "zero changes" state for a merge request does make sense, as during the development workflow developers may doing some rebase & force push work and they may forget commit changes after reset to main and before push to the pull requests.
Closing the "zero changes" merge request have confused in my previous experience when I continue to push re-added commits to the original pull request branch.
Re:GitHub “allows” unauthorized users “merging” PRs, bypass write perm...
on Thursday August 25, 2022 @10:52PM (#32602837)
The "zero-commit-merges" should be treated as a "Closed" event better than being treated as a "Merged" event.
Re:GitHub “allows” unauthorized users “merging” PRs, bypass write perm...
on Thursday August 25, 2022 @10:49PM (#32602810)
+1 for the feature of marking as merged, as in many projects, e.g., apache arrow, the pull requests are merged in a different way and all pull requests are showed as "closed" rather than "merged".
It would cause some confusion for project management, I guess.
Re:GitHub “allows” unauthorized users “merging” PRs, bypass write perm...
on Thursday August 25, 2022 @10:46PM (#32602789)
You don't need to get accept to "merge" a PR. When pushing the main branch to your PR, the PR would get closed as "Merged" and there will be a "Merged" contribution badage showed in your github profile.
Re:GitHub “allows” unauthorized users “merging” PRs, bypass write perm...
on Thursday August 25, 2022 @10:42PM (#32602770)
The concern here is not about who "commit" this, the concern is who "click the merge button" in this case.
Re:GitHub “allows” unauthorized users “merging” PRs, bypass write perm...
on Thursday August 25, 2022 @11:28AM (#32595023)
> 1) Is it a real problem that allows anyone to push into any repository?
It is not a real problem that allows anyone to push into any repo, but is a real problem that shows a incorrect/unexpected "Merged" pull request status on any repository.
> 2) Is it just a very confusing message?
From the links in paste in the comments you could see that github shows the unauthorized user "merged" the pull request into main, and, the repo's owner received a email says:
FROM: XXXXX Content: Merged #xxx into main.
It is exactly the SAME as email notification of a normal authorized merge event.
Re:GitHub “allows” unauthorized users “merging” PRs, bypass write perm...
on Thursday August 25, 2022 @03:04AM (#32590524)
We recently noticed some strange email notifications from Github with content "Merged #948 into main.", which told that pull requests on our repo has been merged by stranger who actually doesn't have the write permission to our repo!
After further inspection, we found that the merge event is triggered by the creator of the pull request pushing current main commits to the PR's "from" branch.
Moreover, when pushing current main to a pull request,
- The pull request is displayed as "Merged"
- A "PR merged into main" email is sent to all subscribers (mainly the repo owners)
- A "PR merged" contribution is displayed on the creator's Github profile
Closing dangling pull requests is a quite resonable design, but mark it as "Merged" rather than "Closed" would confuse people to let them think they are hacked at the first glance (note that there even an email notification "Merged #xxxx into main" to repo's owners).
If such a feature is misused, it may lead to chaos to more open source repos in the futuer, especially those famous ones.
See some example links in
GitHub “allows” unauthorized users “merging” PRs, bypass write permission check?
GraphScope on Colab: Large-Scale Graph Computing from Notebooks to Kubernetes
Re:GraphScope on Colab: Large-Scale Graph Computing from Notebooks to ...
on Monday December 06, 2021 @08:55AM (#29459741)
We are glad to announce the landing of GraphScope on Colab: https://colab.research.google.com/github/alibaba/GraphScope.
GraphScope is a one-stop graph computing systems from Alibaba aimed to address challenges in large-scale graph computation in real production environments. GraphScope releases v0.9, enabling data scientists to develop graph computing workflows for analytical, interactive query and GNN workloads on small graphs in jupyter notebooks in a interactive manner. Once finishing the development and debugging, users can easily deployed their workflows to Kubernetes with one-line change!
To try GraphScope, you could find it on Colab[1], Jupyter Hub[2], or install GraphScope to your environment using pip by:
pip3 install graphscope
For more details of our v0.9 release, please refer to https://github.com/alibaba/GraphScope/releases/tag/v0.9.0
[1]: https://colab.research.google.com/github/alibaba/GraphScope/...
Re:Vineyard serves as the data back end for airflow
on Sunday November 14, 2021 @09:31PM (#29222968)
Vineyard, an open-source immutable in-memory data-manager that is designed for optimize the data exchange between tasks inside a bigdata analytical workflow, now supports airflow and can serve as the data backend for efficient data sharing between tasks.
See how vineyard helps for ETL jobs that large volume of complex data need to be shared between tasks: https://registry.astronomer.io/dags/v6d-etl-pandas
There's also a tutorial to get started: https://v6d.io/notes/airflow.html