News for hackers. Stuff that matters.
Re:The Geometry of Categorical and Hierarchical Concepts in ... by cs702 on Tuesday June 11, 2024 @11:00AM (#40646965)
Very nice. Well-written. Feels "natural."

Besides helping with interpretability, my immediate thought is that maybe we could pretrain models faster by adding regularization terms in the objective function that induce representations of distinct categories to be in subspaces that are orthogonal to each other, and representations of subcategories to be in orthogonal subspaces that can form polytopes. The data necessary for doing so is readily available: Wordnet synsets. Induce representations of synsets to be orthogonal to each other and representations of hierarchically related synsets to be arranged in polytopes. There's already some evidence that we can leverage Wordnet synsets to pretrain some models faster. Take a look at https://news.ycombinator.com/item?id=40160728 for example.

Thank you for sharing this on HN.

There are no comments on this comment yet.