News for hackers. Stuff that matters.

alexmarquardt

Re: on Wednesday January 14, 2026 @04:46PM (#46624077)
I kept running into the same demo problem: it’s hard to find a product catalog that behaves like a real e-commerce catalog (titles, working images, usable categories/attributes), is easy to ingest, and is safe/clear to reuse.

So I built two small OSS pipelines that convert open product sources into a clean, stable NDJSON schema you can bulk-index into Elasticsearch/OpenSearch. One outputs ~100K grocery products (Open Food Facts) and the other ~1M electronics-style products (Open Icecat), with strict “no image = no entry” quality gates and a shared schema contract.

Would love feedback on: • what fields you consider essential for a convincing search/relevance demo dataset • whether the schema choices (flat attrs for faceting + searchable description) match what you’ve seen work in practice

Re:Ranking by Profit and Popularity in Elasticsearch on Wednesday October 29, 2025 @01:08PM (#45749825)
In this article, you will learn how to combine BM25 relevance with real business metrics like profit margin and popularity using Elasticsearch’s function_score query. This step-by-step guide shows how to control scaling with logarithmic boosts and allows full explainability for each ranking calculation.
Looks awesome!
This article from Simon Späti shows how to do some scraping. https://airbyte.com/tutorials/configure-airbyte-with-python-...
Here is a related tweet that touches on exactly this issue: https://twitter.com/sethrosen/status/1252291581320757249?lan...
I like it! Great idea!
For a more detailed explanation of unstructured vs. structured data, see: https://airbyte.com/blog/analyze-unstructured-data

One Does Not Simply “Create a Visualization” from Unstructured Data

When I worked as a database consultant, I often saw conversations similar to the following at clients:Manager: Hey Data Analyst! We have a huge amount of data in our data lake, and I would like you to use it to provide us with some analysis and visualizations. Could you get that to me next week?Data Analyst: Not really. The data that we have in the data lake is unstructured, and I can't do much...

( Read More... | 5 comments | 7 points | on HN )
Re:Whats the difference between ETL and ELT on Tuesday November 08, 2022 @09:58AM (#33519667)
Yes, transforming the data before it hits the destination is often necessary, even in the ELT approach
Re:Whats the difference between ETL and ELT on Tuesday November 08, 2022 @09:57AM (#33519648)
ETL (Extract, Transform, and Load) and ELT (Extract, Load, and Transform) are two paradigms for moving data from one system to another. The main difference between them is that when an ETL approach is used, data is transformed before it is loaded into a destination system. On the other hand, in the case of ELT, any required transformations are done after the data has been written to the destination and are then done inside the destination – often by executing SQL commands.
Re:Using EtLT to improve GDPR compliance on Friday October 21, 2022 @10:26AM (#33288310)
Being GDPR compliant may require obfuscation or removal of some data before it gets loaded into a data warehouse. This can be achieved with an EtLT (Extract, “tweak”, Load, Transform) approach to data integration – in which a very light “tweak” (small “t”) transformation is done on the data before it reaches the destination. This is discussed in detail in the following article: https://airbyte.com/blog/etlt-gdpr-compliance
Thank you!

Older »