mit license: https://github.com/john-friedman/doc2structdoc2struct is an algorithm that can parse documents fast on one thread. For example it can parse every every 2025 annual report 19GB in 40s.It preserves nesting, does mild table cleaning, among other things.It succeeds my earlier document parser https://github.com/john-friedman/doc2dict
The U.S. Securities and Exchange Commission requires companies and individuals to submit data in SEC specific formats. Usually this means taking a columnar dataset and converting it to a specific XML schema.In practice, this usually means paying a company for proprietary filing software that is annoying to use, and is not modifiable.Here it an open source solution:...
It turns out that you can get SEC filings seconds to minutes faster by using URL prediction. This is because the SEC exposes the filing to the internet, before updating feeds like the RSS and PDS.How it works: 1. The SEC accepts a filing, this is recorded as e.g. <ACCEPTANCE-DATETIME>20220204201127 2. The SEC then generates an index page for the filing, with filing metadata. This is...
re: metadata table - it's constructed from the SEC's submissions.zip, which they update daily. What my script does is download the zip, decompress just the bytes where the information (ticker, sic code, etc) is stored, then convert into a csv.
And yep! Agree with most of this. Currently, I'd say my data is in the stage where it's useful for startups / phd research and some hedge funds / quant stuff (at least that's who is using it so far!)
I've seen the trillion dollar trades, and they're hilarious! You see it every so often in Form 3,4,5 disclosures.
re: LLMs, this is something I'm planning to move into in a month or two. I'm mostly planning to use older NLP methods which are cheaper and faster, while using LLMs for specific stuff like structured output. e.g. wrds boardex data can be constructed from 8-k item 5.02s.
I think the biggest difficulty wrt to data is just the raw data ingest is annoying AF. My approach has been to make each step easy -> use it to build the next step.
SGML parsing is fun! - I've opensourced a sgml parser here https://github.com/john-friedman/secsgml
Securities master to link against - Interesting. Here's a pipeline off the top of my head 1. Get CUSIP, nameOfIssuer, titleOfClass using the Institutional Holdings database 2. Use the company metadata crosswalk to link CUSIP + titleOfClass to nameOfIssuer to get cik https://github.com/john-friedman/datamule-data/blob/master/d... (recompiled daily using GH actions) 3. Get e.g. us-gaap:EarningsPerShareBasic from the XBRL database. Link using cik. Types of stock might be a member - so e.g. Class A, Class B? Not sure there.
For form 4, not sure what you mean by event study. Would love to know!
Websocket:
1. Two AWS ec2 t4g.nano instances polling the SEC's RSS and EFTS endpoints. (RSS is faster, EFTS is complete). 2. When new submissions are detected, they are sent to the Websocket (t4g.micro websocket, using Go for greater concurrency). 3. Websocket sends signal to consumers.
Archive:
1. One t4g.micro instance. Receives notifications from websocket, then gets submissions SGML from the SEC. 2. If submission is over size threshold, compresses with zstandard. 3. Uploads submissions to Cloudflare R2 bucket. (Zero egress fee, just class A / B operations). 4. Cloudflare R2 bucket is proxied behind my domain, with caching.
RDS
1. ECS Fargate instances set to run daily at 9 AM UTC. 2. Downloads data from archive, then parses them, and uploads them into AWS dbt.medium MySQL RDS. 3. Also handles reconciliation for the archive in case any filings were missed.
Hey everyone,I'm the developer of an open-source (MIT License) python package to convert SEC submissions into useful data. I've recently put a bunch of stuff in the cloud for a nominal convenience fee.Cloud:1. SEC Websocket - notifies you of new submissions as they come out. (Free)2. SEC Archive - download SEC submissions without rate limits. ($1/100,000 downloads)3. MySQL RDS ($1/million rows...
doc2dict is a python package that converts html and pdf documents into dictionaries preserving hierarchy. It also supports table extraction for html files. https://github.com/john-friedman/doc2dictSpeed:* html - 500 pages per second single threaded.* pdf - 200 pages per second, pdf must have an underlying text structure. Multithreading is not possible due to the limitations of PDFium.Here's an...
Thank you for the gentle advice. Appreciate it.
I was curious if I could use insider trading patterns to predict whether the sentiment of 8-K filings were positive or negative.https://github.com/john-friedman/datamule-python/blob/experimental/examples/predicting-good-news-from-insider-trading/exploration.ipynbDoesn't look like it - not enough data. However, there is an interesting result. Trading behavior after a positive 8-K is very...
https://github.com/john-friedman/datamule-python/blob/main/examples/discord_bot.ipynb
I built a Discord bot that monitors SEC EDGAR in real-time and sends alerts for new filings. You can configure it to track specific form types (e.g. 3/4/5) and companies by CIK, with multi-channel support for different filtering rules.
A basic implementation of a tool-using SEC chatbot without reasoning ability.Features: * Browse and download Company XBRL facts * View specific sections in filings, e.g. "Tesla's Management Discussion & Analysis"Future: * Reasoning ability + toggle * Integration with simplified MD&A database allowing for comparing hundreds of companies at once within one context window * Integration...
https://github.com/john-friedman/datamule-python
Features: * parse textual filings into simplified html, interactive html, or structured json. * download sec filings quickly and easilyDatasets: * 2024 Management Discussion and Analysis * 2024 10Ks in json formatDatasets can be downloaded using the package, or here https://www.dropbox.com/scl/fo/byxiish8jmdtj4zitxfjn/AAaiwwu...All APIs are free and do not require auth, so you can use them...
I also setup an API endpoint. The number of responses per request is limited to 10,000. Endpoint: api.datamule.xyz/submissions. Example Query: https://api.datamule.xyz/submissions?cik=1318605&form=10-K&d...
More on the API here: https://medium.com/@jgfriedman99/sec-router-05a2308b24ce
Hope someone finds this useful.