News for hackers. Stuff that matters.

A comment on Web Scraping 101 with Python.

Re:Web Scraping 101 with Python by philshem on Wednesday February 10, 2021 @01:07PM (#26092674)
Before jumping into frameworks, if your data is lucky enough to be stored in an html table:

    import pandas as pd
    dfs = pd.read_html(url)
Where ‘dfs’ is an array of dataframes - one item for each html table on the page.

https://pandas.pydata.org/pandas-docs/stable/reference/api/p...

[ Parent | on HN ]
Re:Web Scraping 101 with Python by kortex on Wednesday February 10, 2021 @07:27PM (#26096729)
Sometimes it's also helpful to use beautiful soup to isolate the elements you want, feed the text of the elements into StringIO and give that to read_html.
[ Parent | on HN ]
Re:Web Scraping 101 with Python by philshem on Thursday February 11, 2021 @01:29AM (#26098853)
Yes, this is a good idea for more complicated cases.
[ Parent | on HN ]
Re:Web Scraping 101 with Python by mritchie712 on Friday February 12, 2021 @08:55AM (#26113634)
We made a chrome extension that queries any html table in any open tab with SQL:

https://chrome.google.com/webstore/detail/sqanything/naejbcf...

You can export the results to Google Sheets too. One advantage of the extension is it works with JS rendered tables.

[ Parent | on HN ]
Re:Web Scraping 101 with Python by Yenrabbit on Thursday February 11, 2021 @01:17AM (#26098786)
Handy! I'm also a big fan of pd.read_clipboard() for specific selections.
[ Parent | on HN ]
Re:Web Scraping 101 with Python by antisthenes on Wednesday February 10, 2021 @08:00PM (#26096947)
Holy crap, is there anything pandas can't do?
[ Parent | on HN ]
Re:Web Scraping 101 with Python by Frost1x on Wednesday February 10, 2021 @11:10PM (#26098171)
Ingest bamboo? Sorry, couldn't resist.
[ Parent | on HN ]
Re:Web Scraping 101 with Python by JosephRedfern on Wednesday February 10, 2021 @03:50PM (#26094578)
Woah. I’ve used pandas a fair amount and had no idea about this. Thank you!
[ Parent | on HN ]
Re:Web Scraping 101 with Python by andreilys on Wednesday February 10, 2021 @01:41PM (#26093093)
+1 this has saved me countless of hours
[ Parent | on HN ]
Re:Web Scraping 101 with Python by mikesholiu on Wednesday February 10, 2021 @01:57PM (#26093264)
what does this do
[ Parent | on HN ]
Re:Web Scraping 101 with Python by SirSourdough on Wednesday February 10, 2021 @02:18PM (#26093461)
It reads HTML and returns the tables contained in the HTML as pandas dataframes. It’s a simple way to scrape tabular data from websites.
[ Parent | on HN ]