Arnau P.

As a web scraping developer, I feel comfortable using Scrapy (Python framework) in order to scrape any kind of websites. There is no problem with dynamic content (Javascript), as it can be scraping using Selenium and/or Splash. The scraped data can be saved in different formats like ".xls, .csv" or even in SQL databases. This data can also be post-processed, interpreted and displayed using interesting graphs. It is worth noting that I feel comfortable using other kinds of programming languages (in addition to Python) like C++, C# and AWK/SED. I also use Bash Scripts to automate any kind of repetitive order (they can be quite complex). Usually, I develop software in a Linux environment, but Windows and Mac can also be used.