finish python webscraper

Introduction We recently had a scraper build to scrape rental houses from a housing website. It saves these rentals into a database and exposes them through a http json api. The scraper was finished but needs a couple of finishing touches to make it usable for production. The code The project uses two main files: [login to view URL] - contains the parser [login to view URL] - contains the http server DB credentials are in a seperate file. You can use every postgresql db you want in development. Also I already set up a pm2 ecosystem file to run both scripts. [login to view URL] The parser scrapes all the overview pages and puts the rental links found into the database. Then it scrapes the details by loading each detail page found from the overview pages. It uses phantomJS to load the page and a special api to solve the occasional QR code. Requirements ● We believe the scraping process can go a lot faster if requests are distributed better/smarter ○ Target duration: 2-6 hours ● The parser needs to keep availability status, so it needs to check if rentals are still available (the details page gives a 301/302 to the overview page if the rental is expired) [login to view URL] This script exposes the scraped data in JSON using a simple http server. We agreed to use three requests (so it connects easy to other projects we are running). They do not work as expected yet. Requirements 1. GET /valid?since={unixDate} a. Description i. Returns the id of all valid rentals in the database, added after ‘since’ a certain date. b. Parameters i. since (unixdate) 2. GET /expired?since={unixDate} a. Description i. Returns the id of all expired rentals in the database, expired after ‘since’ a certain date. b. Parameters i. since (unixdate) 3. GET /details?id={id} a. Description i. Returns the details of a certain rental in the database, specified by the id parameter. ii. Details should include current status 1. “Valid” or 2. “Expired” iii. Pictures[] should only include selected rental pictures b. Parameters i. id