I am looking for a web scraper to recursively follow all the 'Gametracker' links from a given date/page - like this ([login to view URL]) - and copy the html script (if it exists) for the shot chart graphic to a text or csv file.
The script is in <div class="court">, and contains points for the shot graphic graphic listed like this:
<div class="awayShot made" playerid="2018569" period="1" type="5" ypos="42" xpos="0" result="1" time="19:35" distance="0" style="left: 12.7127659574468px; top: 103.5px;"></div>
The data copied into the text, or csv file, should be, essentially, each of these categories, separated by a comma or tab, in one line:
awayShot made, 2018569, 1, 5, 42, 0, 1, 19:35, 0, 12.7127659574468, 103.5
or, in order, the class, playerid, period, type, ypos, xpos, result, time, distance, styleleft, styletop.
The text/csv files from each day in the schedule should be saved in one folder with the date (the 20141114 seen in the link provided), and the title of each file should be taken from the last portion of the address (CSN@SDGST - remove the @ if it can't be saved in a file name). The destination of this newly created folder should be able to be specified by me with a simple input or user interface.
A few of the recaps won't have the shot chart. The only links/games that should be considered are those with 'Gametracker' links.
I'd like to be able to enter a range of dates to pick up, so I can piecemeal it or run it once a week to get games I don't have. If this is too complex or could run into problems because some days will likely have no games, I'd like the program to start at the earliest date (November 14th, for this season), and continue through to the current date. The date/link to the next date is found near the top of the page at the link provided.
As a related, but one-time scraping addition to this project, I'd like to generate a list of all players from this site: [login to view URL], by going through the 'alphabetical player list' for each letter. The scraped info will include the player ID and player name, in one line, separated by a comma or tab. There can be a separate text file for each letter in the alphabet. The player ID's can be found in the links to the player pages, contained in the tables (<a href="/collegebasketball/players/playerpage/2019329/mike-aaman">Aaman, Mike</a>). At that link, it would be player ID = 2019329, name mike-aaman.
In the generated list, it would be simply 2019329, mike-aaman.
Let me know if there are any additional questions.
Hi sir,
I am scraping expert, I have did too many similar projects, please check my feedback then you will know.
Can you tell me more details? then I will provide demo data for you.
Thanks,
Kimi
HI...I have looked at the webpage in detail and have understood the page design and scripts.I have great experience if developing scripts for web scrapping.I can provide you a result in 12 hours guaranteed...If you are interested, I can provide you a quick demo in 2 hours.
Dear Sir,
I have read your Project Description & understand the initial high level details, we are currently working on two scrapping projects, as we have to scrap the data from few websites & then store in the MySQL database & create an Excel File.
So, let me assure you that our developers can work on this scrapping project but please drop me a message thru' Freelancer PMB to discuss further & finalizing the total cost, payment terms & conditions to move forward.
Just so you know, our current bid is just a placeholder to submit the proposal & start our communication.
Look forward to hearing from you.
Regards,
YK
LEADconcept
Sir,
I am well versed in this kind of jobs and can do your project as per requirement.
I have done lots of these kind of projects and will deliver flawless work
Looking for further discussions in this regard.
with thanks and regards
10+ years custom php data-mining/scraping
If you have/can install php on your machine I can deliver a simple script fast.
I have the first part of script working (scrape a game page)
The actual data is in a javascript array - turn off brower javascript and load the page
<div class='court'> will be empty
but javascript shotData: is where data comes from
Scraping the player page will be an easy task as well.
Yup - getting all players scraped was super easy.
Thanks,
Joe
I am Data Entry, MS Word and MS Excel Expert. i am very much professional in this work i am pretty sure that you cant find a best person for this job like me so i am ready to work on your project with low rate and high quality.
Hi, I did many similar projects in the past. I can write desktop application that do
this job. I've checked this website already, to confirm that job can be done.
I will use C# and .NET 4. There should be no problems with date range as input.
Getting player list is included in price.
Best regards
Mike
Hi,
I have developed software for over 20 years, and have lots of experience with most web-related technologies. Over the years, I have scraped hundreds of sites using many different technologies for various projects.
Your project seems quite straightforward. I have a few questions though:
- do you want a script that you will run on your own server, or do you prefer a turnkey solution where I host it for you?
- if the former, do you have any requirements (operating system, language, framework...)? My preference goes to perl + WWW::Curl + XML::LibXML, but I can do php (and probably others) if required.
- if the latter, I can provide a solution which either does the scraping on a regular basis, or with a web form to enter what you want scraped, and the results e-mailed to you.
For the latter option, there would be small recurring fee for hosting and maintenance, but I would include updates to the script whenever changes on the site break it (which from experience, is not a matter of "if" but "when").
Let me know if you can answer the above questions, or have any questions yourself.
Jacques.
I am interest for a web scraper to recursively follow all the 'Gametracker' links from a given date/page and copy the html script (if it exists) for the shot chart graphic to a text or csv file.
The reason I'm the best person to do this for you is because I've already figured out how the data works.
If you look at the source code, search for "shotData". You'll see a Javascript array with the string of numbers that more or less matches the data you're after, once it's combined with the various other bits and pieces. It's already there in native javascript, and is converted after that point to HTML, in the DIV you notice, by some other Javascript.
Even better, the CBS app they use to turn it into pretty pictures isn't minified (or obfuscated), which means that I can confirm the data transformation they use on the shotData array is what I'll be following.
Pick me, because I've got lots of experience with web-scraping, PHP and Javascript, and I can untangle the data you're looking for. And I'm a native English speaker. Find me on LinkedIn for more details.
Process:
- I'll scrape all players first, so we can match and confirm player IDs from the data extraction.
- I'll then run a preliminary spider, and you can confirm that we're accurately spidering all necessary games.
- I'll run the data extraction on those games.
- You're happy!
Assumptions forming this quote:
- That once the script is proven to work, it'll only need to be run one-time.
- That you can confirm for me that we've targeted all necessary games to scrape from.
- That data isn't dynamically loaded via Javascript, but via page load (as per the Javascript I showed you earlier
I will do it quickly and tailor-made. I have a few months of data scrapping/mining experience and just graduated from the 2nd best IT university in the world.
Hi,
Your detailed specification is excellent.
I have been able to confirm the details.
Is this a one off scrape? Are you using windows, if so which version? Do you have excel, if so which version? Would like the scraping script for future use?
Best regards,
Nathan
Hello Sir,
I can build a web scraper bot which will scraper the website as you said in the description.
Please read my feedbacks,
I have already worked for similar project, it's easy for me.
Best Regards,
