Build a information gathering tool in php or similar

$50-500 USD

Κλειστή

Αναρτήθηκε

περίπου 9 χρόνια πριν

$50-500 USD

Πληρωμή κατά την παράδοση

I have a MySQL database with a close to 5000 URL that I need tested, I need a PHP (or similar, please state in your bid what kind of development language you will be using) tool that can access all those urls and fetch a lot of information for me and save it to a new database. I have attatched the git repository for wappalyzer, can be found here [login to view URL] I have also attatched some tools I got developed here earlier, but the coder is not a member here anymore, not sure how those codes work, but it was supposed to check a lot of urls for a http response code.. If it is just as easy to build from skratch then please dont mind that file. :) So to sum it up, I wish a tool that I can operate from a webpage (hidden behind a login form of course) to feed it a list of url from a mysql database, and it then visits all those urls and saves ALL the information that wappalyzer finds on a page. In addition to the information wappalyzer finds I also wish it to save the http status code and the ip address of the page, and the following html tags: title tag, the keywords, the description tag. And if it is not hard to do, I would also wish to save the email address if it finds a @ on the front page. If it gets a time out or something, I wish that that entry gets put to the end of the queue tried again a second time, before it is put on a manual list for review so that we dont get a never ending loop. Please state in your bid how many days you will use to solve this, and what development languages you will be using. I want the code to be commented, so that I can learn from it (am starting to learn php and java and html myself, but it is a long way to go, so I want to have comments so I can understand it in the future.) I want this installed at my webpage, I have whm, cpanel and ftp access to it.

Ταυτότητα εργασίας: 7553273

Σχετικά με την εργασία

13 προτάσεις

Απομακρυσμένη Εργασία

Ενεργός/ή 9 χρόνια πριν

Ψάχνεις τρόπο για να κερδίσεις μερικά χρήματα;

Διεύθυνση Email

Πλεονεκτήματα πλειοδοσίας στο Freelancer

Καθόρισε τον προϋπολογισμό σου και το χρονοδιάγραμμα

Πληρώσου για τη δουλειά σου

Περίγραψε την πρόταση σου

Η εγγραφή και η πλειοδοσία σε εργασίες είναι δωρεάν

13 freelancers δίνουν μια μέση προσφορά $346 USD για αυτή τη δουλειά

@jfreak53

Good day and thank you for reading my proposal. I would be creating this system for you in PHP with CURL. However, as you mentioned, it would be 1 million times easier and faster to start from scratch than to use the already developed tools. This shouldn't be a massively hard system to make, but a bit time consuming simply depending on what "information" you need from each site. The description was a tad vague in that respect. The HTTP code and header info is simple enough. But the "extra" info you need from the page might take a bit of testing to get it perfect. But, all in all it's doable. My desire would be to make it somewhat dynamic, I don't know what the database structure looks like but I would like to make it as dynamic as possible for you. We can complete this in 7 days for you approximately, depending on changes. Microtronix has been in business now for 12 years and has over 200 customers in 12 countries of the world. We also have offices in the USA and Guatemala, C.A. We work primarily developing large Cloud based PHP systems to replace windows programs or older systems. We recently worked for Irvine Public Schools in California developing an online system to track all schools in the district. We also worked recently on a continuing assessment system for the board of medicine in Chicago. Both systems are 100% cloud based PHP systems. These are only a few of the projects we have worked on lately. If you have any questions please feel free to ask.

$444 USD σε 7 ημέρες

5,0

(55 αξιολογήσεις)

6,8

@programerpk

A proposal has not yet been provided

$333 USD σε 5 ημέρες

5,0

(4 αξιολογήσεις)

4,1

@Taiwaninja

Sounds like a very doable project, I'd be happy to do it in Python. Either python 2 or 3, Looking forward to talking with you on PM Thanks in advance.

$200 USD σε 3 ημέρες

4,7

(12 αξιολογήσεις)

3,8

@eitan1195

A proposal has not yet been provided

$444 USD σε 2 ημέρες

4,9

(11 αξιολογήσεις)

3,8

@wordpressbaker

A proposal has not yet been provided

$333 USD σε 15 ημέρες

4,9

(21 αξιολογήσεις)

3,4

@hemantmahalle

Hi sir have you mentioned

$311 USD σε 20 ημέρες

5,0

(2 αξιολογήσεις)

2,6

@RafaAguilar1987

Hello Buddy, I can do this, i plan to use Python, a very nice language to learn, i can do it on Java instead (if you want) but it will take some days longer, this is the general plan: 1. Install Wappalyzer and learn from it, 2 days. 2. Make a client/plugin for Wappalyzer yo read information that it parse. 3. Make the client connects to DB to save the data from Wappalyzer. 4. Make a simple web Interface, just to make a proof of concept 5. Implement it on your CPanel server 6. Implement extra features (like email scanning and so) 7. Get a new style for the web I think i can late 10-14 days, using a nice comment code, and we can extend this if you want extra lessons for every step, so this way you will be independent for futures changes ^^. Just PM me to discuss further details, Rafael Aguilar

$333 USD σε 9 ημέρες

5,0

(3 αξιολογήσεις)

2,4

@zkutch

Hello. More 20 years programming experience. I need more details to set real time and price - for example: do you accept perl? Regards. ---------------------------------------------------------------------------------------------------------------------------------------------------

$305 USD σε 3 ημέρες

3,3

(9 αξιολογήσεις)

3,8

@milanmartinjuk

Hi, Your project sounds very interesting and simple enough and I believe that I'll be able to perform the task perfectly. As for my experience, you can see for yourself that I've worked here mainly as a content and article writer; but, I've also done some VA work, which included scraping using a custom made software. Recently I've created a program for scraping Freelancer site in order to make bidding easier. You're welcome to check my reviews and should you have any questions, I'll be happy to answer. Kindest regards. Milan

$200 USD σε 3 ημέρες

4,8

(3 αξιολογήσεις)

2,8

@projexan

Please find below my short experience summary specific to this project. * Several years experience developing Text Mining and Information Extraction for web crawling, scraping, extraction and aggregation from unstructured big data such as web-pages and text corpus, assembling and populating them into databases, datastores and search-indexes(Lucene, Solr) for analysis, search, reporting and dashboard. * Have worked extensively on Text Mining techniques for automatically processing, classifying, predicting, clustering, categorization and citation and linkage analysis using referential connection and linkage structure (used markov model). * Extensive experience using Perl, PHP, C, Java, .NET with MySql, Oracle, MS-SQL Server * Information Extraction Tools : Weka, R, Excel, Perl-CPAN Packages for Extraction. Estimated Budget : ~ 340$ ( 7-12 days ) Price,milestones and timelines flexible and negotiable based on exact project specifications and details or for any additional project work. I would use PHP and MySql for this project and/or Perl for text extraction and analysis ( CPAN packages ) Are you running it ( your server ) on a Linux environment ? Would you need to run the extraction/population at a certain schedule .

$340 USD σε 7 ημέρες