Website Footprinting - Website Scraping, Website Mirroring and Website Monitoring

While Footprinting refers to gathering the needed information and getting knowledge of how things work, website footprinting refers to extracting data from a website and knowing how the site works. Basically, working of a website is known on the basis of the javascript files or the js code which executes on an activity. There are many other things which determine the methodology of working od a site and this may be helpful to the attacker. So, let us explore more on the terms and methods.

Website Footprinting is the first step towards hacking a website. To hack a site, we need information such as:

• How the site works?
• How frequent are new article posted on site?
• Is the admin of website active/inactive?
• What type of data is available on the site?
• And much more...

These can be achieved by footprinting a website. Following all the steps in website footprinting leads us to get confidential information from the site and know how the site works in reality. Let us explore more about this.

Website Scraping

The best way to extract information from a webpage is to open the page in browser and then examine it's source code and cookies used by the site. But examining the source code doesn't provide all the needed information and looking at cookies manually is tiresome. So, the concept of extracting data from a website came into existence.

Web Scraping (also termed Screen Scraping, Web Data Extraction, Web Harvesting etc.) is a technique employed to extract large amounts of data from websites whereby the data is extracted and saved to a local file in your computer or to a database in table (spreadsheet) format.

Data displayed by most websites can only be viewed using a web browser. They do not offer the functionality to save a copy of this data for personal use. The only option then is to manually copy and paste the data - a very tedious job which can take many hours or sometimes days to complete. Web Scraping is the technique of automating this process, so that instead of manually copying the data from websites, the Web Scraping software will perform the same task within a fraction of the time.

A web scraping software will automatically load and extract data from multiple pages of websites based on your requirement. It is either custom built for a specific website or is one which can be configured to work with any website. With the click of a button you can easily save the data available in the website to a file in your computer.

One of the useful Web Scraping Software is listed below: 

• Web Data Extractor - http://www.webextractor.com/ 

You can also use any other software/plugin/script for the same job. These are easily available on internet. The main concern is that, the tool must be easy to use.

Website Mirroring

Mirroring refers to downloading the entire website offline on your harddisk for browsing it offline.

Mirroring an entire website onto local machine enables an attacker to browse website offline; it can also assist in finding directory structure and other valuable information from mirrored copy without multiple requests to web server. Sending multiple requests to a web server may be dangerous as the admin when looking to log files, can identify that you were trying to collect sensitive information from the site and it can help the admin to traceback you.

Some well-known web mirroring tools are:

• HTTrack - http://www.httrack.com/
• Surf Offline - http://www.surfoffline.com/ 

There are many other tools which are easily available on Google but these are the best.

Website Monitoring

Monitoring a website refers to getting information such as:

• How frequently the admin posts on the site?
• Which posts are deleted?
• When was an article posted?
• Get alerted when a new article is posted on the site.

There are two methods used for different purposes. The first three purpose listed above are satisfied by Internet Archives. You can refer to its complete guide in this article.

The second method is easy to use and satisfies the fourth (last) purpose of website monitoring. It works the same way when you subscribe to a website. When a new post is posted, you are informed about it through mail service. But the major difference is that, in subscription, the alert mail is controlled by the admin i.e. we are alerted of the new article when the admin wants; while in monitoring, we are the controller. That is, we check regularly if a site has posted a new article or has made any changes.

But doing this task manually is tiresome as said before. So automated tools and services are used with a view to reduce the work. Some of the tools used for this purpose are:

• Website Watcher - http://aignes.com/
• Change Detection - https://www.changedetection.com/
• Follow that Page - https://followthatpage.com
• Watch that Page - http://watchthatpage.com/

The above are some of the best services while you can search google for more such services if you want.