Advanced parameters

To access these options use the [Advanced] button of Scan settings of the [Settings] tabsheet on the right frame :

Use the Advanced scan settings parameters to set:

• the number of download retries when a connect or GET error occurs,
• the time (in seconds) left to connect to an URL before aborting.

Use the Maximum concurrent downloads parameters to set the number of HTML pages downloads running simultaneously (between 4 and 20, according to your ISP speed and your processor is a good range). Please note that a high concurrent downloads value may cause server bottleneck problems and/or computer resources shortage.

Some computers need a Proxy server to access the Internet. Please contact your network administrator to set theses options.

If necessary, use the [Proxy] tabsheet to set the proxy options:

• Direct connection to Internet: all the Proxy parameters are ignored.
• Use proxy parameters: parameters are used to authenticate to the proxy server.
• Use Internet explorer default: the Windows control panel parameters are used.

If the Use proxy parameters is checked, you must set these options:

• Username: username to authenticate to the proxy server.
• Password: password to authenticate to the proxy server.
• IP Address: IP address of the proxy server.
• Port: Port of the proxy server.

By using the [Policy] tabsheet parameters, you can decide to respect or not the "Robot exclusion standard":

• Obey to the "Robot exclusion standard"
• Crawl every Web pages (see below).

Important:

1 - The "Robot exclusion standard" is set by webmasters to limit the directories and files the search engine is allowed to "harvest" (copy and index). See http://www.robotstxt.org, for more information.

2 - You choose to crawl every Web pages under your own responsability.

Read the next part:
Start the Web Mining: Download sites

---