org.apache.nutch.protocol.httpclient: public class: Http

org.apache.nutch.protocol.httpclient
public class: Http [javadoc | source]

java.lang.Object
   org.apache.nutch.protocol.http.api.HttpBase
      org.apache.nutch.protocol.httpclient.Http

All Implemented Interfaces:
Protocol

This class is a protocol plugin that configures an HTTP client for Basic, Digest and NTLM authentication schemes for web server as well as proxy server. It takes care of HTTPS protocol as well as cookies in a single fetch session.

author: Susam - Pal

Field Summary
public static final Log	LOG
int	maxThreadsTotal

Fields inherited from org.apache.nutch.protocol.http.api.HttpBase:
BUFFER_SIZE, proxyHost, proxyPort, useProxy, timeout, maxContent, maxDelays, maxThreadsPerHost, serverDelay, userAgent, useHttp11, maxCrawlDelay, checkBlocking, checkRobots

Constructor:
public Http() { super(LOG); } Constructs this plugin.

Constructor:

 public Http() {
    super(LOG);
}

Constructs this plugin.

Method from org.apache.nutch.protocol.httpclient.Http Summary:
getClient, getResponse, main, setConf

Methods from org.apache.nutch.protocol.http.api.HttpBase:
getConf, getMaxContent, getMaxDelays, getMaxThreadsPerHost, getProtocolOutput, getProxyHost, getProxyPort, getResponse, getRobotRules, getServerDelay, getTimeout, getUseHttp11, getUserAgent, logConf, main, processDeflateEncoded, processGzipEncoded, setConf, useProxy

Methods from java.lang.Object:
equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Method from org.apache.nutch.protocol.httpclient.Http Detail:
static synchronized HttpClient getClient() { return client; } Returns the configured HTTP client.
protected Response getResponse(URL url, CrawlDatum datum, boolean redirect) throws IOException, ProtocolException { resolveCredentials(url); return new HttpResponse(this, url, datum, redirect); } Fetches the `url` with a configured HTTP client and gets the response.
public static void main(String[] args) throws Exception { Http http = new Http(); http.setConf(NutchConfiguration.create()); main(http, args); } Main method.
public void setConf(Configuration conf) { super.setConf(conf); this.conf = conf; this.maxThreadsTotal = conf.getInt("fetcher.threads.fetch", 10); this.proxyUsername = conf.get("http.proxy.username", ""); this.proxyPassword = conf.get("http.proxy.password", ""); this.proxyRealm = conf.get("http.proxy.realm", ""); agentHost = conf.get("http.agent.host", ""); authFile = conf.get("http.auth.file", ""); configureClient(); try { setCredentials(); } catch (Exception ex) { if (LOG.isFatalEnabled()) { LOG.fatal("Could not read " + authFile + " : " + ex.getMessage()); ex.printStackTrace(LogUtil.getErrorStream(LOG)); } } } Reads the configuration from the Nutch configuration files and sets the configuration.

Method from org.apache.nutch.protocol.httpclient.Http Detail:

 static synchronized HttpClient getClient() {
             
    return client;
}

Returns the configured HTTP client.

 protected Response getResponse(URL url,
    CrawlDatum datum,
    boolean redirect) throws IOException, ProtocolException {
    resolveCredentials(url);
    return new HttpResponse(this, url, datum, redirect);
}

url

 public static  void main(String[] args) throws Exception {
    Http http = new Http();
    http.setConf(NutchConfiguration.create());
    main(http, args);
}

Main method.

 public  void setConf(Configuration conf) {
    super.setConf(conf);
    this.conf = conf;
    this.maxThreadsTotal = conf.getInt("fetcher.threads.fetch", 10);
    this.proxyUsername = conf.get("http.proxy.username", "");
    this.proxyPassword = conf.get("http.proxy.password", "");
    this.proxyRealm = conf.get("http.proxy.realm", "");
    agentHost = conf.get("http.agent.host", "");
    authFile = conf.get("http.auth.file", "");
    configureClient();
    try {
      setCredentials();
    } catch (Exception ex) {
      if (LOG.isFatalEnabled()) {
        LOG.fatal("Could not read " + authFile + " : " + ex.getMessage());
        ex.printStackTrace(LogUtil.getErrorStream(LOG));
      }
    }
}

Reads the configuration from the Nutch configuration files and sets the configuration.