org.apache.nutch.net: public final class: URLNormalizers

org.apache.nutch.net
public final class: URLNormalizers [javadoc | source]

java.lang.Object
   org.apache.nutch.net.URLNormalizers

This class uses a "chained filter" pattern to run defined normalizers. Different lists of normalizers may be defined for different "scopes", or contexts where they are used (note however that they need to be activated first through plugin.include property).

There is one global scope defined by default, which consists of all active normalizers. The order in which these normalizers are executed may be defined in "urlnormalizer.order" property, which lists space-separated implementation classes (if this property is missing normalizers will be run in random order). If there are more normalizers activated than explicitly named on this list, the remaining ones will be run in random order after the ones specified on the list are executed.

You can define a set of contexts (or scopes) in which normalizers may be called. Each scope can have its own list of normalizers (defined in "urlnormalizer.scope." property) and its own order (defined in "urlnormalizer.order." property). If any of these properties are missing, default settings are used for the global scope.

In case no normalizers are required for any given scope, a org.apache.nutch.net.urlnormalizer.pass.PassURLNormalizer should be used.

Each normalizer may further select among many configurations, depending on the scope in which it is called, because the scope name is passed as a parameter to each normalizer. You can also use the same normalizer for many scopes.

Several scopes have been defined, and various Nutch tools will attempt using scope-specific normalizers first (and fall back to default config if scope-specific configuration is missing).

Normalizers may be run several times, to ensure that modifications introduced by normalizers at the end of the list can be further reduced by normalizers executed at the beginning. By default this loop is executed just once - if you want to ensure that all possible combinations have been applied you may want to run this loop up to the number of activated normalizers. This loop count can be configured through urlnormalizer.loop.count property. As soon as the url is unchanged the loop will stop and return the result.

author: Andrzej - Bialecki

Field Summary
public static final String	SCOPE_DEFAULT	Default scope. If no scope properties are defined then the configuration for this scope will be used.
public static final String	SCOPE_PARTITION	Scope used by org.apache.nutch.crawl.PartitionUrlByHost .
public static final String	SCOPE_GENERATE_HOST_COUNT	Scope used by org.apache.nutch.crawl.Generator .
public static final String	SCOPE_FETCHER	Scope used by org.apache.nutch.fetcher.Fetcher when processing redirect URLs.
public static final String	SCOPE_CRAWLDB	Scope used when updating the CrawlDb with new URLs.
public static final String	SCOPE_LINKDB	Scope used when updating the LinkDb with new URLs.
public static final String	SCOPE_INJECT	Scope used by org.apache.nutch.crawl.Injector .
public static final String	SCOPE_OUTLINK	Scope used when constructing new org.apache.nutch.parse.Outlink instances.
public static final Log	LOG

Constructor:
public URLNormalizers(Configuration conf, String scope) { this.conf = conf; this.extensionPoint = PluginRepository.get(conf).getExtensionPoint( URLNormalizer.X_POINT_ID); ObjectCache objectCache = ObjectCache.get(conf); if (this.extensionPoint == null) { throw new RuntimeException("x point " + URLNormalizer.X_POINT_ID + " not found."); } normalizers = (URLNormalizer[])objectCache.getObject(URLNormalizer.X_POINT_ID + "_" + scope); if (normalizers == null) { normalizers = getURLNormalizers(scope); } if (normalizers == EMPTY_NORMALIZERS) { normalizers = (URLNormalizer[])objectCache.getObject(URLNormalizer.X_POINT_ID + "_" + SCOPE_DEFAULT); if (normalizers == null) { normalizers = getURLNormalizers(SCOPE_DEFAULT); } } loopCount = conf.getInt("urlnormalizer.loop.count", 1); }

Constructor:

 public URLNormalizers(Configuration conf,
    String scope) {
       
    this.conf = conf;
    this.extensionPoint = PluginRepository.get(conf).getExtensionPoint(
            URLNormalizer.X_POINT_ID);
    ObjectCache objectCache = ObjectCache.get(conf);
    if (this.extensionPoint == null) {
      throw new RuntimeException("x point " + URLNormalizer.X_POINT_ID
              + " not found.");
    }
    normalizers = (URLNormalizer[])objectCache.getObject(URLNormalizer.X_POINT_ID + "_" + scope);
    if (normalizers == null) {
      normalizers = getURLNormalizers(scope);
    }
    if (normalizers == EMPTY_NORMALIZERS) {
      normalizers = (URLNormalizer[])objectCache.getObject(URLNormalizer.X_POINT_ID + "_" + SCOPE_DEFAULT);
      if (normalizers == null) {
        normalizers = getURLNormalizers(SCOPE_DEFAULT);
      }
    }
    loopCount = conf.getInt("urlnormalizer.loop.count", 1);
}

Method from org.apache.nutch.net.URLNormalizers Summary:
getURLNormalizers, normalize

Methods from java.lang.Object:
equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Method from org.apache.nutch.net.URLNormalizers Detail:
URLNormalizer[] getURLNormalizers(String scope) { List< Extension > extensions = getExtensions(scope); ObjectCache objectCache = ObjectCache.get(conf); if (extensions == EMPTY_EXTENSION_LIST) { return EMPTY_NORMALIZERS; } List< URLNormalizer > normalizers = new Vector< URLNormalizer >(extensions.size()); Iterator< Extension > it = extensions.iterator(); while (it.hasNext()) { Extension ext = it.next(); URLNormalizer normalizer = null; try { // check to see if we've cached this URLNormalizer instance yet normalizer = (URLNormalizer) objectCache.getObject(ext.getId()); if (normalizer == null) { // go ahead and instantiate it and then cache it normalizer = (URLNormalizer) ext.getExtensionInstance(); objectCache.setObject(ext.getId(), normalizer); } normalizers.add(normalizer); } catch (PluginRuntimeException e) { e.printStackTrace(); LOG.warn("URLNormalizers:PluginRuntimeException when " + "initializing url normalizer plugin " + ext.getDescriptor().getPluginId() + " instance in getURLNormalizers " + "function: attempting to continue instantiating plugins"); } } return normalizers.toArray(new URLNormalizer[normalizers .size()]); } Function returns an array of URLNormalizer s for a given scope, with a specified order.
public String normalize(String urlString, String scope) throws MalformedURLException { // optionally loop several times, and break if no further changes String initialString = urlString; for (int k = 0; k < loopCount; k++) { for (int i = 0; i < this.normalizers.length; i++) { if (urlString == null) return null; urlString = this.normalizers[i].normalize(urlString, scope); } if (initialString.equals(urlString)) break; initialString = urlString; } return urlString; } Normalize

Method from org.apache.nutch.net.URLNormalizers Detail:

 URLNormalizer[] getURLNormalizers(String scope) {
    List< Extension > extensions = getExtensions(scope);
    ObjectCache objectCache = ObjectCache.get(conf);
    if (extensions == EMPTY_EXTENSION_LIST) {
      return EMPTY_NORMALIZERS;
    }
    List< URLNormalizer > normalizers = new Vector< URLNormalizer >(extensions.size());
    Iterator< Extension > it = extensions.iterator();
    while (it.hasNext()) {
      Extension ext = it.next();
      URLNormalizer normalizer = null;
      try {
        // check to see if we've cached this URLNormalizer instance yet
        normalizer = (URLNormalizer) objectCache.getObject(ext.getId());
        if (normalizer == null) {
          // go ahead and instantiate it and then cache it
          normalizer = (URLNormalizer) ext.getExtensionInstance();
          objectCache.setObject(ext.getId(), normalizer);
        }
        normalizers.add(normalizer);
      } catch (PluginRuntimeException e) {
        e.printStackTrace();
        LOG.warn("URLNormalizers:PluginRuntimeException when "
                + "initializing url normalizer plugin "
                + ext.getDescriptor().getPluginId()
                + " instance in getURLNormalizers "
                + "function: attempting to continue instantiating plugins");
      }
    }
    return normalizers.toArray(new URLNormalizer[normalizers
            .size()]);
}

URLNormalizer

 public String normalize(String urlString,
    String scope) throws MalformedURLException {
    // optionally loop several times, and break if no further changes
    String initialString = urlString;
    for (int k = 0; k <  loopCount; k++) {
      for (int i = 0; i <  this.normalizers.length; i++) {
        if (urlString == null)
          return null;
        urlString = this.normalizers[i].normalize(urlString, scope);
      }
      if (initialString.equals(urlString)) break;
      initialString = urlString;
    }
    return urlString;
}

Normalize