Crawl your website using Nutch Crawler without Indexing the HTML content into SOLR

This article will help you in resolving following issue:

1) If you want to crawl the website using Nutch Crawler without indexing the HTML content into SOLR here are the changes that you need to perform the crawl script of nutch package.

You need to remove the following piece of code:

SOLRURL="$3"

if [ "$SOLRURL" = "" ]; then echo "Missing SOLRURL : crawl <seedDir> <crawlDir> <solrURL> <numberOfRounds>" exit -1;fi

echo "Indexing $SEGMENT on SOLR index -> $SOLRURL" $bin/nutch index -D solr.server.url=$SOLRURL $CRAWL_PATH/crawldb -linkdb $CRAWL_PATH/linkdb $CRAWL_PATH/segments/$SEGMENT if [ $? -ne 0 ]
then exit $?
fi

echo "Cleanup on SOLR index -> $SOLRURL" $bin/nutch clean -D solr.server.url=$SOLRURL $CRAWL_PATH/crawldb if [ $? -ne 0 ]
then exit $?
fi

Hope This Helps!!!

How to configure Nutch in Eclipse for SOLR

Checkout and Build Nutch:
1.    Get the latest source code from SVN using terminal.
For Nutch 1.x (ie.trunk) run this:
svn co https://svn.apache.org/repos/asf/nutch/trunk

2.    Add “http.agent.name” and “http.robots.agents” with appropiate values in “conf/nutch-site.xml”.
Here you have to rename the nutch-site.xml.template file to nutch-site.xml and make the changes accordingly.
See conf/nutch-default.xml for the description of these properties.

3.    Also, add “plugin.folders” and set it to {PATH_TO_NUTCH_CHECKOUT}/build/plugins. eg. If Nutch is present at "/home/Desktop/2.x",
set the property to:

<property>
   <name>plugin.folders</name>
   <value>/home/Desktop/2.x/build/plugins</value>
</property>

There is no /build/plugins folder currently present. But when you run the "ant eclipse" command you will get the "/build/plugins" in your {PATH_TO_NUTCH_CHECKOUT}.
Thats why it is written as set the absolute path as {PATH_TO_NUTCH_CHECKOUT}/build/plugins.
Do not give relative path here as it wont.

4.    Run this command:
ant eclipse


5.    Load project in Eclipse
5.1.    In Eclipse, click on “File” -> “Import...”

5.2.    Select “Existing Projects into Workspace”

5.3.    In the next window, set the root directory to the location where you took the checkout of nutch 2.x (or trunk). Click “Finish”.

5.4.    You will now see a new project named 2.x (or trunk) being added in the workspace. Wait for a moment until Eclipse refreshes its SVN cache and builds its workspace. You can see the status at the bottom right corner of Eclipse.

5.5.    In Package Explorer, right click on the project “2.x” (or trunk), select “Build Path” -> “Configure Build Path”

5.6.    In the “Order and Export” tab, scroll down and select “2.x/conf” (or trunk/conf). Click on “Top” button. Sadly, Eclipse will again build the workspace but this time it won’t take take much.


6.    Need to Download following jar files :
http://mvnrepository.com/artifact/org.elasticsearch/elasticsearch/0.90.1
Configure the above jar file in eclipse.

7.    One error you will get for “ElasticsearchException”. Change it to “ElasticSearchException” (S Capital)


8.    Now you are ready to run the nutch code in eclipse:
8.1.    Lets start off with the inject operation.

8.2.    Right click on the project in “Package Explorer” -> select “Run As” -> select “Run Configurations”.

8.3.    Create a new configuration. Name it as "inject".
For 1.x ie trunk : Set the main class as: org.apache.nutch.crawl.Injector
For 2.x : Set the main class as: org.apache.nutch.crawl.InjectorJob

8.4.    In the arguments tab, for program arguments, provide the path of the input directory which has seed urls.

8.5.    Set VM Arguments to “-Dhadoop.log.dir=logs -Dhadoop.log.file=hadoop.log”

8.6.    Click "Apply" and then click "Run".

8.7.    If everything was set perfectly, then you should see inject operation progressing on console.



Class in Nutch 1.x (i.e.trunk)
inject :- org.apache.nutch.crawl.Injector
generate :- org.apache.nutch.crawl.Generator
fetch :- org.apache.nutch.fetcher.Fetcher
parse :- org.apache.nutch.parse.ParseSegment
updatedb :- org.apache.nutch.crawl.CrawlDb


Class in Nutch 2.x
inject :- org.apache.nutch.crawl.InjectorJob
generate :- org.apache.nutch.crawl.GeneratorJob
fetch :- org.apache.nutch.fetcher.FetcherJob
parse :- org.apache.nutch.parse.ParserJob
updatedb :- org.apache.nutch.crawl.DbUpdaterJob


HOPE THIS HELPS!!!!

Nutch-SOLR Formed Based Authentication

Hi All,

I followed both the below links and I am able to configure Form Based Authentication in Nutch-SOLR setup.
http://lifelongprogrammer.blogspot.com/2014/02/part1-using-apache-http-client-to-do-http-post-form-authentication.html

and

https://issues.apache.org/jira/browse/NUTCH-827

These two links have complete and same code. 
If you follow the each steps correctly then you will be able to achieve Form Based Authentication in Nutch.

Let me know if you need any help in this. I will be Happy To Help You!!!

Regards,
Jayesh Bhoyar