Thursday, May 18, 2017

Password Protected Solr Admin Page







As we all know Solr Admin Page is not password protected and anyone can get into Solr Admin Page. However this article will help you in enabling Password Protected Solr Admin Page.

Yes, Solr Admin Page comes without any password protection. However good thing is that we deploy Solr either on Tomcat or on Jetty or on JBoss. So using the feature of Web Container we can restrict the Solr Admin Page as well as Solr Indexing API call under username and password.


Step 1:
We need to add following piece of code in \solr-6.5.1\server\etc\jetty.xml

<Call name="addBean"> 
     <Arg> 
          <New class="org.eclipse.jetty.security.HashLoginService"> 
           <Set name="name">Secure Realm</Set> 
           <Set name="config">
               <SystemProperty name="jetty.home" default="."/> \\etc\\realm.properties
          </Set>
           <Set name="refreshInterval">0</Set> 
          </New> 
      </Arg> 
</Call>



Step 2:
We need to add following piece of code in \solr-6.5.1\server\solr-webapp\webapp\WEB-INF\web.xml


<security-constraint>
<auth-constraint><login-config>


<web-resource-collection>
<web-resource-name>Solr Search Engine</web-resource-name>
<url-pattern>/*</url-pattern>
</web-resource-collection>

<role-name>admin</role-name>
</auth-constraint>
</security-constraint>

<auth-method>BASIC</auth-method>
<realm-name>Secure Realm</realm-name>

</login-config>



Step 3:
Create the MD5 password using below step. For Username:admin and Password: solr123

\solr-6.5.1\bin>java -cp ..\server\lib\jetty-util-9.3.14.v20161028.jar org.eclipse.jetty.util.security.Password admin solr123

Output
2017-05-17 17:24:08.064:INFO::main: Logging initialized @958ms
solr123
OBF:1m0v1l181k8q1y7z1k5g1kxu1lxb
MD5:77cb23aec2e0ff10c2952948346d9817
CRYPT:adVVnmxPfgwZ6



Step 4:

Create a realm.proprerties in \solr-6.5.1\server\etc\ directory and add following line of code into it.

admin: MD5:77cb23aec2e0ff10c2952948346d9817, admin

i.e. <username>: MD5:<password>, <role>











Monday, May 1, 2017

Training Document - Introduction to Solr



Introduction to Solr


A brief introduction to Solr for the resources who wants to get trained on Solr. 


Table of Content
1. Introduction to Solr 
2. Solr Terminologies 
3.Installation and Configuration 
4. Configuration files schema.xml and solrconfig.xml 
5. Features of SOLR 
    a. Hit Highlighting 
    b. Auto Complete / Suggester 
    c. Stop words 
    d. Synonyms 
    e. SpellCheck 
    f. Geo Spatial Search 
    g. Result Grouping 
    h. Query Syntax 
    i. Query Boosting 
    j. Content Spotlighting / Merchandising / Banner / Elevate 
    k. Block Record / Remove URL Feature 
6. Indexing the Data 
7. Search Queries 
8. DataImportHandler - DIH 
9. Plugins to index various types of Data (XML, CSV, DB, Filesystem) 
10. Solr Client APIs 
11. Overview of SOLRJ API 
12. Running Solr on Tomcat 
13. Enabling SSL on Solr 
14. Zookeeper Configuration 
15. Solr Cloud Deployment 
16. Production Indexing Architecture 
17. Production Serving Architecture 
18. Solr Upgradation 
19. References

Sunday, September 18, 2016

Oracle SQL Command - To Check Total Free Space In Database



Being non-DBA person, I always face this issue of checking the free space available in my production Database. If you are using ASM and wanted to check the same please us below command to get the required output:


SELECT df.tablespace_name TABLESPACE, df.total_space TOTAL_SPACE,
fs.free_space FREE_SPACE, df.total_space_mb TOTAL_SPACE_MB,
(df.total_space_mb - fs.free_space_mb) USED_SPACE_MB,
fs.free_space_mb FREE_SPACE_MB,
ROUND(100 * (fs.free_space / df.total_space),2) PCT_FREE
FROM (SELECT tablespace_name, SUM(bytes) TOTAL_SPACE,
      ROUND(SUM(bytes) / 1048576) TOTAL_SPACE_MB
      FROM dba_data_files
      GROUP BY tablespace_name) df,
     (SELECT tablespace_name, SUM(bytes) FREE_SPACE,
       ROUND(SUM(bytes) / 1048576) FREE_SPACE_MB
       FROM dba_free_space
       GROUP BY tablespace_name) fs
WHERE df.tablespace_name = fs.tablespace_name(+)
ORDER BY fs.tablespace_name;














Saturday, May 21, 2016

[ Solr ] Error Document is missing mandatory uniqueKey field id


In the schema.xml file, It is mentioned id as required field = true.
Also the document that we try to index in SOLR do not contain this id field and hence SOLR throws this error.


Solution

  1. Either add id to all your documents
OR
  1. Remove required = true form schema file for id field.

Hope This Helps!!!!

Sunday, May 15, 2016

[Google Search Appliance ] Migrate Autosuggestions from one production GSA Box to Another Production GSA Box


We all know that Google Search Appliance learns all the search queries that are getting executed and then it generates the Autosuggestion list.

However If you want to migrate all autosuggestions from one production box to another then there is no feature currently available to achieve this task. 

There is a feature request # 4411407 for allowing import query suggestion into GSA box.

However I wanted to achieve this by some custom means So I followed the below steps and it worked perfectly fine.

1) Export the suggestion in text file from first GSA Box.
2) Write a script to read each line from that files and execute the search query against word /phrase to new GSA Box.
3) This will create a new autosuggestion list in new GSA Box.

Hope this Helps!!!

Saturday, February 6, 2016

[Google Search Appliance] Semantic Search Support

Below features can be configured in GSA to enable Semantic Search
  1. Query Expansion
  2. Related Search
  3. KeyMatch Search
All the above features works on a predefined list/set of words. In order to implement these features we have to configure the list into GSA.

1)   Query Expansion
Query expansion enables GSA to automatically add extra terms to a user's search query, in order to return additional relevant results. 
When query expansion is enabled, the appliance can expand two types of terms:

Stem Words: Words that share the same word stem as the word given by the user. For example, if a user searches for "engineer," the appliance could add "engineers" to the query. (Fast, Faster, Fastest)

Terms of one or more space-separated words that are synonymous or closely related to the words given by the Business user.
For example, if a user searches for "FAQ," the appliance could add "frequently asked questions" to the query
For example if recruiter searches for “P1” the appliance could add “priority 1” to the query.

Query Expansion feature can be implemented by configuring the Synonym file into GSA.

Rule
Meaning
Comments
Java = Spring
Bi-Directional
Synonym
If user searches for Java, then user will get results for Java as well as Spring.
If user searches for Spring, then user will get results for Java as well as Spring.
Java > Spring, Java > Hibernate
One-Directional
Synonym
If user searches for Java, then user will get results for Java as well as for Spring and Hibernate.
However If user searches for Spring, then user will only get result for Spring and same applies for Hibernate
{Java, Spring, Hibernate]
Bi-Directional
Synonym
If user searches either of the words (Java, Spring, Hibernate), then user will get results for all the search terms.

Pros: If proper and accurate Synonym file is configured in GSA then it will enhanced the user search experience.
Cons: Improper Synonym file can result into false positive search results being returned.


2)  Related Search
Related Search feature can be used to associate alternative words or phrases with specified search terms. When a user enters the specified search term, the alternative word or phrase appears as a suggestion. The user can click on the suggested alternative to start another search.

Search Term
Related Query
P1
Priority 1
It works as the replacement of original “Search Term” with “Related Query”


3)  KeyMatch Search

For a given search term or phrase search; GSA can return a specific set of results to user.


Search Term
Type
Url of Match
Title of Match
GSA
Keyword Match
http://tinyurl.com
Search Architect for Google Search and Match Project

Wednesday, July 29, 2015

ELASTICSEARCH JDBC CONNECTOR

ELASTICSEARCH JDBC CONNECTOR


1) Check the dependency and download appropriate JDBC connector version code as per your elasticsearch version.

2) Download the code from below location:

3) Compile the code using “mvn clean install”. It will create a jar file in target folder.

4) Copy the jar file into elasticsearch lib folder.

5) Download the JDBC importer distribution from below location

6) unpack
unzip elasticsearch-jdbc-<version>-dist.zip

7) This will be our $JDBC_IMPORTER_HOME

8) Set the below path 
#Elastic Search
$> ES_HOME=/Users/jayeshbhoyar/Desktop/Programs/elasticsearchinstance/
$> export ES_HOME

# JDBC_IMPORTER_HOME
$> JDBC_IMPORTER_HOME=/Users/jayeshbhoyar/Desktop /Programs/elasticsearchinstance/elasticsearchjdbc1.5.2.0/
$> export JDBC_IMPORTER_HOME

9) Download the JDBC driver jar file from vendor site and copy it in “ES_HOME/lib” and “JDBC_IMPORTER_HOME/lib” folders

10) Go to “JDBC_IMPORTER_HOME” Folder and execute the below command:

bin=$JDBC_IMPORTER_HOME/bin 
lib=$JDBC_IMPORTER_HOME/lib 

echo '{     
"type" : "jdbc",     
"jdbc" : {         
"url" : "jdbc:mysql://localhost:3306/test",         
"user" : "",         "
password" : "",         
"sql" : "select *, id as _id from orders"     
}' | java \        
-cp "${lib}/*" \        
-Dlog4j.configurationFile=${bin}/log4j2.xml \        
org.xbib.tools.Runner \        
org.xbib.tools.JDBCImporter


It will start pulling the records from orders table and insert it into elasticsearch.

Sample File:
{
    "type" : "jdbc",
    "bulk_size" : 1000,
"max_bulk_requests" : 50,
"bulk_flush_interval" : "10s",
"max_concurrent_bulk_requests" : 16,
"max_bulk_actions" : 1000,
    "jdbc" : {
        "url" : "jdbc:oracle:thin:@hostname:port:SID",
        "driver": "oracle.jdbc.OracleDriver",
        "user" : "username",
        "fetchsize" : 1000,
        "password" : "password",
        "sql" : "SELECT * FROM ABC”,
        "index" : "INDEXNAME",
        "type" : "INDEXTYPE",
        "index_settings": {
            "index": {
                "refresh_interval": "5s"
            }
        } ,
        "max_bulk_actions" : 1000,
    "metrics" : {
        "enabled" : true,
        "interval" : "1m",
        "logger" : {
            "plain" : true,
            "json" : false
        }
    },
        "max_concurrent_bulk_requests" : 16,
        "connection_properties" : {
            "oracle.jdbc.TcpNoDelay" : false,
            "useFetchSizeWithLongColumn" : false,
            "oracle.net.CONNECT_TIMEOUT" : 10000,
            "oracle.jdbc.ReadTimeout" : 50000
        },
        "elasticsearch" : {
         "cluster" : "CLUSTERNAME",
         "host" : "ELASTICSEARCHHOSTNAME",
         "port" : 9300
    }
    }
}

For more settings and more information visit below link.


Hope this Helps!!!!
Jayesh Bhoyar

Reference:
https://github.com/jprante/elasticsearch-jdbc

Check More Articles on ElasticSearch:
http://technical-fundas.blogspot.com/2015/07/elasticsearch-bigdesk-plugin.html

Password Protected Solr Admin Page

As we all know Solr Admin Page is not password protected and anyone can get into Solr Admin Page. However this article will ...