Smart Data Access with HADOOP HIVE IMPALA. SAP HANA smart data access enables remote data to be accessed as if they are local tables in SAP HANA, without copying the data into SAP HANA. Not only does this capability provide operational and cost benefits, but most importantly it supports the development and deployment of the next generation of analytical applications which require the ability to access, synthesize and integrate data from multiple systems in real time regardless of where the data is located or what systems are generating it. Reference  http help. WhatsNewSAPHANAPlatformReleaseNotesen. Section 2. 4. 2. Currently Supported databases by SAP HANA smart data access include Teradata Database version 1. SAPSybase IQ version 1. ESD3 and 1. 6. 0. SAP Sybase Adaptive Service Enterprise version 1. ESD4. Intel Distribution for Apache Hadoop version 2. This includes Apache Hadoop version 1. Apache Hive 0. 9. Also Refer to SAP Note 1. Additional information about SPS0. SAP Note 1. 86. 87. Information about installing the drivers that SAP HANA smart data access supports. UPDATE Dec 0. 4 2. As of SPS0. 7 Hortonworks HDP1. Whens HDP 2. 0 coming appears to have been added to the official list, and remote caching of HADOOP Sources has been added, which should hopefully speed queries up for those tables in HADOOP that arent changing frequently. Jive. Servletpreview. Body4. 29. 6 1. HANASPS0. NEWSDA. UPDATE Jan 2. SAP HANA Academy now has a great collection of videos using Smart Data Access. Thanks SAP e. SAP HANA Academy SAP HANASDA HADOOP Configuring ODBC drivers SAP HANASDA HADOOP using the remote HADOOP data source SAP HANAUsing Smart Data Access SDA with HADOOP seems to me a great idea for balancing the strengths of both tools. Unfortunately for real time responsiveness  HIVE SQL currently isnt the most optimal tool in HADOOP instead its better used for batched SQL commands. Clouderas Impala, Hortonworks Stinger initiative  and  Map. Rs Drill  are all trying to address real time reporting. Ive only tested Impala so far, but Ive noticed speeds of 1. This is an educational video showing how to setup Eclipse for Hadoop Programming. If you would like to learn more please visit us at httptraining. Ive run into an error when installing the Windows SDK that Ive seen posted around the Internet, but none of the solutions are working for me. Here is the log Im. This article will guide you on how you can install and configure Apache Hadoop on a single node cluster in CentOS 7, RHEL 7 and Fedora 23 releases. Print this Post. Install Hadoop on Windows in 3 Easy Steps for Hortonworks Sandbox Tutorial. HIVE SQL queries. With that in mind I thought it would be interesting to test them both in HANA using SDA. Unfortunately Im using Clouderas open source Apache Hadoop distribution CDH, which isnt on SAPs approved list yet. However since SDA uses ODBC Ive managed to get it working using a third party ODBC driver from ProgressData. Direct. http www. NOTE Since CDH is not currently on this list Im sure SAP will NOT recommend you using this in a production environment. Hadoop-1.0-Installation-25.jpg' alt='Hadoop Installation On Windows 7' title='Hadoop Installation On Windows 7' />PUBLIC SAP Data Services Document Version 4. Support Package 8 14. Installation Guide for Windows. Hadoop installation on windows, This tutorial will explain you how to Hadoop installation on windows without cygwin in 10 mints hadoop 2. Stepbystep instructions on configuring Tomcat 6 and 7 and deploying apps on it, with and without Eclipse. YARN is the architectural center of Hadoop that allows multiple data processing engines such as interactive SQL, realtime streaming, data science and batch processing. KB/java/757934/4.png' alt='Hadoop Installation On Windows 7' title='Hadoop Installation On Windows 7' />HDFS Tutorial The Only Guide Youll Ever Need I think you guys will agree with me when I say The coming time is of BIG DATA ANALYTICS You are. I am getting a NoClassDefFoundError when I run my Java application. What is typically the cause of thisIf you do though get it working in a sandbox environment why not help by adding your voice for it be certified and added to the official list. With the disclaimers out of the way this is how SDA works. Remote Data Sources  Once you have your ODBC drivers install properly Remote Sources can be added for both HIVE and IMPALAExpanding the Remote Sources shows the tables that can be access by HANA. NOTE For me expanding the HIVE1 tree takes almost 2. IMPALA1 nodes in the hierarchy expanded quickly. In the above screen shots you will notice that both HIVE1 IMPALA1 share the same tables as they use the same HADOOP metastore. Data is NOT replicated in HIVE tables and IMPALA tables. The metastore just points to the tables files location within the HADOOP ecosystem, whether stored as text files, HBASE tables or column store PARQUET files to list just a few. There are some tables types file types that can only be read by HIVE or IMPALA, but there is a large overlap and this may converge over time. Virtual Tables Select Create virtual tables, from your Remote Source, in the schema of your choice. NOTE Ive previously created an HADOOP schema in HANA to store these virtual tables. Once created you can open the definition of the new virtual tables, as per normal HANA tables. Run some queries Simple HIVE query on my extremely small and low powered HADOOP cluster 2. SecondsNOTE In the HADOOP system, you can see above the HIVEs map reduce is kicked off. Simple IMPALA query on my extremely small and low powered HADOOP cluster reading the SAME table as HIVE lt 1 SecondNOTE Impala does not use MAPREDUCEWith Impala the source table type may impact speeds as well as these 2 simple examples demonstrate. IMPALA  HBASE table  4. K records in 4 seconds IMPALA  PARQUET Column Store 6. Million Records in 3 SecondsHADOOP HBASE source tables are better for small writes and updates, but are slower at reporting. HADOOP IMPALA PARQUET tables use Column store logic similar to HANA column tables which need which take more effort to write too efficiently, but are much faster at reads assuming not all the fields in a row are return, not that dis similar to HANA Column tables as well. You can think of Parquet tables, like the part of the HANA column table after MERGE DELTA, whereas the HBASE table is more like the uncompressed part of a HANA column table PRIOR to MERGE DELTA. HADOOP tables are still stored on Disk using HDFS rather than in memory, however they are making progress in caching tables into memory on the nodes, to better improve performance of queries. SQL for creating HADOOP Remote Source Unfortunately Hadoop remote source cant be manually configured yet. They do not appear in the drop down Since the HADOOP adapter doesnt appear in the list, use the HANA SQL editor to create the HADOOP Remote Sources e. DROP REMOTE SOURCE HIVE1 CASCADE DROP REMOTE SOURCE IMPALA1 CASCADE CREATE REMOTE SOURCE HIVE1 ADAPTER hiveodbc CONFIGURATION DSNhwp    WITH CREDENTIAL TYPE PASSWORD USING userhive passwordhive CREATE REMOTE SOURCE IMPALA1 ADAPTER hiveodbc CONFIGURATION DSNiwp     WITH CREDENTIAL TYPE PASSWORD USING userhive passwordhive CDH Driver Installation Unfortunately Cloudera doesnt yet provide ODBC drivers for SAP. I tried some of their other ODBC drivers for Micro Strategy without success. Fortunately a third party, Progress Data direct supplies  ODBC drivers for  HIVE and IMPALA running on CDH. Dowload their 1. 5 day trial and follow their steps for compiling it for HANA in Linux e. PROGRESSDATADIRECTCONNECT6. ODBC7. 1. 2LINUX6. Zgunzip PROGRESSDATADIRECTCONNECT6. ODBC7. 1. 2LINUX6. Ztar xf PROGRESSDATADIRECTCONNECT6. ODBC7. 1. 2LINUX6. In the HOME directory of  your hdbadm user you need to add odbc settings. Create 2 files    . ODBC DSN connections used need when creating a Remote Source. My 2 files appear as follows. LDLIBRARYPATHLDLIBRARYPATH usrlib optProgressData. DirectConnect. 64forODBC7. ODBCINIHOME. odbc. ODBC Data SourcesiwpData. Direct 7. 1 Impala Wire ProtocolhwpData. Direct 7. 1 Apache Hive Wire ProtocolODBCIANAApp. Code. Page4. Install. DiroptProgressData. DirectConnect. 64forODBC7. Trace0. Trace. Filetmpodbctrace. Trace. DlloptProgressData. DirectConnect. 64forODBC7. DriveroptProgressData. DirectConnect. 64forODBC7. DescriptionData. Direct 7. Impala Wire Protocol. Array. Size1. 02. Databasedefault. Default. Long. Data. Buff. Len1. 02. 4Default. Order. By. Limit 1. Enable. Describe. Param0. Host. NamePut the IP address of your HIVE gateway hereLogin. Timeout3. 0Logon. French Drain Around House Installation Electric. IDPasswordPort. Number2. Remove. Column. Qualifiers0. String. Describe. Type 9. Transaction. Mode0. Use. Current. Schema0hwpDriveroptProgressData. DirectConnect. 64forODBC7.