Support Azure Databricks jdbc connection

[RyanDickinson1]

Please create a connector for Azure Databricks. ESRI support was unable to get me connected to Databricks, even with the jdbc files. Furthermore, I don't need an analysis of the data just connection to not have to manually retrieve and publish content. 

[AkshayHarshe]

Hi @RyanDickinson1 

Thanks for the information. We recognize that we may not be able to officially support different flavors of the datatabases which is why we built this https://github.com/Esri/insights-connector-guide if you haven't had a chance to take a look. This guide will help you create your connector. 

We do have a few samples that you can use as a starting point building Azure Databricks connector is something that you will have to experiment with on your own. That said, If you find any specific issues then please feel free to post on here we will take a look. 

[RyanDickinson1]

@AkshayHarshe thank you for your response. I have reviewed the connector guide and submitted a ticket to support for this issue. According to documentation, all jdbc connections are supported, so I'm surprised to learn that jdbc for Databricks didn't work.

ESRI support didn't believe that I would be able to connect, even with a custom config file. The support tech suggested posting an idea since they were unable to help. I am still attempting to connect it might just something not COTS, which I was hoping for. I had created the dialect and properties yml, but it didn't matter. Insight's rejected the jdbc files saying they are invalid. This folder does not contain all of the required JDBC drivers. Remove the folder and upload a folder with all of the required files. I downloaded the drivers directly from Azure Databricks.

[AkshayHarshe]

@RyanDickinson1 ,

Unfortunately, it is not possible to write and test many databases through the connectors which is why I said it will be an experiment and we would love to consider any specific issues you were facing. Also, If I understand correctly, Azure Databricks is a distributed system that sits between applications and data. The insights connector is sort of doing a similar role so I am not sure who that will work. The connector supports standard SQL so theoretically it should work.

is there any specific area where you are stuck?

Unfortunately, it is an off-the-shelf solution.

[RyanDickinson1]

Yes. The jdbc drivers for databricks is here. I placed the dialect and properties yml in that downloaded zip. When I add the zip to the connector type the error message read:

"This folder does not contain all of the required JDBC drivers. Remove the folder and upload a folder with all of the required files."

I downloaded the drivers directly from Azure Databricks. I attempted to upload only the windows drivers, only apple, and both. No combination of the connector file was accepted with the error maintaining that the folder doesn't have the drivers.

[AkshayHarshe]

@RyanDickinson1 ,

Did you update the properties and dialect yml files? They need to be configured to use the driver file and map the functions as your database needs.

The properties file holds information on the connection and the dialect file will map all the functions that Insights uses to send queries.

[RyanDickinson1]

@AkshayHarshe I did modify the template that ESRI provided in github. I updated the parameters of the dialect for what the database required.

[AkshayHarshe]

@RyanDickinson1 Will it be possible for you to share the connector package you built? Please obfuscate any connection details before you do. I will have a look.

[MuneerMajid]

@RyanDickinson1 @AkshayHarshe Did you all have any success with this requirement ? I am trying to access a table in Databricks and publish a feature service from it. 

[jakubk]

I've gotten it working. got the idea from here - https://gis.stackexchange.com/questions/438176/wkb-to-feature-class-in-arcgis-pro

you need to get the simba odbc driver and use the OleDb connection in Arc Pro

And select OleDb for ODBC

Type in your odbc connection string for databricks and off you go