It is essential to have reliable tools for data visualization and analysis. Grafana and Druid are two such powerful tools that, when integrated, can provide insights into your data in an efficient and visually appealing manner. In this tutorial, we'll walk through the process of integrating Grafana with Druid to create insightful dashboards.

Setting up Druid

First, we need to set up Druid and create a sample datasource. Follow these steps:

1. Navigate to the Druid directory (in this tutorial, we use Apache Druid 29.0.1 release):

cd apache-druid-29.0.0

2. Start the Druid micro-quickstart:

./bin/start-micro-quickstart

3. Follow the tutorial Load data with native batch ingestion to create a sample datasource in Druid.

Note: To stop Druid at any time, use CTRL+C in the terminal. This exits the 'bin/start-druid' script and terminates all Druid processes.

Installing Grafana

Now, let's install Grafana locally on Ubuntu:

1. Install prerequisite packages:

sudo apt-get install -y apt-transport-https software-properties-common wget

2. Import the GPG key:

sudo mkdir -p /etc/apt/keyrings/
wget -q -O - https://apt.grafana.com/gpg.key | gpg --dearmor | sudo tee /etc/apt/keyrings/grafana.gpg > /dev/null

3. Update the list of available packages:

sudo apt-get update

4. Install Grafana OSS:

sudo apt-get install grafana

For more information, refer to the Install Grafana on Debian or Ubuntu |  Grafana documentation.

Installing Druid Plugin for Grafana

To integrate Druid with Grafana, install the Druid plugin:

grafana-cli plugins install grafadruid-druid-datasource

(Source: Druid plugin for Grafana | Grafana Labs)

Running Grafana

Start the Grafana server with systemd:

sudo systemctl start grafana-server

To verify that the service is running:

sudo systemctl status grafana-server

You can stop Grafana using:

sudo systemctl stop grafana-server

For more information, see Start the Grafana server |  Grafana documentation.

Configuring Grafana

Grafana's backend configuration options are defined in its config file, usually located at /etc/grafana/grafana.ini on Linux systems.

In this config file, you can change things like the default admin password, HTTP port, Grafana database (sqlite3, MySQL, Postgres), authentication options (Google, GitHub, LDAP, auth proxy), and many other options.

Signing in to Grafana

1. Open your web browser and go to 'http://localhost:3000'. (The default Grafana port is '3000').

2. Login with your admin user (default is 'admin/admin').

3. Change the password when prompted.

For detailed instructions, see Sign in to Grafana |  Grafana documentation.

Integrating Grafana with Druid

1. In Grafana, navigate to Home menu → Connections → Data sources.

You’ll see the Druid data source, which appeared here due to Druid-Grafana plugin installation.

2. Select 'grafadruid-druid-datasource'.

3. Type the Druid URL (e.g., 'http://localhost:8888') and provide credentials for Druid user if your Druid has Basic Authentication.

4. Press Save&test the connection.

Creating Sample Dashboard

Now, let's proceed to create a sample dashboard in Grafana, leveraging the integration with Druid as the datasource.

1. Go to Home menu → Dashboards → New → New dashboard.

2. Click Add visualization.

3. Select the Druid datasource.

4. Write a query in the query language of your data source (in this tutorial - SQL).

Example Queries:

Pie Chart: Real Users Quantity by Country (Non-Robots)

SELECT
  "countryName",
  COUNT("isRobot") AS "Count"
FROM "wikipedia"
WHERE "countryName" IS NOT NULL AND "isRobot"=false
GROUP BY 1
ORDER BY 2 DESC
LIMIT 10

5. Click the refresh dashboard icon to query the data source and update the visualizations.

6. Within the visualization list, select a visualization type (pie chart).

7. To visualize our pie chart, choose All values in the section Value options.

8. Under Panel options, enter a title and description for your panel.

9. Here is our pie chart:

10. Click Apply to save the visualization.

11. To add more visualizations to the dashboard, click Add in the dashboard header and select Visualization in the drop-down.

12. Then we repeat the process to create 3 more sample visualizations:

• Top-10 Maximum comment length by country (Bar chart):

SELECT 
  MAX("commentLength") AS max_comment_length,
  "countryName"
FROM "wikipedia"
WHERE "cityName" IS NOT NULL
GROUP BY "countryName"
ORDER BY 1 DESC
LIMIT 10

• Top-10 Cities with max qty of the longest comments (Bar gauge):

SELECT
  "cityName",
  COUNT("commentLength") AS "Count"
FROM "wikipedia"
WHERE "cityName" IS NOT NULL
GROUP BY 1, "commentLength"
ORDER BY 2 DESC
LIMIT 10

• Qty of real users vs robots (Gauge):

SELECT
"isRobot",
COUNT(isRobot) AS users_qty
FROM "wikipedia"
GROUP BY "isRobot"

13. Finally, organize the visualizations on the dashboard and save the dashboard.

14. The resulting dashboard will be like this:

Creating Grafana User in Druid

Let's set up a Grafana user in Druid with read-only access to ensure secure data access.

• user: grafana_reader
• pass: pass_grafana

1. Create a Grafana User: Run the following command to create a Grafana user named 'grafana_reader':

curl -u admin:password1 -XPOST http://localhost:8081/druid-ext/basic-security/authentication/db/MyBasicMetadataAuthenticator/users/grafana_reader

2. Add Credentials for Grafana User: Add a password for the Grafana user using the following command:

curl -u admin:password1 -H'Content-Type: application/json' -XPOST http://localhost:8081/druid-ext/basic-security/authentication/db/MyBasicMetadataAuthenticator/users/grafana_reader/credentials --data-raw '{"password": "pass_grafana"}'

3. Create a corresponding authorizer:

curl -u admin:password1 -XPOST http://localhost:8081/druid-ext/basic-security/authorization/db/MyBasicMetadataAuthorizer/users/grafana_reader

4. Create an authorizer role to control permissions:

curl -u admin:password1 -XPOST http://localhost:8081/druid-ext/basic-security/authorization/db/MyBasicMetadataAuthorizer/roles/grafanareadrole

5. Assign 'grafanareadrole' role to 'grafana_reader user':

curl -u admin:password1 -XPOST http://localhost:8081/druid-ext/basic-security/authorization/db/MyBasicMetadataAuthorizer/users/grafana_reader/roles/grafanareadrole | jq 

6. Create a file with permissions for our role:‍


'perms_grafana_reader.json':

[
   {
     "resource": {
       "type": "DATASOURCE",
       "name": ".*"
     },
     "action": "READ"
   },
   {
     "resource": {
       "type": "STATE",
       "name": ".*"
     },
     "action": "READ"
   },
   {
     "resource": {
       "type": "CONFIG",
       "name": ".*"
     },
     "action": "READ"
   }
]

7. Attach permissions to the roles:

curl -u admin:password1 -H'Content-Type: application/json' -XPOST --data-binary @perms_grafana_reader.json http://localhost:8081/druid-ext/basic-security/authorization/db/MyBasicMetadataAuthorizer/roles/grafanareadrole/permissions

Testing RBAC

Verify the role-based access control setup by checking user information and role details.

1. Verify User Information: To see all created users' names, execute the following command:

curl -u admin:password1 http://localhost:8081/druid-ext/basic-security/authorization/db/MyBasicMetadataAuthorizer/users

2. Check Role Information: Check the role information of the Grafana user 'grafana_reader' using the following command:

curl -u admin:password1 http://localhost:8081/druid-ext/basic-security/authorization/db/MyBasicMetadataAuthorizer/users/grafana_reader?full

Conclusion

That's it! You've successfully integrated Grafana with Apache Druid and created sample dashboards for data visualization and analysis. Experiment with different queries and visualization types to gain deeper insights into your data.