Spotify Insights with Elasticsearch is a Big Data project by Emmanuel Martínez that demonstrates how modern search-and-analytics infrastructure can be applied to a real-world music dataset. By loading a Spotify songs dataset into Elasticsearch and wiring up Kibana for visual dashboards, the project makes it easy to query song characteristics — duration, energy, valence, and more — directly from Jupyter Notebooks running inside a fully containerized Docker environment.Documentation Index
Fetch the complete documentation index at: https://mintlify.com/Emmanuel-Mtz-777/My-Personal-Portfolio/llms.txt
Use this file to discover all available pages before exploring further.
Project Metadata
| Field | Details |
|---|---|
| GitHub Repository | Emmanuel-Mtz-777/Spotify-Insights-with-Elasticsearch |
| Primary Language / Framework | Python / Elasticsearch DSL |
| Status | Open Source |
Tech Stack
| Technology | Purpose |
|---|---|
| ElasticSearch | Search & analytics engine — stores and indexes the Spotify dataset |
| Kibana | Data visualization dashboard — renders charts, histograms, and filters |
| Python / Jupyter Notebook | Data loading scripts and interactive query notebooks |
| Docker | Containerized environment running both Elasticsearch and Kibana via Docker Compose |
Overview
The project ingests a Spotify songs CSV dataset into an Elasticsearch index, making each track’s attributes — such asduration_ms, energy, valence, danceability, and tempo — available for full-text and structured queries. Python scripts handle the initial data-loading pipeline, mapping CSV columns to Elasticsearch fields. Once indexed, analysts can run Elasticsearch DSL queries from Jupyter Notebooks to slice and filter the dataset in seconds, without any database migrations or schema changes.
Kibana serves as the visual layer: pre-built dashboards surface distributions and correlations across thousands of tracks, turning raw JSON query results into readable bar charts, line graphs, and heat maps. Because both Elasticsearch and Kibana run as Docker containers orchestrated by Docker Compose, there is zero manual installation — a single docker compose up command stands up the entire analytics stack.
Key Features
Rich Song Querying
Filter and aggregate tracks by duration, energy, valence, danceability, and more using Elasticsearch DSL queries in Jupyter Notebooks.
Kibana Visualizations
Pre-configured Kibana dashboards present histograms, scatter plots, and distribution charts that make dataset patterns immediately visible.
Fully Containerized
Docker Compose brings up both Elasticsearch and Kibana with a single command — no local installation or manual configuration required.
Jupyter Notebook Analysis
Interactive notebooks document every query step, blending executable code, results, and markdown commentary in one place.
Technical Details
The entire infrastructure is defined in adocker-compose.yml file that provisions two containers: one running Elasticsearch (configured as a single-node cluster) and one running Kibana pointed at that cluster. A Python environment with the elasticsearch-py client and pandas library handles dataset ingestion — the loader script reads the CSV, normalises column types, and bulk-indexes documents into a dedicated spotify-tracks index.
Queries are authored using the Elasticsearch Python DSL, which maps cleanly to Elasticsearch’s JSON query syntax. Jupyter Notebooks act as the analysis interface, executing search() calls and rendering results inline. Kibana connects to the same index via an index pattern, enabling drag-and-drop dashboard construction without writing any additional code.
This architecture deliberately separates concerns: Docker owns infrastructure, Python owns data loading and querying, and Kibana owns presentation — making each layer independently replaceable.