Ever feel like you're drowning in a sea of DevOps data? Jira tickets here, GitHub commits there, Jenkins builds somewhere else... it's enough to make any engineering team's head spin. Trying to make sense of this fragmented information to improve your development process can feel like searching for a needle in a haystack.
Enter Apache DevLake, an open-source platform that's here to rescue you from data chaos. DevLake ingests, analyzes, and visualizes data from all your favorite DevOps tools, transforming it into actionable insights that can boost your team's efficiency and drive engineering excellence.
In this blog, we'll explore what DevLake is, why it's essential for modern software development, and how it works. But we won't stop there! I'll also share an exclusive guide on how to set up DevLake with HTTPS using Docker Compose and Traefik, ensuring your valuable data stays secure.
Why DevLake? Taming the Data Beast
The problem is simple: your DevOps data lives in silos. Jira, GitHub, GitLab, Jenkins – each tool holds a piece of the puzzle, but getting a complete picture requires tedious manual effort. This fragmentation leads to:
Difficulty tracking progress: Are your sprints on track? Are you hitting your deadlines? It's hard to tell when data is scattered.
Bottlenecks and inefficiencies: Identifying and addressing bottlenecks becomes a guessing game without a unified view of your workflow.
Limited performance measurement: Measuring team performance and identifying areas for improvement becomes challenging.
DevLake tackles these challenges head-on. By collecting and centralizing your DevOps data, it provides a single source of truth, allowing you to:
Track key metrics: Monitor DORA metrics (Deployment Frequency, Lead Time for Changes, Mean Time to Recovery, Change Failure Rate) to understand your team's performance and identify areas for improvement.
Identify high-risk areas: Pinpoint bottlenecks, slowdowns, and potential problems early on.
Optimize workflows: Make data-driven decisions to streamline your development process.
How DevLake Works: From Chaos to Clarity
DevLake operates in three key stages:
Data Ingestion: DevLake connects to various DevOps tools, including Jira, GitHub, GitLab, Jenkins, and more, extracting relevant data through their APIs.
Data Processing and Analysis: The collected data is transformed and analyzed, creating valuable metrics and insights.
Visualization and Dashboards: DevLake provides built-in dashboards and visualization tools to present the data in a clear and understandable way. You can also create custom dashboards to meet your specific needs.
Setting up DevLake
Ready to experience the power of DevLake? Let's set it up with a secure HTTPS connection using Docker Compose and Traefik.
Prerequisites:
An AWS account
Basic knowledge of Docker and Docker Compose
Steps:
Launch an EC2 instance: Choose an appropriate instance type and operating system.
Install Docker and Docker Compose: Follow the official instructions for your OS.
Configure Traefik: Install Traefik and create a
docker-compose.yml
file to define it as your reverse proxy.Obtain SSL certificates: Use Let's Encrypt (Certbot) to get free SSL certificates for your domain.
Deploy DevLake: Add DevLake to your
docker-compose.yml
file, specifying the necessary configurations.Configure Traefik routing: Configure Traefik to route HTTPS traffic to the DevLake container.
version: "3"
services:
traefik:
image: "traefik:v2.8"
container_name: "traefik"
command:
- "--api.insecure=true"
- "--providers.docker=true"
- "--providers.docker.exposedbydefault=false"
- "--entrypoints.web.address=:80"
- "--entrypoints.websecure.address=:443"
- "--certificatesresolvers.myresolver.acme.tlschallenge=true"
- "--certificatesresolvers.myresolver.acme.email=NAME@EMAIL.COM"
- "--certificatesresolvers.myresolver.acme.storage=/letsencrypt/acme.json"
ports:
- "443:443"
- "80:80"
volumes:
- "/var/run/docker.sock:/var/run/docker.sock:ro"
- "./letsencrypt:/letsencrypt"
mysql:
image: mysql:8
volumes:
- mysql-storage:/var/lib/mysql
restart: always
ports:
- 3306:3306
environment:
MYSQL_ROOT_PASSWORD: admin
MYSQL_DATABASE: lake
MYSQL_USER: merico
MYSQL_PASSWORD: merico
TZ: UTC
command: --character-set-server=utf8mb4
--collation-server=utf8mb4_bin
--skip-log-bin
grafana:
image: devlake.docker.scarf.sh/apache/devlake-dashboard:v1.0.2-beta1
ports:
- 3002:3000
volumes:
- grafana-storage:/var/lib/grafana
environment:
GF_SERVER_ROOT_URL: "http://localhost:4000/grafana"
GF_USERS_DEFAULT_THEME: "light"
MYSQL_URL: mysql:3306
MYSQL_DATABASE: lake
MYSQL_USER: merico
MYSQL_PASSWORD: merico
TZ: UTC
restart: always
depends_on:
- mysql
devlake:
image: devlake.docker.scarf.sh/apache/devlake:v1.0.2-beta1
ports:
- 8080:8080
restart: always
volumes:
- devlake-log:/app/logs
env_file:
- ./.env
environment:
LOGGING_DIR: /app/logs
TZ: UTC
depends_on:
- mysql
config-ui:
image: devlake.docker.scarf.sh/apache/devlake-config-ui:v1.0.2-beta1
ports:
- 4000:4000
labels:
- "traefik.enable=true"
- "traefik.http.routers.whoami-http.entrypoints=web"
- "traefik.http.routers.whoami-http.rule=Host(`devlake.DOMAIN.COM`)"
- "traefik.http.routers.whoami-http.middlewares=whoami-https"
- "traefik.http.middlewares.whoami-https.redirectscheme.scheme=https"
- "traefik.http.routers.whoami.rule=Host(`devlake.DOMAIN.COM`)"
- "traefik.http.routers.whoami.entrypoints=websecure"
- "traefik.http.routers.whoami.tls.certresolver=myresolver"
- "traefik.http.services.whoami.loadbalancer.server.port=4000"
env_file:
- ./.env
environment:
DEVLAKE_ENDPOINT: devlake:8080
GRAFANA_ENDPOINT: grafana:3000
TZ: UTC
#ADMIN_USER: devlake
#ADMIN_PASS: merico
depends_on:
- devlake
volumes:
mysql-storage:
grafana-storage:
devlake-log:
Troubleshooting:
- If you encounter issues, check the DevLake documentation and community forums for solutions.
Exploring DevLake's Features: A Deeper Dive
DevLake offers a rich set of features to help you analyze your DevOps data:
Extensive Data Sources: Connect to a wide range of DevOps tools, including Jira, GitHub, GitLab, Jenkins, Bitbucket, and more.
Informative Dashboards: Access pre-built dashboards for DORA metrics, code quality, and other key performance indicators.
Customizable Metrics: Define your own metrics to track the specific data points that matter most to your team.
API and Integrations: Leverage DevLake's API to integrate with other tools and build custom applications.
Conclusion: Take Control of Your DevOps Data
DevLake empowers you to move beyond data silos and gain a holistic understanding of your software development process. By providing valuable insights and facilitating data-driven decisions, DevLake helps you optimize your workflows, improve team performance, and achieve engineering excellence.
Resources:
DevLake website: https://devlake.apache.org/
DevLake documentation: https://devlake.apache.org/docs/Overview/Introduction
DevLake community: https://devlake.apache.org/community/