Step-by-Step Building Internal Developer Platforms Guide

Featured image for Step-by-Step Building Internal Developer Platforms Guide
Spread the love

Step-by-Step Building Internal Developer Platforms Guide

Step-by-Step Building Internal Developer Platforms Guide

In the current landscape of cloud‑native development, organizations are constantly looking for ways to accelerate delivery, improve consistency, and reduce cognitive load on engineers. One of the most effective strategies is building internal developer platforms (IDPs) that provide a self‑service experience for developers while codifying best practices. This guide walks DevOps engineers and Site Reliability Engineers (SREs) through a practical, hands‑on implementation using Backstage, an open‑source framework that has become a de‑facto standard for internal tooling.

We will cover everything from initial project scaffolding to production‑grade security hardening, and we will sprinkle the discussion with real‑world examples, trade‑off analysis, and a curated list of learning resources. By the end of this article you will have a reusable blueprint – a building internal developer checklist – that you can adapt to any organization’s size and maturity level.

Why an Internal Developer Platform Matters

Before diving into the technical steps, it is worth understanding the strategic value of an IDP. At a high level, an IDP:

  • Provides a single pane of glass for developers to discover services, APIs, and infrastructure.
  • Encapsulates compliance, security, and cost‑optimization policies as code.
  • Reduces context switching by surfacing logs, metrics, and CI/CD pipelines in one place.
  • Accelerates onboarding of new engineers by presenting a curated, opinionated environment.

From a business perspective, these capabilities translate into faster time‑to‑market, lower operational overhead, and a more predictable engineering velocity. The challenge, however, is to design an IDP that is both flexible enough to serve diverse teams and strict enough to enforce organization‑wide standards – a classic trade‑off that we will revisit throughout the guide.

Prerequisites and High‑Level Architecture

Before you start, ensure you have the following baseline capabilities:

  1. A Kubernetes cluster (any managed service or on‑premises installation).
  2. Access to a Git repository that will host the Backstage source code and plugins.
  3. Existing CI/CD pipelines (e.g., GitHub Actions, Jenkins, or Tekton) that can be integrated with Backstage.
  4. Identity provider (IdP) support for Single Sign‑On (SSO) – typically OIDC or SAML.

The recommended architecture consists of three layers:

  • Presentation Layer: The Backstage UI, served via an ingress controller with TLS termination.
  • Service Layer: Backstage core services (catalog, scaffolder, techdocs) running as Deployments inside the cluster.
  • Integration Layer: Plugins that talk to external systems such as GitHub, Argo CD, Prometheus, and internal secret stores.

Below is a simplified diagram (expressed in ASCII for brevity):

+-------------------+       +-------------------+       +-------------------+
|   Developer UI   | <---> |   Backstage API  | <---> |   External Tools  |
| (React Frontend) |       | (Node.js Server) |       | (GitHub, Argo…)   |
+-------------------+       +-------------------+       +-------------------+
          ^                         ^                         ^
          |                         |                         |
          v                         v                         v
   Ingress (TLS)            Service Mesh (Istio)      Cloud APIs / SDKs

Step 1: Bootstrapping a Backstage Instance

Backstage provides a CLI that generates a production‑ready skeleton. Run the following commands on a workstation that has node (v18+) and npm installed:

# Create a new Backstage app named "internal-platform"
npx @backstage/create-app@latest internal-platform

# Change into the project directory
cd internal-platform

# Install dependencies and start the development server
yarn install && yarn start

After a few moments, the UI becomes reachable at http://localhost:3000. This local instance is useful for rapid iteration, but the next step is to containerize it for deployment.

Dockerizing the Application

Backstage ships with a Dockerfile, but you may want to customise it to include additional plugins. Below is an example Dockerfile that adds a custom security-scanner plugin:

FROM node:18-alpine AS builder
WORKDIR /app
COPY . .
RUN yarn install --frozen-lockfile && \\
    yarn tsc && \\
    yarn build

FROM node:18-alpine
WORKDIR /app
COPY --from=builder /app/packages/backend/dist ./packages/backend/dist
COPY --from=builder /app/packages/app/dist ./packages/app/dist
COPY --from=builder /app/node_modules ./node_modules
EXPOSE 7000
CMD ["node", "packages/backend/dist/index.js"]

Build the image and push it to your container registry:

docker build -t myregistry.com/internal-platform:latest .
docker push myregistry.com/internal-platform:latest

Step 2: Deploying to Kubernetes

Deploying Backstage to Kubernetes follows a Helm‑based approach. The official Backstage Helm chart supports configurable values for ingress, service accounts, and environment variables.

# Add the Backstage Helm repo
helm repo add backstage https://backstage.github.io/charts
helm repo update

# Create a namespace for the platform
kubectl create namespace internal-platform

# Install the chart with custom values (see values.yaml below)
helm install internal-platform backstage/backstage \\
    --namespace internal-platform \\
    -f values.yaml

Sample values.yaml that enables TLS via cert‑manager and configures OIDC SSO:

ingress:
  enabled: true
  className: nginx
  annotations:
    cert-manager.io/cluster-issuer: letsencrypt-prod
  hosts:
    - host: platform.mycompany.com
      paths:
        - path: /
          pathType: Prefix
  tls:
    - secretName: platform-tls
      hosts:
        - platform.mycompany.com

auth:
  environment: production
  provider: oidc
  clientId: "backstage-client-id"
  clientSecret: "${BACKSTAGE_CLIENT_SECRET}"
  discoveryUrl: "https://idp.mycompany.com/.well-known/openid-configuration"

Apply the manifest and verify that the UI is reachable via the configured domain. At this point you have a functional Backstage instance running in production, but the platform is still a blank canvas – the real power comes from extending it with plugins.

Step 3: Adding Core Plugins for a Complete Developer Experience

Backstage’s extensibility model revolves around plugins. For a robust IDP, we recommend the following core plugins:

  • Catalog: Central source of truth for all services, APIs, and resources.
  • Scaffolder: Template‑driven service creation (GitOps, Helm, Docker).
  • TechDocs: Documentation site generated from Markdown or MkDocs.
  • CI/CD Integration: Connect to your pipeline tool (GitHub Actions, Jenkins, Tekton).
  • Observability: Pull metrics and logs from Prometheus, Loki, and Jaeger.

Below is a snippet showing how to register a custom security-scanner plugin in packages/backend/src/plugins.ts:

import { createRouter } from '@backstage/plugin-security-scanner';
import { Router } from 'express';

export default async function registerPlugins(app: Router) {
  const securityRouter = await createRouter({
    logger: app.get('logger'),
    config: app.get('config'),
    discovery: app.get('discovery'),
  });
  app.use('/security-scanner', securityRouter);
}

Each plugin can expose its own UI components, API routes, and backend logic, enabling you to weave together a cohesive developer workflow.

Implementing a Template‑Driven Scaffolder

The Scaffolder plugin allows teams to generate new services from predefined templates. A typical template includes a cookiecutter‑style template.yaml file that defines the files to be rendered, the parameters to collect, and the actions to execute after creation (e.g., push to Git, register with the catalog).

apiVersion: scaffolder.backstage.io/v1beta3
kind: Template
metadata:
  name: node-service
  title: Node.js Service
  description: Scaffold a new Node.js microservice with Express
spec:
  owner: team-devops
  type: service
  parameters:
    - title: Service name
      description: Unique name for the new service
      required: true
      type: string
      name: name
  steps:
    - id: fetch
      name: Fetch template
      action: fetch:template
      input:
        url: ./template
    - id: publish
      name: Publish to GitHub
      action: publish:github
      input:
        repoUrl: "{{ parameters.repoUrl }}"
        defaultBranch: main
        description: "{{ parameters.description }}"
        protectBranch: true
    - id: register
      name: Register in catalog
      action: catalog:register
      input:
        catalogInfoUrl: "{{ steps.publish.output.remoteUrl }}/catalog-info.yaml"

When a developer clicks “Create” in the UI, Backstage will render a form based on the parameters, execute the steps, and automatically add the new component to the catalog – a perfect illustration of a building internal developer workflow.

Step 4: Enforcing Security and Compliance

Security is a non‑negotiable pillar of any IDP. Backstage can be hardened in several ways:

  • RBAC via OIDC claims: Map groups from your IdP to Backstage roles (admin, developer, auditor).
  • Network policies: Restrict pod‑to‑pod communication using Kubernetes NetworkPolicy objects.
  • Pod security standards: Enforce read‑only root filesystem, drop all capabilities, and run as non‑root.
  • Secrets management: Integrate with external secret stores (e.g., HashiCorp Vault) instead of storing credentials in ConfigMaps.

Here is an example NetworkPolicy that only allows the Backstage UI pod to talk to the backend service on port 7000:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: backstage-allow-backend
  namespace: internal-platform
spec:
  podSelector:
    matchLabels:
      app.kubernetes.io/name: backstage
  policyTypes:
    - Ingress
  ingress:
    - from:
        - podSelector:
            matchLabels:
              app.kubernetes.io/name: backstage-backend
      ports:
        - protocol: TCP
          port: 7000

Applying such policies ensures that a compromised UI component cannot reach other services in the cluster.

Expert Insight

“When building an internal developer platform, treat the platform itself as a first‑class service. Apply the same SLOs, monitoring, and incident response you would for any customer‑facing application. This mindset eliminates blind spots early and yields a more reliable developer experience.”
— Dr. Elena Martinez, Principal Engineer at a leading cloud‑native consultancy

Step 5: Observability and Incident Management

Developers need immediate feedback when something goes wrong. Backstage can surface alerts, logs, and traces directly in the UI through dedicated plugins.

  • Alert plugin: Pulls alerts from Alertmanager and displays them on the service’s overview page.
  • Log viewer plugin: Streams logs from Loki based on the service’s label selectors.
  • Trace plugin: Embeds Jaeger traces for end‑to‑end request debugging.

Integrating these observability tools not only improves the building internal developer performance but also aligns the platform with SRE best practices.

Sample Integration with Loki

The following snippet shows how to configure the LogViewer plugin to query Loki using a service’s component label:

export const LogViewerPage = () => {
  const { entity } = useEntity();
  const component = entity?.metadata?.annotations?.['backstage.io/component'];
  return (
    
  );
};

Step 6: Scaling the Platform and Managing Multi‑Team Governance

As adoption grows, you will encounter the classic

1. Architectural Foundations and System Design

When implementing robust solutions for building internal developer platforms, system architects must focus on structural durability, low latency, and decoupled designs. In projects involving Building internal developer platforms with Backstage, a modular design pattern is highly advantageous. This approach allows developers to isolate components, scale them independently, and optimize resource usage based on real-time request patterns. Using asynchronous messaging queues (such as RabbitMQ, Celery, or Apache Kafka) can offload intense tasks from the primary request thread, thereby ensuring high availability and protecting the system from cascading service failures.

Furthermore, the database layer must be designed with transaction safety, connection pooling, and replication in mind. Using read replicas can significantly reduce the load on the master node during heavy traffic spikes. Implementing an API gateway enables clean traffic routing, rate limiting, request validation, and unified security policies. This unified layout simplifies operational maintenance and speeds up troubleshooting workflows for technical teams.

2. Security Hardening and Threat Mitigation

Security is a paramount concern for any application operating with building internal developer platforms. Adhering to the principle of least privilege, access controls should be strictly limited across all components. For deployments related to Building internal developer platforms with Backstage, sensitive variables (such as database passwords, third-party API credentials, and TLS certificates) should never be stored directly in the source code or deployment scripts. Instead, they should be managed via cloud-native secrets managers (like AWS Secrets Manager, HashiCorp Vault, or Google Cloud Secret Manager) and loaded securely at runtime.

To secure the data layer, all external communication channels must be encrypted with modern TLS protocols. Input parameters should undergo rigorous validation and sanitization at the API gateway layer to prevent SQL injection, cross-site scripting (XSS), and malicious parameter tampering. Regular dependency vulnerability scanning (using tools like Snyk, Dependabot, or Bandit) should be integrated into the deployment pipeline to identify and remediate vulnerable packages early in the release cycle.

3. Scaling Strategies and Performance Optimization

Minimizing application latency and maximizing throughput are key indicators of a successful building internal developer platforms rollout. For systems executing workflows for Building internal developer platforms with Backstage, adopting a multi-tiered caching structure yields immediate performance gains. Tools like Redis or Memcached can store frequently accessed database queries, transient session variables, and parsed system configurations. This relieves pressure on back-end databases and decreases API response times to the low millisecond range.

In addition, using reverse proxies (such as Nginx or HAProxy) and Content Delivery Networks (CDNs) helps distribute request loads geographically and serve static assets with minimal delay. Autoscale rules (such as Horizontal Pod Autoscaling in Kubernetes or VM scale sets in cloud environments) should be defined using CPU, memory, and custom message queue length metrics to align compute resources with real-time user activity, optimizing hosting expenditures.

Related Posts

Scroll to Top