Installation of Grafana Cloud in Timefold managed Kubernetes clusters

Login to Grafana Cloud Timefold account and follow the steps to prepare installation

https://timefoldai.grafana.net/a/grafana-k8s-app/configuration/cluster-config
In step 4. Deploy Monitoring Resources on the Cluster copy the values for helm chart and save it into a file called grafana-values.yaml

The content for that file should be extracted from the command - all that is within
```
<<EOF
...

EOF
```

Create new file processor_config.river and add following rules into it

discovery.relabel "timefold_orbit_backend_pods" {
  targets = discovery.kubernetes.pods.targets  // Gets all pods that have label ai.timefold/component set to backend
  rule {
    source_labels = ["__meta_kubernetes_pod_label_ai_timefold_component"]
    regex = "backend"
    action = "keep"
  }
}

prometheus.scrape "timefold_orbit_backend" {
  targets = discovery.relabel.timefold_orbit_backend_pods.output
  metrics_path = "/q/metrics"
  forward_to = [prometheus.relabel.timefold_orbit.receiver]
}

prometheus.relabel "timefold_orbit" {
  rule {
    source_labels = ["__name__"]
    regex = "pod.*|namespace.*|container.*|cluster|up|timefold.*|jvm.*|http.*|worker.*|process.*"
    action = "keep"
  }

  rule {
    source_labels = ["instance"]
    regex = ".*"
    replacement = "NAME_OF_THE_CLUSTER"
    target_label = "cluster"
  }
  forward_to = [prometheus.remote_write.metrics_service.receiver]
}

Replace NAME_OF_THE_CLUSTER in processor_config.river file with the same name used in Grafana agent configuration.

This defines processing rules to collect metrics from Timefold Platform pods. Currently focuses on backend components as the metrics path is set to /q/metrics. In case other components will expose metrics new rules will have to be added.

Install the Grafana Agent into the cluster with following command

helm repo add grafana https://grafana.github.io/helm-charts &&
helm repo update &&
helm upgrade --install --atomic --timeout 120s grafana-k8s-monitoring grafana/k8s-monitoring --namespace "grafana" --create-namespace --values grafana-values.yaml --set-file extraConfig=processor_config.river