Your browser does not seem to support JavaScript. As a result, your viewing experience will be diminished, and you may not be able to execute some actions.

Please download a browser that supports JavaScript, or enable it if it's disabled (i.e. NoScript).

How to purge / prevent using the prometheus local storage when using remote_write

A
Avante last edited by

I configured Prometheus to push metrics using remote_write but the local storage is still in use and keeps growing

How could Prometheus be configured to stop storing locally and only push metrics or to minimize the retention policy from the local storage?
Reply Quote 0
1 Reply Last reply
S
shizuka last edited by

Prometheus can run in agent mode. In this mode it doesn't store data locally - it just sends the scraped samples to the configured remote storage systems. See https://prometheus.io/blog/2021/11/16/agent/ for details.

P.S. see also the project I work on - https://docs.victoriametrics.com/vmagent.html - this is lightweight Prometheus-compatible agent, which understands Prometheus scrape configs. It usually requires lower amounts of memory and CPU comparing to Prometheus when scraping the same targets.
Reply Quote 0
1 Reply Last reply

prometheus 58

2
Posts

0
Views

Suggested Topics

O

Cost-efficient hosting for CI/CD?
Continuous Integration and Delivery (CI, CD) • kubernetes gitlab google cloud platform • • Oba22

2

0
Votes

2
Posts

1
Views

D

Given that you mentioned rarely using some k8s resources, you could explore some alternatives: With gcloud SDK you could create scripts to be executed for the periods you want to reduce GKE nodes (even to zero) during periods you're not using it With Google Cloud Functions (GCF) you can create some python/Node.js/Go script to access GKE APIs and also execute the operations to stop/reduce your instances. These GCF could be triggered by schedulers configured by Cloud Scheduler. The cost here for such executions will be really low or even zero.
G

Deploy an application using dll files to IIS on a VM
Continuous Integration and Delivery (CI, CD) • continuous integration deployment azure devops azure pipelines • • gionna

2

0
Votes

2
Posts

1
Views

It seems that there is no other way than creating NuGet packages from those .dll files required for the build, as this post on SO points out.
Z

Spring boot prometheus - difference between system_cpu_usage and process_cpu_usage
Continuous Integration and Delivery (CI, CD) • spring boot prometheus • • zymir

2

0
Votes

2
Posts

1
Views

As far as I understand, system.cpu.usage is host’s cumulative CPU usage in nanoseconds this includes user, system, idle CPU modes. (the sum of the /proc/stat CPU line) process.cpu.usage is the cpu usage for the JVM process aka CPU time used by the JVM process CPU usage aka CPU time is measured by counting the seconds that a CPU spends processing instructions from an application or from OS. Both metrics are of type Gauge because their values that can go up or down as their showing a resource usage .
How to append namespace before metric name in prometheus?
Continuous Integration and Delivery (CI, CD) • prometheus • • Bogopo

2

0
Votes

2
Posts

1
Views

J

Found this solution: static_configs: - targets: ['monitoring.api.endpoint.net'] metric_relabel_configs: - source_labels: [__name__] regex: '(.*)' replacement: 'compute_${1}' target_label: __name__
A

AWS Setting instance to use domain from another account
Continuous Integration and Delivery (CI, CD) • dns amazon web services amazon ec2 • • Amritpal

2

0
Votes

2
Posts

1
Views

A

If I'm getting this right; Maybe you can just create on your old account's Route53 public hosted zone a CNAME record (if you have a load balancer on your new account) or an A record for the instance on your new account. Just don't use the Alias functionality from Route53 as you wont find the resources from the new account.
J

Cloud DB operator responsibilities / common tasks
Continuous Integration and Delivery (CI, CD) • databases automation job role • • jules

2

0
Votes

2
Posts

1
Views

I always saw a DBA as someone who fits in between developers and operations when it comes to database management. On the one hand, they often take care of backups, clustering, replication, actual binary installation, file system management, etc. On the other, they also take care of relational DB schema, do performance or query optimization, and advise developers on the best way to lay out their application storage and configuration schemas. In theory, the first job can be done by a system administrator. The second job can be done by a senior developer or architect. In practice, it often makes sense to have someone who's well versed in all parts of running a database. Provisioning the instance is only a small part of what they do. Having your instance hosted in the cloud doesn't remove the need to have someone manage your schema or optimize performance. A cloud RDS with default settings likely won't be aware that key_buffer_size value needs to be set to 16M or some of your application's features won't run.
C

Amazon EKS from local kubectl does not cache MFA token/session
Continuous Integration and Delivery (CI, CD) • kubernetes amazon web services aws cli • • courtlanda

2

0
Votes

2
Posts

1
Views

T

Indeed, this is a pain. I've been fighting with it for the whole day yesterday, but managed to get it working, let's see if our cases are similar enough. My guess is that the usual aws configure configuration with a MFA ARN + the AWS SDK used to develop kubectl are not entirely compatible: I have two profiles set for this, one that has its mfa_serial_arn and role_arn that works as follows: First aws command I type, I get prompted for MFA token From there onwards, while session is active, I can work without typing in more tokens. kubectl asks for a token EVERY SINGLE TIME. This is bad. Now, I set a different profile, without any of the above, and I do the following: ``` keys=($(aws sts assume-role --role-arn <ARN_OF_DELEGATED_ROLE> \ --role-session-name <RANDOM_UNIQUE_NAME> \ --serial-number <ARN_OF_MFA_DEVICE>\ --token-code \ --query 'Credentials.[AccessKeyId,SecretAccessKey,SessionToken]' --output text)) # profile used here will be configured in ~/.aws/credentials export AWS_PROFILE=<PROFILE YOU WANT> aws configure set aws_access_key_id ${temp_keys[0]} aws configure set aws_secret_access_key ${temp_keys[1]} aws configure set aws_session_token ${temp_keys[2]} ``` This creates a temporary valid aws profile, that can be used BOTH with the aws cli, and kubectl.
A

Micronaut: How do you get the Prometheus metrics?
Software Programming • prometheus • • Amritpal

2

0
Votes

2
Posts

0
Views

K

That's what we' solved right now.Add a new endpoint. Or create a maping control. /metrics;Endpoint is returning scrape();Smash the endpoint. /prometheus(new endpoint doesn't work out /metrics);Turn off defolt endpoint on /metrics♪Config:micronaut: ... metrics: enabled: true export: prometheus: enabled: true ... endpoints: ... metrics: enabled: false prometheus: enabled: true
B

Jenkins plugin recomendation
Continuous Integration and Delivery (CI, CD) • jenkins jenkins plugins prometheus • • baileigh

2

0
Votes

2
Posts

1
Views

D

Have a look at the Summary Display plugin. Some more details about it (= quote from the linked page): ... allows an easy and fully customized build report display. Features Allow a rich summary report visible from both project and build page Reports must be written in an XML format according to the syntax described in section Syntax Description This plugin allow the parsing of several XML files Reports displayed are ordered according to the XML file names Element displayed are based on In general, the answer to such questions is usually found by searching the endless Jenkins plugin index.
What is the advantage of using Prometheus over ingesting metrics using logs?
Continuous Integration and Delivery (CI, CD) • monitoring metrics prometheus • • Mystic

2

0
Votes

2
Posts

1
Views

Efficiency If you don't need the file it saves space and trouble to leave it out. Logs require rotation and retention policies which are nice to skip. Reliability Skipping the file avoids issues with the file that could caused lost metrics like a full disk. Latency Skipping the file means that you remove latency from disk I/O. For most metrics you don't care if they show up a few seconds late, but if you're trying to build things to respond to some metric quickly. You might want to put it on Kafka also.
N

Using Prometheus to monitor Spring Boot Applications in Kubernetes Cluster
Continuous Integration and Delivery (CI, CD) • java spring boot kubernetes prometheus • • nishika

2

0
Votes

2
Posts

1
Views

A

The best way to deploy prometheus to kubernetes is with the helm chart: https://github.com/helm/charts/tree/master/stable/prometheus If you haven't used helm yet, its a very simple install to your laptop/etc and the cluster (and the newest version doesn't even require a cluster side install). It takes like 30 seconds. It's a package manager for kubernetes and it allows you to install, upgrade, list, and delete "charts" which are basically simple wrappers around kubernetes YAML files (with some nice templating). It would even be good to wrap your spring boot apps in a helm chart =). Once you've got prometheus running well, there is a good tutorial here on setting up actuator's prometheus endpoint in spring-boot and getting it picked up by prometheus by editing its values.yaml. https://www.callicoder.com/spring-boot-actuator-metrics-monitoring-dashboard-prometheus-grafana/ # my global config global: scrape_interval: 15s # Set the scrape interval to every 15 seconds. Default is every 1 minute. evaluation_interval: 15s # Evaluate rules every 15 seconds. The default is every 1 minute. # scrape_timeout is set to the global default (10s). # Load rules once and periodically evaluate them according to the global 'evaluation_interval'. rule_files: # - "first_rules.yml" # - "second_rules.yml" # A scrape configuration containing exactly one endpoint to scrape: # Here it's Prometheus itself. scrape_configs: # The job name is added as a label `job=<job_name>` to any timeseries scraped from this config. - job_name: 'prometheus' # metrics_path defaults to '/metrics' # scheme defaults to 'http'. static_configs: - targets: ['127.0.0.1:9090'] - job_name: 'spring-actuator' metrics_path: '/actuator/prometheus' scrape_interval: 5s static_configs: - targets: ['HOST_IP:8080'] There's also some info at the end of the tutorial about surfacing all this in Grafana for slick visualization.
B

How to calculate disk space required by Prometheus v2.2?
Continuous Integration and Delivery (CI, CD) • prometheus • • Burma

2

0
Votes

2
Posts

1
Views

C

To calculate disk space required by Prometheus v2.20 in bytes, use the query: retention_time_seconds * rate(prometheus_tsdb_head_samples_appended_total[2h]) * (rate(prometheus_tsdb_compaction_chunk_size_bytes_sum[2h]) / rate(prometheus_tsdb_compaction_chunk_samples_sum[2h])) Where retention_time_seconds is the value you've configured for --storage.tsdb.retention.time, which defaults to 15d = 1296000 seconds.
T

Pometheus-operator - modify Grafana mixin dashboard
Continuous Integration and Delivery (CI, CD) • prometheus grafana • • Trenton

2

0
Votes

2
Posts

1
Views

You should use https://github.com/helm/charts/tree/master/stable/prometheus-operator helm chart , also you can write Grafana dashboard script that would create dashboard dynamically , can be update as well later on.
C

How to setup basic auth for Prometheus deployed on K8s cluster using yamls?
Continuous Integration and Delivery (CI, CD) • kubernetes prometheus • • chanisef

2

0
Votes

2
Posts

1
Views

F

You need to add the required annotations to the ingress as required.For nginx, you can follow this
A

Jenkins: Re-use repository with reference option and do changes on it
Continuous Integration and Delivery (CI, CD) • git jenkins jenkins pipeline • • Anderson

2

0
Votes

2
Posts

1
Views

If this is not possible is there any other way without duplicating the 100GB for each jenkins job? To avoid frequent clones of the repository, you can put a single clone of the repository on-disk on your executor nodes outside of the normal workspace path. Then, in your Jenkins job, you can cd to that directory with the dir step, update the repo, and checkout the branch you want, e.g.: dir('/path/to/repo') { sh('git fetch --all') sh('git checkout origin/lineage-16.0') // your build steps here } However, this has the major drawback that the one repo will be shared across all of your jobs, which means you cannot run those jobs in parallel without risking corruption. It sounds like running jobs in parallel was one of your requirements, so this may be a no-go. You could probably get parallel builds to work by using a copy-on-write filesystem so that the repo appears to be in multiple independent directories but only actually has to be cloned once, but I've never tried something like this, so I can't say how well it would work.

How to purge / prevent using the prometheus local storage when using remote_write

Suggested Topics

Cost-efficient hosting for CI/CD? Continuous Integration and Delivery (CI, CD) • kubernetes gitlab google cloud platform • • Oba22

Deploy an application using dll files to IIS on a VM Continuous Integration and Delivery (CI, CD) • continuous integration deployment azure devops azure pipelines • • gionna

Spring boot prometheus - difference between system_cpu_usage and process_cpu_usage Continuous Integration and Delivery (CI, CD) • spring boot prometheus • • zymir

How to append namespace before metric name in prometheus? Continuous Integration and Delivery (CI, CD) • prometheus • • Bogopo

AWS Setting instance to use domain from another account Continuous Integration and Delivery (CI, CD) • dns amazon web services amazon ec2 • • Amritpal

Cloud DB operator responsibilities / common tasks Continuous Integration and Delivery (CI, CD) • databases automation job role • • jules

Amazon EKS from local kubectl does not cache MFA token/session Continuous Integration and Delivery (CI, CD) • kubernetes amazon web services aws cli • • courtlanda

Micronaut: How do you get the Prometheus metrics? Software Programming • prometheus • • Amritpal

Jenkins plugin recomendation Continuous Integration and Delivery (CI, CD) • jenkins jenkins plugins prometheus • • baileigh

What is the advantage of using Prometheus over ingesting metrics using logs? Continuous Integration and Delivery (CI, CD) • monitoring metrics prometheus • • Mystic

Using Prometheus to monitor Spring Boot Applications in Kubernetes Cluster Continuous Integration and Delivery (CI, CD) • java spring boot kubernetes prometheus • • nishika

How to calculate disk space required by Prometheus v2.2? Continuous Integration and Delivery (CI, CD) • prometheus • • Burma

Pometheus-operator - modify Grafana mixin dashboard Continuous Integration and Delivery (CI, CD) • prometheus grafana • • Trenton

How to setup basic auth for Prometheus deployed on K8s cluster using yamls? Continuous Integration and Delivery (CI, CD) • kubernetes prometheus • • chanisef

Jenkins: Re-use repository with reference option and do changes on it Continuous Integration and Delivery (CI, CD) • git jenkins jenkins pipeline • • Anderson

Cost-efficient hosting for CI/CD?
Continuous Integration and Delivery (CI, CD) • kubernetes gitlab google cloud platform • • Oba22

Deploy an application using dll files to IIS on a VM
Continuous Integration and Delivery (CI, CD) • continuous integration deployment azure devops azure pipelines • • gionna

Spring boot prometheus - difference between system_cpu_usage and process_cpu_usage
Continuous Integration and Delivery (CI, CD) • spring boot prometheus • • zymir

How to append namespace before metric name in prometheus?
Continuous Integration and Delivery (CI, CD) • prometheus • • Bogopo

AWS Setting instance to use domain from another account
Continuous Integration and Delivery (CI, CD) • dns amazon web services amazon ec2 • • Amritpal

Cloud DB operator responsibilities / common tasks
Continuous Integration and Delivery (CI, CD) • databases automation job role • • jules

Amazon EKS from local kubectl does not cache MFA token/session
Continuous Integration and Delivery (CI, CD) • kubernetes amazon web services aws cli • • courtlanda

Micronaut: How do you get the Prometheus metrics?
Software Programming • prometheus • • Amritpal

Jenkins plugin recomendation
Continuous Integration and Delivery (CI, CD) • jenkins jenkins plugins prometheus • • baileigh

What is the advantage of using Prometheus over ingesting metrics using logs?
Continuous Integration and Delivery (CI, CD) • monitoring metrics prometheus • • Mystic

Using Prometheus to monitor Spring Boot Applications in Kubernetes Cluster
Continuous Integration and Delivery (CI, CD) • java spring boot kubernetes prometheus • • nishika

How to calculate disk space required by Prometheus v2.2?
Continuous Integration and Delivery (CI, CD) • prometheus • • Burma

Pometheus-operator - modify Grafana mixin dashboard
Continuous Integration and Delivery (CI, CD) • prometheus grafana • • Trenton

How to setup basic auth for Prometheus deployed on K8s cluster using yamls?
Continuous Integration and Delivery (CI, CD) • kubernetes prometheus • • chanisef

Jenkins: Re-use repository with reference option and do changes on it
Continuous Integration and Delivery (CI, CD) • git jenkins jenkins pipeline • • Anderson