Building and mon­i­tor­ing Flask Application with Prometheus and Grafana

1. Setting up the Quote Application

The start­ing point is building the web ap­pli­ca­tion that will dis­play quotes (along with au­thor’s name and im­age) and then ex­pose sev­er­al met­rics used to mon­i­tor over­all ap­pli­ca­tion health and per­for­mance.

Flask is a web ap­pli­ca­tion frame­work de­signed to sim­pli­fy the process of build­ing web ap­pli­ca­tions us­ing Python. Flask phi­los­o­phy re­volves around sim­plic­i­ty, flex­i­bil­i­ty, built-in de­vel­op­ment serv­er, etc. which makes it per­fect fit for this blog’s pur­pose. Ap­pli­ca­tion will serve as the centerpiece of mon­i­tor­ing so­lu­tion, al­low­ing users to in­ter­act with it by dis­play­ing quotes, han­dling er­rors and ex­pos­ing per­for­mance met­rics.

1.2. Ap­pli­ca­tion Endpoints

Ap­pli­ca­tion de­fines mul­ti­ple spe­cif­ic URLs (API end­points), that can be used to ac­cess a par­tic­u­lar func­tion or data:

Home End­point (/)

Main en­try point of the ap­pli­ca­tion. When ac­cessed, it serves an HTML page that dis­plays a ran­dom quote (with au­thor’s name and im­age) from a pre­de­fined list. The user can click on the dis­played quote to view the next one in the se­quence. Each time a quote is dis­played, a met­rics that stores the num­ber of times that par­tic­u­lar quote has been dis­played is up­dat­ed.

Met­rics End­point(/metrics)

This end­point ex­pos­es all col­lect­ed met­rics in a for­mat com­pat­i­ble with Prometheus. Met­rics in­clude re­quest counts, er­ror counts, CPU and mem­o­ry us­age, up­time, and var­i­ous oth­ers re­lat­ed to the ap­pli­ca­tion’s per­for­mance and re­source con­sump­tion.

It is cre­at­ed to pro­vide a cen­tral­ized end­point for col­lect­ing and ex­port­ing met­rics (Prometheus scrapes this end­point to gath­er data for analy­sis, alert­ing and visualization).

Health Check End­point (/health)

This end­point re­turns the cur­rent health sta­tus of the ap­pli­ca­tion. It al­ways re­sponds with a cus­tom sta­tus of “Sys­tem is healthy”, along with a met­ric HEALTH_STATUS set to 1, in­di­cat­ing that the ap­pli­ca­tion is run­ning as ex­pect­ed.

Quote End­point (/quote/<int:index>)

This end­point re­turns the next quote in the se­quence based on the pro­vid­ed in­dex. If the next quote exists, endpoint returns a custom message “Quote counter incremented”. Each time a quote is re­trieved, the cor­re­spond­ing counter for that quote is in­cre­ment­ed.

1.3. Application Metrics

Met­rics are key in or­der to make im­prove­ments to the sys­tem (ap­pli­ca­tion) and user ex­pe­ri­ence as well as for main­tain­ing ex­pect­ed up­time and per­for­mance. Met­rics are used to mea­sure and track the per­for­mance and over­all health of Quote ap­pli­ca­tion:

1.4. Running the Application

Ap­pli­ca­tion is con­tainer­ized and de­pend­ing on im­age cre­ation, it can be cre­at­ed in two ways.

How to create the application:

2. Setting up Prometheus

Prometheus is an open-source system mon­i­tor­ing and alert­ing tool. It col­lects and stores its met­rics as time se­ries data, i.e., met­rics in­for­ma­tion is stored with the time­stamp at which it was record­ed, along­side op­tion­al key-val­ue pairs called la­bels. It uses PromQL, a pow­er­ful query lan­guage for query­ing time se­ries data.

Tip: More in­for­ma­tion on Prometheus can be found here.

2.1. In­stalling Prometheus in a Dock­er Net­work

Quote ap­pli­ca­tion is in­stalled as Dock­er con­tain­er, and lo­cat­ed in Dock­er net­work called mon­i­tor­ing-net­work, so Prometheus will also be in­stalled and used as Dock­er con­tain­er:

Prometheus con­tain­er uses last of­fi­cial Prometheus Dock­er im­age, and is ex­posed on stan­dard Prometheus port (9090). To spec­i­fy the scrape con­fig­u­ra­tion the prometheus.yml file is cre­at­ed and bind mount­ed to de­fault con­fig­u­ra­tion file lo­cat­ed in /etc/prometheus di­rec­to­ry.

Since both ser­vices are con­nect­ed to the same net­work (mon­i­tor­ing-net­work), the Prometheus con­tain­er can com­mu­ni­cate with the Quote ap­pli­ca­tion con­tain­er and scrape its met­rics.

Cre­ate the Prometheus con­tain­er, by run­ning the fol­low­ing com­mand:

2.2. Con­fig­ur­ing Prometheus to Scrape Met­rics

As men­tioned above, a cus­tom prometheus.yml file is cre­at­ed and bind mount­ed to de­fault con­fig­u­ra­tion file lo­cat­ed in /etc/prometheus di­rec­to­ry. Prometheus uses this file to determine where to scrape met­rics from and how fre­quent­ly to scrape them.

The con­tent of prometheus.yml file:

Con­fig­u­ra­tion file de­scrip­tion:

• Con­fig­u­ra­tion file tells Prometheus to scrape the Quote ap­pli­ca­tion met­rics every 15 sec­onds.
• job_name: Spec­i­fies the name of the scrap­ing job, which can be used in queries to fil­ter met­rics (re­lat­ed to this spe­cif­ic job).
• metrics_path: De­fines the end­point Prometheus will use to scrape met­rics (in this case /metrics).
• targets: A list of tar­gets from which Prometheus will col­lect met­rics (in this case dc-quote-app; name of Quote ap­pli­ca­tion con­tain­er).

2.3. Exploring Prometheus Metrics

Once the Prometheus is con­fig­ured, queries can be used to check if Prometheus is scrap­ing met­rics cor­rect­ly.

Check out these examples:

Now, when Prometheus is con­fig­ured to scrape the Quote ap­pli­ca­tion met­rics, the next step is to set up Grafana to vi­su­al­ize the met­rics col­lect­ed by Prometheus.

3. Setting up Grafana

Grafana is an open-source an­a­lyt­ics andin­ter­ac­tive vi­su­al­iza­tion web ap­pli­ca­tion that al­lows users to in­gest data from various sources, query this data, and dis­play it. Grafana comes with va­ri­ety of vi­su­al­iza­tion op­tions, which are split into pan­els that are then used to build the Grafana dash­board.

Tip: More in­for­ma­tion on Grafana can be found here.

4. Visualizing Quote Application Metrics in Grafana

Once the Prometheus is set as a data source, the next step is to cre­ate a dash­board to vi­su­al­ize the Quote ap­pli­ca­tion met­rics.

Steps to cre­ate a dash­board (com­mon steps for all pan­els):

1. In the Grafana side­bar, click “+” icon and se­lect “Dash­board”
2. Click “Add new pan­el”
3. In the Query sec­tion, se­lect “Prometheus” as the data­source.

Pan­els are grouped into rows, and vi­su­al­ized based on the com­mon cri­te­ria (e.g. ap­pli­ca­tion sta­tus met­rics, re­quests sta­tus met­rics and quote re­lat­ed met­rics).

4.1. Mon­i­tor­ing Ap­pli­ca­tion Health, Up­time and Re­source Us­age

All vi­su­al­iza­tions are cre­at­ed us­ing Gauge or Time Se­ries pan­els. Met­rics used in ap­pli­ca­tion health sta­tus, up­time and re­source us­age vi­su­al­iza­tions:

• HEALTH_STATUS
• APP_TIME
• CPU_USAGE
• MEMORY_USAGE
• MEMORY_USAGE_MB

4.2. Vi­su­al­iz­ing Re­quest and Er­ror Met­rics

All vi­su­al­iza­tions are cre­at­ed us­ing Stat or Time Se­ries pan­els. Met­rics used in re­quests vi­su­al­iza­tions:

• REQUEST_COUNT
• ERROR_COUNT
• SUCCESSFUL_REQUESTS

All vi­su­al­iza­tions are cre­at­ed us­ing Stat, Time Se­ries, and Bar Gauge pan­els. Met­rics used in quotes vi­su­al­iza­tions:

• TO­TAL_QUOTES_DIS­PLAYED
• quote_<in­dex>_count

5. Scaling and Enhancing Application Monitoring

5.1. Scal­ing the Quote Ap­pli­ca­tion

Quote ap­pli­ca­tion cur­rent­ly has one func­tion, which is to dis­play ran­dom quote (with au­thors name and im­age), which makes it a pret­ty light­weight in terms of ap­pli­ca­tion. But if the func­tion­al­i­ty of Quote ap­pli­ca­tions in­creas­es, ap­pli­ca­tion scal­ing could come in handy.

There are cou­ple of ways to achieve ap­pli­ca­tion scal­ing, but since Quote ap­pli­ca­tion is al­ready con­tainer­ized, adding more in­stances it the sim­plest ap­proach.

The first step is to en­sure that Quote ap­pli­ca­tions is state­less, which means that no persistent state should be held with­in the ap­pli­ca­tion it­self.

To scale the ex­ist­ing Quote ap­pli­ca­tion, sim­ply spec­i­fy the num­ber of repli­cas:

The deploy.replicas op­tion un­der quote-app in­di­cates that two in­stances of the Quote ap­pli­ca­tion should be cre­at­ed.

5.2. Scal­ing the Prometheus

As the num­ber of Quote in­stances in­creas­es, a sin­gle Prometheus in­stance may not be able to han­dle the in­creased load. Sim­i­lar for Quote ap­pli­ca­tion, there are sev­er­al strate­gies for scal­ing Prometheus, and sim­plest ap­proach is to use fed­er­a­tion.

Fed­er­a­tion al­lows a Prometheus serv­er to scrape se­lect­ed time se­ries from an­oth­er Prometheus in­stance. Ba­si­cal­ly, it al­lows scal­ing by de­ploy­ing mul­ti­ple Prometheus in­stances, each scrap­ing a sub­set of Quote ap­pli­ca­tion met­rics, and then ag­gre­gat­ing the data into a sin­gle glob­al in­stance.

First, update the docker-compose.yml file to deploy three instances of Prometheus (two instances that will monitor Quote applications, and a federate instance that will be used to aggregate the data):

The /federate end­point al­lows re­triev­ing the cur­rent val­ue for a se­lect­ed set of time se­ries in that serv­er. At least one match[] URL pa­ra­me­ter must be spec­i­fied to se­lect the se­ries to ex­pose. To fed­er­ate met­rics from one serv­er to an­oth­er, con­fig­ure des­ti­na­tion Prometheus serv­er to scrape from the /federate end­point of a source serv­er while also en­abling the honor_labels scrape op­tion (to not over­write any la­bels ex­posed by the source serv­er).

Fed­er­a­tion en­ables the sys­tem to scale hor­i­zon­tal­ly while en­sur­ing high avail­abil­i­ty and per­for­mance. Ad­di­tion­al­ly, each lo­cal in­stance is op­ti­mized for spe­cif­ic ser­vice.

5.3. Con­fig­ur­ing Alerts and No­ti­fi­ca­tions in Prometheus

Alert­ing with Prometheus is sep­a­rat­ed into two parts. Alert­ing rules in Prometheus servers send alerts to an Alert­man­ag­er. The Alert­man­ag­er then man­ages those alerts, in­clud­ing send­ing out no­ti­fi­ca­tions via mul­ti­ple meth­ods (e.g. email, chat plat­form, etc.).

The main steps to set­ting up alert­ing and no­ti­fi­ca­tions are:

• Set­up and con­fig­ure the Alert­man­ag­er
• Con­fig­ure Prometheus to talk to the Alert­man­ag­er
• Cre­ate alert­ing rules in Prometheus

Add the fol­low­ing code snip­pet in docker-compose.yml file to add the Alert­man­ag­er into ex­ist­ing con­fig­u­ra­tion:

New ser­vice for Alert­man­ag­er ex­pos­es its UI on port 9093 and over­writes the de­fault con­fig­u­ra­tion file in /etc/alertmanager di­rec­to­ry. The cus­tom alermanager.yml file con­tains con­fig­u­ra­tions for rout­ing alerts and han­dling no­ti­fi­ca­tions:

Note: Email configuration uses arbitrary values since email notifications were not tested!

Cre­ate the Prometheus con­tain­er, by run­ning the fol­low­ing com­mand:

If every­thing is set­up cor­rect­ly, and there were no er­rors while cre­at­ing a con­tain­er, Alert­man­ag­er UI will be avail­able on http://localhost:9093.

5.3.2. Con­fig­ure Prometheus to talk to the Alert­man­ag­er

Once the Alert­man­ag­er is up and run­ning, it needs to be con­nect­ed with the Prometheus, which can be done by adding the alerting sec­tion to the ex­ist­ing Prometheus con­fig­u­ra­tion file (prometheus.yml):

The alerting sec­tion de­fines where Prometheus sends alerts.

Once Alert­man­ag­er is up and run­ning and con­nect­ed to Prometheus, the last thing is to de­fine the alert­ing rules. Cre­ate new file (e.g. alert_rules.yml) and de­fine the rule to trig­ger an alert based on ap­pli­ca­tion re­source us­age (e.g. Memory Us­age):

Ex­pla­na­tion:

• Name: Com­mon name for all alerts re­lated to Quote ap­pli­ca­tion
• Alert: Name of the alert that will be trig­gered based on the de­fined pa­ra­me­ters
• Expr: Ex­pres­sion that will be checked in Prometheus, which is ba­si­cal­ly used as a trigger for alert
• For: Time that Prometheus will wait un­til the alerts is trig­gered (ex­pres­sion must per­sist for 1 minute)
• Labels: Used for fil­ter­ing and group­ing
• Annotations: De­tailed in­for­ma­tion about alert

Up­date the Prometheus vol­umes sec­tion in ex­ist­ing docker-compose.yml file to bind mount the cus­tom rules file to rules file in the /etc/prometheus di­rec­to­ry:

Up­date the de­fault Prometheus con­fig­u­ra­tion file (prometheus.yml) with the alert file, by defin­ing rule_files sec­tion:

Once the alert is trig­gered, it will ap­pear in Alert­man­ag­er on http://localhost:9093, in  ad­di­tion, pend­ing or fir­ing alerts can also be checked in Prometheus at http://localhost:9090/alerts.

Above con­fig­u­ra­tion en­sures that Quote ap­pli­ca­tion is con­tin­u­ous­ly mon­i­tored, and alerts are gen­er­at­ed for crit­i­cal is­sues, pro­vid­ing no­ti­fi­ca­tions to take cor­rec­tive ac­tions.

Conclusion

In this blog, a Quote ap­pli­ca­tion that dis­plays quotes and ex­pos­es op­er­a­tional met­rics was cre­at­ed. Met­rics like re­sources us­age, ap­pli­ca­tion health sta­tus, up­time, num­ber of suc­cess­ful and er­ror re­quests, and the num­ber of quotes dis­played, were cru­cial for gain­ing vis­i­bil­i­ty into the ap­pli­ca­tion’s health. Ap­pli­ca­tion stack was ex­pand­ed with Prometheus, which was used for scrap­ing ap­pli­ca­tion met­rics and Grafana for vi­su­al­iz­ing the data.

In Grafana, sev­er­al types of pan­els (Gauge, Stat, Time se­ries, etc.) were used to mon­i­tor the met­rics. Each vi­su­al­iza­tion pro­vid­ed in­sights into the ap­pli­ca­tion’s real-time state, en­abling to mea­sure re­source uti­liza­tion, and eval­u­ate the over­all per­for­mance.

Ap­pli­ca­tion func­tion­al­i­ty in­crease is han­dled by im­ple­ment­ing scal­ing op­er­a­tions on ap­pli­ca­tion it­self as well as oth­er com­po­nents like scal­ing Prometheus through fed­er­a­tion. Ad­di­tion­al­ly, an Alert­man­ag­er was in­tro­duced to han­dle alerts. To iden­ti­fy slow re­spons­es and fail­ure rates from ap­pli­ca­tion, ad­vanced met­rics like re­quest la­ten­cy and HTTP Er­ror rates were im­ple­ment­ed. These met­rics are im­por­tant in de­tect­ing per­for­mance bot­tle­necks, en­hanc­ing the ap­pli­ca­tion’s over­all re­li­a­bil­i­ty.

Mon­i­tor­ing met­rics is crit­i­cal for build­ing re­li­able and scal­able ap­pli­ca­tions. With the Prometheus and Grafana stack, users gain real-time vis­i­bil­i­ty into the ap­pli­ca­tion’s per­for­mance and are able to re­spond quick­ly to in­ci­dents and op­ti­mize re­source us­age.

Fu­ture im­prove­ments like in­te­grat­ing dis­trib­uted trac­ing with tools like Jaeger, log ag­gre­ga­tion with tools like Loki, and man­ag­ing de­ploy­ments with Helm charts will fur­ther en­hance this ob­serv­abil­i­ty stack.
In con­clu­sion, this blog pin­point­ed the pow­er of com­bin­ing ap­pli­ca­tion met­rics with mon­i­tor­ing tools to build a strong foun­da­tion for un­der­stand­ing ba­sic mon­i­tor­ing prin­ci­ples. Con­tin­u­ous­ly track­ing met­rics, set­ting up ef­fec­tive alert­ing, and plan­ning for scal­ing un­der­scored the im­por­tance of hav­ing a mon­i­tor­ing and alert­ing sys­tem in place to en­sure the re­li­a­bil­i­ty, per­for­mance, and scal­a­bil­i­ty of ap­pli­ca­tion.