Sentry.io
Enterprise Account
The Ministry of Justice have an enterprise account with Sentry.io which is centrally managed and paid for via the Operations Engineering Team within Platforms and Architecture.
The purpose of this is to enable teams to have an easily accessible performance monitoring, alerting and error tracking tool for digital products.
Quota
The Ministry of Justices’ Sentry enterprise account reserves the following usage quotas (per month):
- 30 Million Transactions
- 4 Million Errors
- 1 Gb of Attachments
- 800000 Replays
The quota renews on the 14th of each month and is shared across all projects in Sentry.
Teams are responsible for actively managing their Sentry usage. We provide Quota Management guidance to ensure teams get the most value from Sentry.
Our Sentry plan also includes performance monitoring.
Error Spike Protection
The Operations Engineering team have enabled the Spike Protection feature at an organisational level.
Sentry uses an algorithm to keep track of normal error consumption per project. When an abnormal amount of errors occur within a project, Sentry dynamically rate limits the project accordingly.
The purpose of enabling Spike Protection is to prevent an abnormally high volume of errors unnecessarily consuming a large portion of the error quota that may have been caused by an intermittent network issue. This also helps in ensure protection of the error quota so all teams can make use of it.
User Access
All members of the MoJ GitHub Organization can create and sign in to their Sentry.io account using this SSO link.
Anyone in the MoJ Organisation that joins Sentry could potentially view the information sent to it. While this account is a presumed trusted group, it is still advisable to remove any Personal Identifiable Information (PII) from your application transactions and errors that are sent to Sentry. See Sentry’s Advanced Data Scrubbing documentation to redact sensitive information before it is saved in Sentry.
Quota Management
If a quota is exceeded then no further usage in that category is available until the next monthly renewal. For example, if the Error or Transaction quota is consumed then any further errors or transactions sent to Sentry are dropped. Similarly, if the Replay quota is consumed, no further replays are available. To prevent exceeding the quota, teams should actively manage their usage of Sentry to ensure they are getting the most value for their consumption and not unnecessarily impacting others. This section details some best practices.
Error Rate Limiting
Rate limiting is the best way to manage the error consumption of your project. The feature allows you to define how many errors can be sent to Sentry over a period of time. Essentially, enabling a safety net to ensure a predictable worse case scenario regarding excessive error consumption.
Rate limiting should be enabled on every project. Rate limiting should be applied to each DSN key. The Operations Engineering team monitor which projects have set up rate limiting enabled.
See the Sentry Rate limiting documentation to set up rate limiting for each project.
Also see the Setting Useful Rate Limits documentation to help create a suitable rate limit for your application.
Removing/Ignoring Errors
There will some errors generated by your application that you may be unwilling or unable to resolve. This can be due to a technical constraint or an agreement within the team to ignore the error.
For these type of errors, there’s typically little value in persistently recording the error in Sentry everytime it occurs.
Sentry allows you to Delete and Discard errors. Doing so will ensure these errors are not recorded by Sentry, allowing your team to have better visibility of new error within your application.
We recommend deleting and discarding these type of persistent errors. They can quickly consume large portions of the quota for high traffic applications.
You can find a list of deleted and discarded errors in the “Discarded Issues” tab in [Project] > Settings > Inbound Filters. From here, you can un-discarded any of these issues to receive future events.
Transaction Sampling
Transactions represent interactions with your application, such as users going to a /account endpoint. Recording
transactions are useful for knowing how your application behaves day to day and can create useful insights into how your
application is used.
Typically, there’s little value in recording all transactions of a user for analytical purposes. Transaction sampling is used to record a percentage of user activity. This helps reduce the cost of transaction monitoring whilst still being able to reap the benefits.
We recommend a transaction sampling rate of between 0.01 - 0.05 for most applications. Teams should review the value they get from Sentry transactions and adjust their sampling rates accordingly.
For example, if teams don’t make use of the feature you can disable it completely.
Alternatively, teams may want to temporarily increase sample rates of a new feature.
Transaction sampling can be implemented within the application code. See the Example Project Configurations for a recommended starting point in your SDK.
Ignoring Transactions on Administrative Endpoints
Most applications will contain administrative endpoints, which are intended to be used by other applications to check monitoring metrics such as health status.
There is little value in recording these transactions in Sentry, and they can very quickly consume large portion of the
transaction quota. For example, a /ping endpoint being called every 2 seconds by 3 application will generate
~4Million transactions a month consuming 10% of the quota.
Here are some examples of administrative
endpoints /ping,/ping.json,/health,/health.json,/status/**,/robots.txt
Ignoring specific endpoints (or drastically reducing their sampling rates) can be implemented within the application code. See the Example Project Configurations for a recommended starting point in your SDK.
Replay session sampling rate
Sampling is a Sentry.io configuration option that controls the rate at which sessions are collected for replay purposes. Session replay enables developers to see and understand the user interactions that led up to a mistake or issue, providing vital context for debugging and problem solutions.
By default, session replay is turned off or set to a low sample rate to conserve resources and prevent Sentry.io quota limits from being surpassed too rapidly. However, some users may be tempted to increase the replay session sample rate in order to collect more sessions for analysis.
There are two sample rates you can adjust to get the replays relevant to you:
replaysSessionSampleRate - The sample rate for replays that begin recording immediately and last the entirety of the user’s session.
replaysOnErrorSampleRate - The sample rate for replays that are recorded when an error happens. This type of replay will record up to a minute of events prior to the error and continue recording until the session ends.
We recommend setting a replay session sample rate of 0. Setting anything other than 0 will result in capture of replays for any session, not just errors. For high transaction services, this will result in quick consumption of the monthly quota and result in no further replays being recorded during that period.
Service Teams should use replaysOnErrorSampleRate to set an appropriate sample rate of replays. In line with vendor recommendations, this should be between 0.1 and 0.2 (i.e, 10% to 20% of errors).
Refer to Sentry.io configuration guidance for more detail.
While increasing the replay SessionSampleRate may appear to be a good way to learn more about user interactions, it might have serious consequences, particularly in terms of quota use. Enabling replays SessionSampleRate is not recommended for the following reasons
Ignore certain Errors for Error Sampling
In addition to setting sample rates, you can also use the beforeErrorSampling callback to exclude types of errors that you are not interested in. This will ensure that replays are not captured for these types of errors.
Replays and Privacy
Sentry’s Session Replay is not an actual “screen” recording, but instead a recording of changes in the web browser’s Document Object Model (DOM) (i.e., the data representation of the page). This important distinction means that the data captured by session Replay can be searched and stripped.
For full details of Privacy and additional configuration, please refer to Sentry.io Protecting User Privacy in Session Replay guidance.
Privacy Configuration
Sentry.io by default, captures all the data sent by your application, potentially including sensitive user information.
Sentry provides several features to control data access and privacy:
Data Scrubbing - Automatically removes sensitive information from error messages and other captured data before it’s stored.
Enhanced privacy mode - Limits the amount of data displayed in alerts and notifications, restricting access to sensitive details.
Permissions - Define granular permissions for user roles and teams within organisations, controlling access to specific data sets within the projects.
IP Address Management - Control whether IP addresses are captured and stored. Disabling IP address storage can be crucial for user privacy.
Full details can be found in the Privacy Configuration Overview.
Example Project Configurations
Please contact the Operation-Engineering team on Slack #ask-operations-engineering if you have any questions or need assistance in setting up transaction filtering / rate limiting within your Sentry project or your application.