diff --git a/CHANGELOG.md b/CHANGELOG.md index 648533ec18..dbf6673311 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -4,6 +4,7 @@ ### Features +- Add opt-in async event processing ([#5558](https://github.com/getsentry/sentry-java/pull/5558)) - Add `enableStandaloneAppStartTracing` option to send app start as a standalone transaction instead of attaching it as a child span of the first activity transaction ([#5342](https://github.com/getsentry/sentry-java/pull/5342)) - Disabled by default; opt in via `options.isEnableStandaloneAppStartTracing = true` or manifest meta-data `io.sentry.standalone-app-start-tracing.enable` - Emits a transaction named `App Start` with op `app.start`, carrying the existing app start measurements and phase spans (`process.load`, `contentprovider.load`, `application.load`, activity lifecycle spans) as direct children of the root diff --git a/docs/superpowers/specs/2026-06-15-async-event-processing-design.md b/docs/superpowers/specs/2026-06-15-async-event-processing-design.md new file mode 100644 index 0000000000..eb3e193659 --- /dev/null +++ b/docs/superpowers/specs/2026-06-15-async-event-processing-design.md @@ -0,0 +1,126 @@ +# Async Event Processing Design + +## Context + +The Java SDK currently invokes `EventProcessor.process(...)` synchronously in `SentryClient` before the relevant `beforeSend*` callback and before enqueueing work for transport or batching. This means custom processor and `beforeSend*` code can run on the caller thread. + +We want to introduce an opt-in mode that moves the late mutation/drop phase off the caller thread while keeping the existing synchronous processor phase in place. The first iteration should avoid Android public API compatibility risk from `CompletionStage`/`CompletableFuture`, so async processor methods use the same return shape as the existing synchronous methods. + +## Goals + +- Add async event processor callbacks for every event type currently supported by `EventProcessor`. +- Keep existing synchronous `process(...)` callbacks in their current positions. +- Add an opt-in `SentryOptions.enableAsyncProcessing` option, defaulting to `false`. +- When async processing is enabled, return from `capture*` after the item is accepted by the async processing queue. +- Isolate potentially slow user callbacks from the shared `SentryOptions.executorService`. +- Make `flush` and `close` wait for accepted async processing work. + +## Non-goals + +- Do not add `CompletionStage`, `CompletableFuture`, Kotlin coroutines, or other asynchronous return types to the public API. +- Do not change session, check-in, or profile chunk processing; these paths do not use `EventProcessor` today. +- Do not remove or move existing synchronous `process(...)` callbacks. +- Do not guarantee that a returned event ID means the item reaches Sentry. + +## Public API + +`EventProcessor` gets default `processAsync(...)` overloads that mirror the current `process(...)` overloads: + +- `SentryEvent processAsync(SentryEvent event, Hint hint)` +- `SentryTransaction processAsync(SentryTransaction transaction, Hint hint)` +- `SentryReplayEvent processAsync(SentryReplayEvent event, Hint hint)` +- `SentryLogEvent processAsync(SentryLogEvent event)` +- `SentryMetricsEvent processAsync(SentryMetricsEvent event, Hint hint)` + +Each default implementation returns the input item unchanged. Returning `null` drops the item, matching existing processor semantics. + +`SentryOptions` gets a direct boolean option: + +- `isEnableAsyncProcessing()` +- `setEnableAsyncProcessing(boolean enableAsyncProcessing)` + +The default is `false`. External configuration support should use the key `enable-async-processing`. + +## Runtime Architecture + +`SentryClient` owns a dedicated bounded single-thread async processing queue. The queue is separate from `SentryOptions.executorService` so user processor and `beforeSend*` code cannot clog SDK maintenance tasks. + +The async processing queue is active only when `enableAsyncProcessing=true`. Its capacity reuses `SentryOptions.maxQueueSize` to avoid adding a second queue-size option in the first iteration. + +When `enableAsyncProcessing=false`, `SentryClient` still invokes the new `processAsync(...)` stage, but inline on the caller thread. This keeps the processing pipeline shape consistent and avoids conditional logic around whether async processor callbacks are invoked. + +## Telemetry Processor Compatibility + +The async processing queue is a callback execution stage, not a delivery buffer or telemetry scheduler. It owns only running `processAsync(...)`, running the matching `beforeSend*` callback, and forwarding accepted items to the existing downstream delivery layer. + +Future telemetry processor integration should keep batching, prioritization, overflow policy, trace-aware span bucketing, and offline/cache behavior in the telemetry processor. Async processing should happen before telemetry buffer/scheduler ingestion, and future work may replace or reshape this queue to avoid keeping both a generic FIFO callback queue and category-aware telemetry buffers. + +The first iteration uses one bounded FIFO queue because the feature is opt-in. This can cause priority inversion under mixed high-volume traffic, for example logs or metrics filling the callback queue before higher-priority errors reach a future scheduler. Queue overflow at this stage is recorded as `queue_overflow`; future telemetry buffers should use a distinct `buffer_overflow` reason. + +`flush(timeoutMillis)` and `close()` must treat async processing and downstream queues as one pipeline and use a shared deadline instead of giving every layer the full timeout independently. + +## Processing Pipeline + +For each supported capture path, the order is: + +1. Existing early filtering and scope work. +2. Existing scoped synchronous `process(...)`, where applicable. +3. Existing global synchronous `process(...)`. +4. New scoped `processAsync(...)`, where applicable. +5. New global `processAsync(...)`. +6. Matching `beforeSend*` callback. +7. Existing send, log batch, or metrics batch queue. + +Supported paths: + +- Error events: async processors run before `beforeSend`. +- Feedback events: async processors run before `beforeSendFeedback`. +- Transactions: async processors run before `beforeSendTransaction`. +- Replay events: async processors run before `beforeSendReplay`. +- Logs: async processors run before logs `beforeSend`. +- Metrics: async processors run before metrics `beforeSend`. + +Sessions, check-ins, and profile chunks are unchanged. + +## Capture Return Behavior + +When `enableAsyncProcessing=true`, ID-returning capture methods enqueue the late processing task and return immediately if enqueue succeeds. They return the event, transaction, replay, or feedback ID even though the item may later be dropped by async processors, `beforeSend*`, sampling, rate limiting, transport failures, or downstream queue overflow. + +If the async processing queue is full, enqueue fails immediately. The SDK drops the item, records `DiscardReason.QUEUE_OVERFLOW`, and returns `SentryId.EMPTY_ID` for ID-returning capture methods. + +Log and metrics capture methods do not return an ID. If the async processing queue is full, they drop the item and record the appropriate queue-overflow client report. + +## Error Handling and Client Reports + +`processAsync(...)` uses the same error policy as `process(...)`: + +- Exceptions from processors are logged. +- Processing continues with the current item. +- Returning `null` drops the item. +- Drops are recorded with `DiscardReason.EVENT_PROCESSOR` and the existing data category for the item type. + +`beforeSend*` keeps its current behavior. Exceptions drop the item for PII safety and record `DiscardReason.BEFORE_SEND` where applicable. + +Transaction accounting must remain consistent: + +- Dropping a transaction records the transaction and all spans. +- Removing spans from a transaction in an async processor or `beforeSendTransaction` records the dropped span count. + +## Flush and Close + +`SentryClient.flush(timeoutMillis)` waits for the async processing queue to become idle and then waits for downstream log, metrics, and transport queues. + +`SentryClient.close()` drains async processing during shutdown and then closes downstream processors and transport. Closing remains bounded by existing shutdown and flush timeout behavior. + +## Testing Plan + +- Verify `EventProcessor.processAsync(...)` default methods return input items unchanged. +- Verify `SentryOptions.enableAsyncProcessing` defaults to `false` and can be set directly and through external options. +- Verify inline mode invokes `processAsync(...)` on the caller thread before the matching `beforeSend*` callback. +- Verify async-enabled mode returns before `processAsync(...)` and `beforeSend*` finish. +- Verify ordering: scoped sync processor, global sync processor, scoped async processor, global async processor, `beforeSend*`. +- Verify async queue overflow drops the item, records `queue_overflow`, and returns `SentryId.EMPTY_ID` where applicable. +- Verify `flush` waits for accepted async processing work. +- Verify processor exceptions are logged and do not drop the item. +- Verify `processAsync(...)` returning `null` records `event_processor` drops for events, feedback, transactions and spans, replay, logs, and metrics. +- Verify `beforeSendTransaction` and async transaction processors preserve existing span-loss accounting. diff --git a/sentry/api/sentry.api b/sentry/api/sentry.api index 4757be4894..ab76964a7a 100644 --- a/sentry/api/sentry.api +++ b/sentry/api/sentry.api @@ -474,6 +474,11 @@ public abstract interface class io/sentry/EventProcessor { public fun process (Lio/sentry/SentryMetricsEvent;Lio/sentry/Hint;)Lio/sentry/SentryMetricsEvent; public fun process (Lio/sentry/SentryReplayEvent;Lio/sentry/Hint;)Lio/sentry/SentryReplayEvent; public fun process (Lio/sentry/protocol/SentryTransaction;Lio/sentry/Hint;)Lio/sentry/protocol/SentryTransaction; + public fun processAsync (Lio/sentry/SentryEvent;Lio/sentry/Hint;)Lio/sentry/SentryEvent; + public fun processAsync (Lio/sentry/SentryLogEvent;)Lio/sentry/SentryLogEvent; + public fun processAsync (Lio/sentry/SentryMetricsEvent;Lio/sentry/Hint;)Lio/sentry/SentryMetricsEvent; + public fun processAsync (Lio/sentry/SentryReplayEvent;Lio/sentry/Hint;)Lio/sentry/SentryReplayEvent; + public fun processAsync (Lio/sentry/protocol/SentryTransaction;Lio/sentry/Hint;)Lio/sentry/protocol/SentryTransaction; } public final class io/sentry/ExperimentalOptions { @@ -525,6 +530,7 @@ public final class io/sentry/ExternalOptions { public fun getTracePropagationTargets ()Ljava/util/List; public fun getTracesSampleRate ()Ljava/lang/Double; public fun isCaptureOpenTelemetryEvents ()Ljava/lang/Boolean; + public fun isEnableAsyncProcessing ()Ljava/lang/Boolean; public fun isEnableBackpressureHandling ()Ljava/lang/Boolean; public fun isEnableCacheTracing ()Ljava/lang/Boolean; public fun isEnableDatabaseTransactionTracing ()Ljava/lang/Boolean; @@ -544,6 +550,7 @@ public final class io/sentry/ExternalOptions { public fun setDebug (Ljava/lang/Boolean;)V public fun setDist (Ljava/lang/String;)V public fun setDsn (Ljava/lang/String;)V + public fun setEnableAsyncProcessing (Ljava/lang/Boolean;)V public fun setEnableBackpressureHandling (Ljava/lang/Boolean;)V public fun setEnableCacheTracing (Ljava/lang/Boolean;)V public fun setEnableDatabaseTransactionTracing (Ljava/lang/Boolean;)V @@ -3717,6 +3724,7 @@ public class io/sentry/SentryOptions { public fun isContinuousProfilingEnabled ()Z public fun isDebug ()Z public fun isEnableAppStartProfiling ()Z + public fun isEnableAsyncProcessing ()Z public fun isEnableAutoSessionTracking ()Z public fun isEnableBackpressureHandling ()Z public fun isEnableCacheTracing ()Z @@ -3778,6 +3786,7 @@ public class io/sentry/SentryOptions { public fun setDistributionController (Lio/sentry/IDistributionApi;)V public fun setDsn (Ljava/lang/String;)V public fun setEnableAppStartProfiling (Z)V + public fun setEnableAsyncProcessing (Z)V public fun setEnableAutoSessionTracking (Z)V public fun setEnableBackpressureHandling (Z)V public fun setEnableCacheTracing (Z)V diff --git a/sentry/src/main/java/io/sentry/AsyncEventProcessingExecutor.java b/sentry/src/main/java/io/sentry/AsyncEventProcessingExecutor.java new file mode 100644 index 0000000000..f4f5eedcb2 --- /dev/null +++ b/sentry/src/main/java/io/sentry/AsyncEventProcessingExecutor.java @@ -0,0 +1,87 @@ +package io.sentry; + +import io.sentry.transport.ReusableCountLatch; +import java.util.concurrent.LinkedBlockingQueue; +import java.util.concurrent.ThreadFactory; +import java.util.concurrent.ThreadPoolExecutor; +import java.util.concurrent.TimeUnit; +import org.jetbrains.annotations.ApiStatus; +import org.jetbrains.annotations.NotNull; + +@ApiStatus.Internal +final class AsyncEventProcessingExecutor { + private final int maxQueueSize; + private final @NotNull ILogger logger; + private final @NotNull ThreadPoolExecutor executor; + private final @NotNull ReusableCountLatch unfinishedTasksCount = new ReusableCountLatch(); + private boolean closed = false; + + AsyncEventProcessingExecutor(final int maxQueueSize, final @NotNull ILogger logger) { + this.maxQueueSize = maxQueueSize; + this.logger = logger; + this.executor = + new ThreadPoolExecutor( + 1, + 1, + 0L, + TimeUnit.MILLISECONDS, + new LinkedBlockingQueue<>(), + new AsyncEventProcessingThreadFactory()); + } + + synchronized boolean submit(final @NotNull Runnable task) { + if (closed || unfinishedTasksCount.getCount() >= maxQueueSize) { + return false; + } + + unfinishedTasksCount.increment(); + try { + executor.execute( + () -> { + try { + task.run(); + } finally { + unfinishedTasksCount.decrement(); + } + }); + return true; + } catch (Throwable e) { + unfinishedTasksCount.decrement(); + logger.log(SentryLevel.WARNING, "Async event processing task rejected.", e); + return false; + } + } + + void waitTillIdle(final long timeoutMillis) { + try { + unfinishedTasksCount.waitTillZero(timeoutMillis, TimeUnit.MILLISECONDS); + } catch (InterruptedException e) { + logger.log(SentryLevel.ERROR, "Failed to wait for async event processing queue to drain.", e); + Thread.currentThread().interrupt(); + } + } + + synchronized void close(final long timeoutMillis) { + closed = true; + executor.shutdown(); + try { + if (!executor.awaitTermination(timeoutMillis, TimeUnit.MILLISECONDS)) { + executor.shutdownNow(); + } + } catch (InterruptedException e) { + executor.shutdownNow(); + Thread.currentThread().interrupt(); + } + } + + private static final class AsyncEventProcessingThreadFactory implements ThreadFactory { + private int cnt; + + @Override + public @NotNull Thread newThread(final @NotNull Runnable r) { + final Thread thread = new Thread(r, "SentryAsyncEventProcessing-" + cnt++); + thread.setDaemon(true); + return thread; + } + } +} diff --git a/sentry/src/main/java/io/sentry/EventProcessor.java b/sentry/src/main/java/io/sentry/EventProcessor.java index 928fd02209..3ab654ef82 100644 --- a/sentry/src/main/java/io/sentry/EventProcessor.java +++ b/sentry/src/main/java/io/sentry/EventProcessor.java @@ -67,6 +67,66 @@ default SentryMetricsEvent process(@NotNull SentryMetricsEvent event, @NotNull H return event; } + /** + * May mutate or drop a SentryEvent during the async processing stage. + * + * @param event the SentryEvent + * @param hint the Hint + * @return the event itself, a mutated SentryEvent or null + */ + @Nullable + default SentryEvent processAsync(@NotNull SentryEvent event, @NotNull Hint hint) { + return event; + } + + /** + * May mutate or drop a SentryTransaction during the async processing stage. + * + * @param transaction the SentryTransaction + * @param hint the Hint + * @return the event itself, a mutated SentryTransaction or null + */ + @Nullable + default SentryTransaction processAsync( + @NotNull SentryTransaction transaction, @NotNull Hint hint) { + return transaction; + } + + /** + * May mutate or drop a SentryReplayEvent during the async processing stage. + * + * @param event the SentryReplayEvent + * @param hint the Hint + * @return the event itself, a mutated SentryReplayEvent or null + */ + @Nullable + default SentryReplayEvent processAsync(@NotNull SentryReplayEvent event, @NotNull Hint hint) { + return event; + } + + /** + * May mutate or drop a SentryLogEvent during the async processing stage. + * + * @param event the SentryLogEvent + * @return the event itself, a mutated SentryLogEvent or null + */ + @Nullable + default SentryLogEvent processAsync(@NotNull SentryLogEvent event) { + return event; + } + + /** + * May mutate or drop a SentryMetricsEvent during the async processing stage. + * + * @param event the SentryMetricsEvent + * @param hint the Hint + * @return the event itself, a mutated SentryMetricsEvent or null + */ + @Nullable + default SentryMetricsEvent processAsync(@NotNull SentryMetricsEvent event, @NotNull Hint hint) { + return event; + } + /** * Controls when this EventProcessor is invoked. * diff --git a/sentry/src/main/java/io/sentry/ExternalOptions.java b/sentry/src/main/java/io/sentry/ExternalOptions.java index 4e44ea422e..159548ac56 100644 --- a/sentry/src/main/java/io/sentry/ExternalOptions.java +++ b/sentry/src/main/java/io/sentry/ExternalOptions.java @@ -23,6 +23,7 @@ public final class ExternalOptions { private @Nullable Boolean enableUncaughtExceptionHandler; private @Nullable Boolean debug; private @Nullable Boolean enableDeduplication; + private @Nullable Boolean enableAsyncProcessing; private @Nullable Double sampleRate; private @Nullable Double tracesSampleRate; private @Nullable Double profilesSampleRate; @@ -90,6 +91,8 @@ public final class ExternalOptions { options.setProfilesSampleRate(propertiesProvider.getDoubleProperty("profiles-sample-rate")); options.setDebug(propertiesProvider.getBooleanProperty("debug")); options.setEnableDeduplication(propertiesProvider.getBooleanProperty("enable-deduplication")); + options.setEnableAsyncProcessing( + propertiesProvider.getBooleanProperty("enable-async-processing")); options.setSendClientReports(propertiesProvider.getBooleanProperty("send-client-reports")); options.setForceInit(propertiesProvider.getBooleanProperty("force-init")); final String maxRequestBodySize = propertiesProvider.getProperty("max-request-body-size"); @@ -315,6 +318,14 @@ public void setEnableDeduplication(final @Nullable Boolean enableDeduplication) this.enableDeduplication = enableDeduplication; } + public @Nullable Boolean isEnableAsyncProcessing() { + return enableAsyncProcessing; + } + + public void setEnableAsyncProcessing(final @Nullable Boolean enableAsyncProcessing) { + this.enableAsyncProcessing = enableAsyncProcessing; + } + public @Nullable Double getSampleRate() { return sampleRate; } diff --git a/sentry/src/main/java/io/sentry/SentryClient.java b/sentry/src/main/java/io/sentry/SentryClient.java index 5ac81c4493..470358985a 100644 --- a/sentry/src/main/java/io/sentry/SentryClient.java +++ b/sentry/src/main/java/io/sentry/SentryClient.java @@ -45,6 +45,7 @@ public final class SentryClient implements ISentryClient { private final @NotNull SortBreadcrumbsByDate sortBreadcrumbsByDate = new SortBreadcrumbsByDate(); private final @NotNull ILoggerBatchProcessor loggerBatchProcessor; private final @NotNull IMetricsBatchProcessor metricsBatchProcessor; + private final @NotNull AsyncEventProcessingExecutor asyncEventProcessingExecutor; @Override public boolean isEnabled() { @@ -75,6 +76,8 @@ public SentryClient(final @NotNull SentryOptions options) { } else { metricsBatchProcessor = NoOpMetricsBatchProcessor.getInstance(); } + asyncEventProcessingExecutor = + new AsyncEventProcessingExecutor(options.getMaxQueueSize(), options.getLogger()); } private boolean shouldApplyScopeData( @@ -162,6 +165,41 @@ private boolean shouldApplyScopeData(final @NotNull CheckIn event, final @NotNul event = processEvent(event, hint, options.getEventProcessors()); + if (event == null) { + return SentryId.EMPTY_ID; + } + + final SentryId sentryId = event.getEventId() != null ? event.getEventId() : SentryId.EMPTY_ID; + final @NotNull SentryEvent finalEvent = event; + final @NotNull Hint finalHint = hint; + if (options.isEnableAsyncProcessing()) { + final boolean applyScopedProcessors = HintUtils.shouldApplyScopeData(finalHint); + if (!asyncEventProcessingExecutor.submit( + () -> captureEventAfterProcessing(finalEvent, finalHint, scope, applyScopedProcessors))) { + options + .getClientReportRecorder() + .recordLostEvent(DiscardReason.QUEUE_OVERFLOW, DataCategory.Error); + return SentryId.EMPTY_ID; + } + return sentryId; + } + + return captureEventAfterProcessing(event, hint, scope, HintUtils.shouldApplyScopeData(hint)); + } + + private @NotNull SentryId captureEventAfterProcessing( + @NotNull SentryEvent event, + final @NotNull Hint hint, + final @Nullable IScope scope, + final boolean applyScopedProcessors) { + if (applyScopedProcessors && scope != null) { + event = processEventAsync(event, hint, scope.getEventProcessors(), DataCategory.Error); + } + + if (event != null) { + event = processEventAsync(event, hint, options.getEventProcessors(), DataCategory.Error); + } + if (event != null) { event = executeBeforeSend(event, hint); @@ -177,10 +215,6 @@ private boolean shouldApplyScopeData(final @NotNull CheckIn event, final @NotNul event = EventSizeLimitingUtils.limitEventSize(event, hint, options); } - if (event == null) { - return SentryId.EMPTY_ID; - } - @Nullable Session sessionBeforeUpdate = scope != null ? scope.withSession((@Nullable Session session) -> {}) : null; @@ -316,6 +350,30 @@ private void finalizeTransaction(final @NotNull IScope scope, final @NotNull Hin event = processReplayEvent(event, hint, options.getEventProcessors()); + if (event == null) { + return SentryId.EMPTY_ID; + } + + final @NotNull SentryReplayEvent finalEvent = event; + final @NotNull Hint finalHint = hint; + if (options.isEnableAsyncProcessing()) { + if (!asyncEventProcessingExecutor.submit( + () -> captureReplayEventAfterProcessing(finalEvent, scope, finalHint))) { + options + .getClientReportRecorder() + .recordLostEvent(DiscardReason.QUEUE_OVERFLOW, DataCategory.Replay); + return SentryId.EMPTY_ID; + } + return sentryId; + } + + return captureReplayEventAfterProcessing(event, scope, hint); + } + + private @NotNull SentryId captureReplayEventAfterProcessing( + @NotNull SentryReplayEvent event, final @Nullable IScope scope, final @NotNull Hint hint) { + event = processReplayEventAsync(event, hint, options.getEventProcessors()); + if (event != null) { event = executeBeforeSendReplay(event, hint); @@ -331,6 +389,7 @@ private void finalizeTransaction(final @NotNull IScope scope, final @NotNull Hin return SentryId.EMPTY_ID; } + SentryId sentryId = event.getEventId() != null ? event.getEventId() : SentryId.EMPTY_ID; try { final @Nullable TraceContext traceContext = getTraceContext(scope, hint, event, null); final boolean cleanupReplayFolder = HintUtils.hasType(hint, Backfillable.class); @@ -693,6 +752,224 @@ private SentryEvent processFeedbackEvent( return feedbackEvent; } + @Nullable + private SentryEvent processEventAsync( + @NotNull SentryEvent event, + final @NotNull Hint hint, + final @NotNull List eventProcessors, + final @NotNull DataCategory dataCategory) { + for (final EventProcessor processor : eventProcessors) { + try { + final boolean isBackfillingProcessor = processor instanceof BackfillingEventProcessor; + final boolean isBackfillable = HintUtils.hasType(hint, Backfillable.class); + if (isBackfillable && isBackfillingProcessor) { + event = processor.processAsync(event, hint); + } else if (!isBackfillable && !isBackfillingProcessor) { + event = processor.processAsync(event, hint); + } + } catch (Throwable e) { + options + .getLogger() + .log( + SentryLevel.ERROR, + e, + "An exception occurred while async processing event by processor: %s", + processor.getClass().getName()); + } + + if (event == null) { + options + .getLogger() + .log( + SentryLevel.DEBUG, + "Event was dropped by an async processor: %s", + processor.getClass().getName()); + options + .getClientReportRecorder() + .recordLostEvent(DiscardReason.EVENT_PROCESSOR, dataCategory); + break; + } + } + return event; + } + + @Nullable + private SentryLogEvent processLogEventAsync( + @NotNull SentryLogEvent event, final @NotNull List eventProcessors) { + for (final EventProcessor processor : eventProcessors) { + try { + event = processor.processAsync(event); + } catch (Throwable e) { + options + .getLogger() + .log( + SentryLevel.ERROR, + e, + "An exception occurred while async processing log event by processor: %s", + processor.getClass().getName()); + } + + if (event == null) { + options + .getLogger() + .log( + SentryLevel.DEBUG, + "Log event was dropped by an async processor: %s", + processor.getClass().getName()); + options + .getClientReportRecorder() + .recordLostEvent(DiscardReason.EVENT_PROCESSOR, DataCategory.LogItem); + break; + } + } + return event; + } + + @Nullable + private SentryMetricsEvent processMetricsEventAsync( + @NotNull SentryMetricsEvent event, + final @NotNull List eventProcessors, + final @NotNull Hint hint) { + for (final EventProcessor processor : eventProcessors) { + try { + event = processor.processAsync(event, hint); + } catch (Throwable e) { + options + .getLogger() + .log( + SentryLevel.ERROR, + e, + "An exception occurred while async processing metrics event by processor: %s", + processor.getClass().getName()); + } + + if (event == null) { + options + .getLogger() + .log( + SentryLevel.DEBUG, + "Metrics event was dropped by an async processor: %s", + processor.getClass().getName()); + options + .getClientReportRecorder() + .recordLostEvent(DiscardReason.EVENT_PROCESSOR, DataCategory.TraceMetric); + break; + } + } + return event; + } + + private @Nullable SentryTransaction processTransactionAsync( + @NotNull SentryTransaction transaction, + final @NotNull Hint hint, + final @NotNull List eventProcessors) { + for (final EventProcessor processor : eventProcessors) { + final int spanCountBeforeProcessor = transaction.getSpans().size(); + try { + transaction = processor.processAsync(transaction, hint); + } catch (Throwable e) { + options + .getLogger() + .log( + SentryLevel.ERROR, + e, + "An exception occurred while async processing transaction by processor: %s", + processor.getClass().getName()); + } + final int spanCountAfterProcessor = transaction == null ? 0 : transaction.getSpans().size(); + + if (transaction == null) { + options + .getLogger() + .log( + SentryLevel.DEBUG, + "Transaction was dropped by an async processor: %s", + processor.getClass().getName()); + options + .getClientReportRecorder() + .recordLostEvent(DiscardReason.EVENT_PROCESSOR, DataCategory.Transaction); + options + .getClientReportRecorder() + .recordLostEvent( + DiscardReason.EVENT_PROCESSOR, DataCategory.Span, spanCountBeforeProcessor + 1); + break; + } else if (spanCountAfterProcessor < spanCountBeforeProcessor) { + final int droppedSpanCount = spanCountBeforeProcessor - spanCountAfterProcessor; + options + .getLogger() + .log( + SentryLevel.DEBUG, + "%d spans were dropped by an async processor: %s", + droppedSpanCount, + processor.getClass().getName()); + options + .getClientReportRecorder() + .recordLostEvent(DiscardReason.EVENT_PROCESSOR, DataCategory.Span, droppedSpanCount); + } + } + return transaction; + } + + @Nullable + private SentryReplayEvent processReplayEventAsync( + @NotNull SentryReplayEvent replayEvent, + final @NotNull Hint hint, + final @NotNull List eventProcessors) { + for (final EventProcessor processor : eventProcessors) { + try { + replayEvent = processor.processAsync(replayEvent, hint); + } catch (Throwable e) { + options + .getLogger() + .log( + SentryLevel.ERROR, + e, + "An exception occurred while async processing replay event by processor: %s", + processor.getClass().getName()); + } + + if (replayEvent == null) { + options + .getLogger() + .log( + SentryLevel.DEBUG, + "Replay event was dropped by an async processor: %s", + processor.getClass().getName()); + options + .getClientReportRecorder() + .recordLostEvent(DiscardReason.EVENT_PROCESSOR, DataCategory.Replay); + break; + } + } + return replayEvent; + } + + @Nullable + private SentryEvent processFeedbackEventAsync( + @NotNull SentryEvent feedbackEvent, + final @NotNull Hint hint, + final @NotNull List eventProcessors) { + return processEventAsync(feedbackEvent, hint, eventProcessors, DataCategory.Feedback); + } + + private void recordLostTransaction( + final @NotNull DiscardReason discardReason, final @NotNull SentryTransaction transaction) { + options.getClientReportRecorder().recordLostEvent(discardReason, DataCategory.Transaction); + options + .getClientReportRecorder() + .recordLostEvent(discardReason, DataCategory.Span, transaction.getSpans().size() + 1); + } + + private void recordLostLog( + final @NotNull DiscardReason discardReason, final @NotNull SentryLogEvent logEvent) { + options.getClientReportRecorder().recordLostEvent(discardReason, DataCategory.LogItem); + final long logEventNumberOfBytes = + JsonSerializationUtils.byteSizeOf(options.getSerializer(), options.getLogger(), logEvent); + options + .getClientReportRecorder() + .recordLostEvent(discardReason, DataCategory.LogByte, logEventNumberOfBytes); + } + @Deprecated @Override public void captureUserFeedback(final @NotNull UserFeedback userFeedback) { @@ -993,6 +1270,54 @@ public void captureSession(final @NotNull Session session, final @Nullable Hint return SentryId.EMPTY_ID; } + final @NotNull SentryTransaction finalTransaction = transaction; + final @NotNull Hint finalHint = hint; + final @Nullable TraceContext finalTraceContext = traceContext; + if (options.isEnableAsyncProcessing()) { + if (!asyncEventProcessingExecutor.submit( + () -> + captureTransactionAfterProcessing( + finalTransaction, + finalTraceContext, + scope, + finalHint, + profilingTraceData, + HintUtils.shouldApplyScopeData(finalHint)))) { + recordLostTransaction(DiscardReason.QUEUE_OVERFLOW, transaction); + return SentryId.EMPTY_ID; + } + return sentryId; + } + + return captureTransactionAfterProcessing( + transaction, + traceContext, + scope, + hint, + profilingTraceData, + HintUtils.shouldApplyScopeData(hint)); + } + + private @NotNull SentryId captureTransactionAfterProcessing( + @NotNull SentryTransaction transaction, + final @Nullable TraceContext traceContext, + final @Nullable IScope scope, + final @NotNull Hint hint, + final @Nullable ProfilingTraceData profilingTraceData, + final boolean applyScopedProcessors) { + if (applyScopedProcessors && scope != null) { + transaction = processTransactionAsync(transaction, hint, scope.getEventProcessors()); + } + + if (transaction != null) { + transaction = processTransactionAsync(transaction, hint, options.getEventProcessors()); + } + + if (transaction == null) { + options.getLogger().log(SentryLevel.DEBUG, "Transaction was dropped by Event processors."); + return SentryId.EMPTY_ID; + } + final int spanCountBeforeCallback = transaction.getSpans().size(); transaction = executeBeforeSendTransaction(transaction, hint); final int spanCountAfterCallback = transaction == null ? 0 : transaction.getSpans().size(); @@ -1024,6 +1349,8 @@ public void captureSession(final @NotNull Session session, final @Nullable Hint .recordLostEvent(DiscardReason.BEFORE_SEND, DataCategory.Span, droppedSpanCount); } + SentryId sentryId = + transaction.getEventId() != null ? transaction.getEventId() : SentryId.EMPTY_ID; try { final SentryEnvelope envelope = buildEnvelope( @@ -1179,6 +1506,45 @@ public void captureSession(final @NotNull Session session, final @Nullable Hint event = processFeedbackEvent(event, hint, options.getEventProcessors()); + if (event == null) { + return SentryId.EMPTY_ID; + } + + final SentryId sentryId = event.getEventId() != null ? event.getEventId() : SentryId.EMPTY_ID; + final @NotNull SentryEvent finalEvent = event; + final @NotNull Hint finalHint = hint; + if (options.isEnableAsyncProcessing()) { + final boolean applyScopedProcessors = HintUtils.shouldApplyScopeData(finalHint); + if (!asyncEventProcessingExecutor.submit( + () -> + captureFeedbackAfterProcessing( + finalEvent, feedback, finalHint, scope, applyScopedProcessors))) { + options + .getClientReportRecorder() + .recordLostEvent(DiscardReason.QUEUE_OVERFLOW, DataCategory.Feedback); + return SentryId.EMPTY_ID; + } + return sentryId; + } + + return captureFeedbackAfterProcessing( + event, feedback, hint, scope, HintUtils.shouldApplyScopeData(hint)); + } + + private @NotNull SentryId captureFeedbackAfterProcessing( + @NotNull SentryEvent event, + final @NotNull Feedback feedback, + final @NotNull Hint hint, + final @NotNull IScope scope, + final boolean applyScopedProcessors) { + if (applyScopedProcessors) { + event = processFeedbackEventAsync(event, hint, scope.getEventProcessors()); + } + + if (event != null) { + event = processFeedbackEventAsync(event, hint, options.getEventProcessors()); + } + if (event != null) { event = executeBeforeSendFeedback(event, hint); @@ -1194,10 +1560,7 @@ public void captureSession(final @NotNull Session session, final @Nullable Hint return SentryId.EMPTY_ID; } - SentryId sentryId = SentryId.EMPTY_ID; - if (event.getEventId() != null) { - sentryId = event.getEventId(); - } + SentryId sentryId = event.getEventId() != null ? event.getEventId() : SentryId.EMPTY_ID; // If feedback already has a replayId, we don't want to overwrite it. if (feedback.getReplayId() == null) { @@ -1276,22 +1639,37 @@ public void captureLog(@Nullable SentryLogEvent logEvent, @Nullable IScope scope } } + if (logEvent != null) { + final @NotNull SentryLogEvent finalLogEvent = logEvent; + if (options.isEnableAsyncProcessing()) { + if (!asyncEventProcessingExecutor.submit( + () -> captureLogAfterProcessing(finalLogEvent, scope))) { + recordLostLog(DiscardReason.QUEUE_OVERFLOW, finalLogEvent); + } + return; + } + + captureLogAfterProcessing(logEvent, scope); + } + } + + private void captureLogAfterProcessing( + @NotNull SentryLogEvent logEvent, final @Nullable IScope scope) { + if (scope != null) { + logEvent = processLogEventAsync(logEvent, scope.getEventProcessors()); + } + + if (logEvent != null) { + logEvent = processLogEventAsync(logEvent, options.getEventProcessors()); + } + if (logEvent != null) { final @NotNull SentryLogEvent tmpLogEvent = logEvent; logEvent = executeBeforeSendLog(logEvent); if (logEvent == null) { options.getLogger().log(SentryLevel.DEBUG, "Log Event was dropped by beforeSendLog"); - options - .getClientReportRecorder() - .recordLostEvent(DiscardReason.BEFORE_SEND, DataCategory.LogItem); - final @NotNull long logEventNumberOfBytes = - JsonSerializationUtils.byteSizeOf( - options.getSerializer(), options.getLogger(), tmpLogEvent); - options - .getClientReportRecorder() - .recordLostEvent( - DiscardReason.BEFORE_SEND, DataCategory.LogByte, logEventNumberOfBytes); + recordLostLog(DiscardReason.BEFORE_SEND, tmpLogEvent); return; } @@ -1334,6 +1712,35 @@ public void captureMetric( } } + if (metricsEvent != null) { + final @NotNull SentryMetricsEvent finalMetricsEvent = metricsEvent; + final @NotNull Hint finalHint = hint; + if (options.isEnableAsyncProcessing()) { + if (!asyncEventProcessingExecutor.submit( + () -> captureMetricAfterProcessing(finalMetricsEvent, scope, finalHint))) { + options + .getClientReportRecorder() + .recordLostEvent(DiscardReason.QUEUE_OVERFLOW, DataCategory.TraceMetric); + } + return; + } + + captureMetricAfterProcessing(metricsEvent, scope, hint); + } + } + + private void captureMetricAfterProcessing( + @NotNull SentryMetricsEvent metricsEvent, + final @Nullable IScope scope, + final @NotNull Hint hint) { + if (scope != null) { + metricsEvent = processMetricsEventAsync(metricsEvent, scope.getEventProcessors(), hint); + } + + if (metricsEvent != null) { + metricsEvent = processMetricsEventAsync(metricsEvent, options.getEventProcessors(), hint); + } + if (metricsEvent != null) { metricsEvent = executeBeforeSendMetric(metricsEvent, hint); @@ -1697,7 +2104,10 @@ public void close() { public void close(final boolean isRestarting) { options.getLogger().log(SentryLevel.INFO, "Closing SentryClient."); try { - flush(isRestarting ? 0 : options.getShutdownTimeoutMillis()); + final long timeoutMillis = isRestarting ? 0 : options.getShutdownTimeoutMillis(); + final long deadline = System.currentTimeMillis() + timeoutMillis; + flush(timeoutMillis); + asyncEventProcessingExecutor.close(remainingTimeout(deadline)); loggerBatchProcessor.close(isRestarting); metricsBatchProcessor.close(isRestarting); transport.close(isRestarting); @@ -1726,9 +2136,15 @@ public void close(final boolean isRestarting) { @Override public void flush(final long timeoutMillis) { - loggerBatchProcessor.flush(timeoutMillis); - metricsBatchProcessor.flush(timeoutMillis); - transport.flush(timeoutMillis); + final long deadline = System.currentTimeMillis() + timeoutMillis; + asyncEventProcessingExecutor.waitTillIdle(timeoutMillis); + loggerBatchProcessor.flush(remainingTimeout(deadline)); + metricsBatchProcessor.flush(remainingTimeout(deadline)); + transport.flush(remainingTimeout(deadline)); + } + + private long remainingTimeout(final long deadline) { + return Math.max(0, deadline - System.currentTimeMillis()); } @Override diff --git a/sentry/src/main/java/io/sentry/SentryOptions.java b/sentry/src/main/java/io/sentry/SentryOptions.java index 0d038482d0..7fb0791fc8 100644 --- a/sentry/src/main/java/io/sentry/SentryOptions.java +++ b/sentry/src/main/java/io/sentry/SentryOptions.java @@ -365,6 +365,9 @@ public class SentryOptions { */ private boolean enableDeduplication = true; + /** Whether EventProcessor.processAsync and beforeSend callbacks run off the caller thread. */ + private boolean enableAsyncProcessing = false; + /** * Enables event size limiting with {@link EventSizeLimitingEventProcessor}. When enabled, events * exceeding 1MB will have breadcrumbs and stack frames reduced to stay under the limit. @@ -1827,6 +1830,24 @@ public void setEnableDeduplication(final boolean enableDeduplication) { this.enableDeduplication = enableDeduplication; } + /** + * Returns whether async event processing is enabled. + * + * @return true if async event processing is enabled, false otherwise + */ + public boolean isEnableAsyncProcessing() { + return enableAsyncProcessing; + } + + /** + * Enables or disables async event processing. + * + * @param enableAsyncProcessing true if enabled, false otherwise + */ + public void setEnableAsyncProcessing(final boolean enableAsyncProcessing) { + this.enableAsyncProcessing = enableAsyncProcessing; + } + /** * Returns if event size limiting is enabled. * @@ -3490,6 +3511,9 @@ public void merge(final @NotNull ExternalOptions options) { if (options.getEnableDeduplication() != null) { setEnableDeduplication(options.getEnableDeduplication()); } + if (options.isEnableAsyncProcessing() != null) { + setEnableAsyncProcessing(options.isEnableAsyncProcessing()); + } if (options.getSendClientReports() != null) { setSendClientReports(options.getSendClientReports()); } diff --git a/sentry/src/test/java/io/sentry/ExternalOptionsTest.kt b/sentry/src/test/java/io/sentry/ExternalOptionsTest.kt index fee707d31f..eeb1169a27 100644 --- a/sentry/src/test/java/io/sentry/ExternalOptionsTest.kt +++ b/sentry/src/test/java/io/sentry/ExternalOptionsTest.kt @@ -123,6 +123,18 @@ class ExternalOptionsTest { } } + @Test + fun `creates options with enableAsyncProcessing using external properties`() { + withPropertiesFile("enable-async-processing=true") { + assertNotNull(it.isEnableAsyncProcessing) { assertTrue(it) } + } + } + + @Test + fun `creates options with enableAsyncProcessing set to null when not set`() { + withPropertiesFile { assertNull(it.isEnableAsyncProcessing) } + } + @Test fun `creates options with sendClientReports using external properties`() { withPropertiesFile("send-client-reports=false") { diff --git a/sentry/src/test/java/io/sentry/SentryClientTest.kt b/sentry/src/test/java/io/sentry/SentryClientTest.kt index d5b2f0f82a..bc3056c7d2 100644 --- a/sentry/src/test/java/io/sentry/SentryClientTest.kt +++ b/sentry/src/test/java/io/sentry/SentryClientTest.kt @@ -43,6 +43,8 @@ import java.nio.file.Files import java.util.Arrays import java.util.LinkedList import java.util.UUID +import java.util.concurrent.CountDownLatch +import java.util.concurrent.TimeUnit import kotlin.test.Test import kotlin.test.assertEquals import kotlin.test.assertFails @@ -182,9 +184,15 @@ class SentryClientTest { assertTrue(sut.isEnabled) sut.close(false) assertNotEquals(0, fixture.sentryOptions.shutdownTimeoutMillis) - verify(fixture.transport).flush(eq(fixture.sentryOptions.shutdownTimeoutMillis)) - verify(fixture.loggerBatchProcessor).flush(eq(fixture.sentryOptions.shutdownTimeoutMillis)) - verify(fixture.metricsBatchProcessor).flush(eq(fixture.sentryOptions.shutdownTimeoutMillis)) + val transportFlushTimeout = argumentCaptor() + verify(fixture.transport).flush(transportFlushTimeout.capture()) + assertTrue(transportFlushTimeout.firstValue in 0..fixture.sentryOptions.shutdownTimeoutMillis) + val loggerFlushTimeout = argumentCaptor() + verify(fixture.loggerBatchProcessor).flush(loggerFlushTimeout.capture()) + assertTrue(loggerFlushTimeout.firstValue in 0..fixture.sentryOptions.shutdownTimeoutMillis) + val metricsFlushTimeout = argumentCaptor() + verify(fixture.metricsBatchProcessor).flush(metricsFlushTimeout.capture()) + assertTrue(metricsFlushTimeout.firstValue in 0..fixture.sentryOptions.shutdownTimeoutMillis) verify(fixture.transport).close(eq(false)) verify(fixture.loggerBatchProcessor).close(eq(false)) verify(fixture.metricsBatchProcessor).close(eq(false)) @@ -233,6 +241,110 @@ class SentryClientTest { assertTrue(invoked) } + @Test + fun `EventProcessor processAsync defaults return input`() { + val processor = object : EventProcessor {} + val hint = Hint() + val event = SentryEvent() + val transaction = SentryTransaction(fixture.sentryTracer) + val replay = SentryReplayEvent() + val log = SentryLogEvent(SentryId(), 1.0, "body", SentryLogLevel.INFO) + val metrics = SentryMetricsEvent(SentryId(), 1.0, "metric", "counter", 1.0) + + assertSame(event, processor.processAsync(event, hint)) + assertSame(transaction, processor.processAsync(transaction, hint)) + assertSame(replay, processor.processAsync(replay, hint)) + assertSame(log, processor.processAsync(log)) + assertSame(metrics, processor.processAsync(metrics, hint)) + } + + @Test + fun `when async processing is disabled, processAsync runs inline before beforeSend`() { + val order = mutableListOf() + fixture.sentryOptions.addEventProcessor( + object : EventProcessor { + override fun process(event: SentryEvent, hint: Hint): SentryEvent? { + order.add("process") + return event + } + + override fun processAsync(event: SentryEvent, hint: Hint): SentryEvent? { + order.add("processAsync") + return event + } + } + ) + fixture.sentryOptions.setBeforeSend { event, _ -> + order.add("beforeSend") + event + } + + val sut = fixture.getSut() + sut.captureEvent(SentryEvent()) + + assertEquals(listOf("process", "processAsync", "beforeSend"), order) + } + + @Test + fun `when async processing is enabled, captureEvent returns after task is accepted`() { + val asyncStarted = CountDownLatch(1) + val continueAsync = CountDownLatch(1) + val eventId = SentryId() + fixture.sentryOptions.isEnableAsyncProcessing = true + fixture.sentryOptions.addEventProcessor( + object : EventProcessor { + override fun processAsync(event: SentryEvent, hint: Hint): SentryEvent? { + asyncStarted.countDown() + assertTrue(continueAsync.await(5, TimeUnit.SECONDS)) + return event + } + } + ) + val sut = fixture.getSut() + val event = SentryEvent().apply { this.eventId = eventId } + + val actual = sut.captureEvent(event) + + assertEquals(eventId, actual) + assertTrue(asyncStarted.await(5, TimeUnit.SECONDS)) + verify(fixture.transport, never()).send(any(), anyOrNull()) + + continueAsync.countDown() + sut.flush(5000) + + verify(fixture.transport).send(any(), anyOrNull()) + } + + @Test + fun `when async processing queue is full, captureEvent returns empty id and records lost event`() { + val asyncStarted = CountDownLatch(1) + val continueAsync = CountDownLatch(1) + fixture.sentryOptions.isEnableAsyncProcessing = true + fixture.sentryOptions.maxQueueSize = 1 + fixture.sentryOptions.addEventProcessor( + object : EventProcessor { + override fun processAsync(event: SentryEvent, hint: Hint): SentryEvent? { + asyncStarted.countDown() + assertTrue(continueAsync.await(5, TimeUnit.SECONDS)) + return event + } + } + ) + val sut = fixture.getSut() + + assertNotEquals(SentryId.EMPTY_ID, sut.captureEvent(SentryEvent())) + assertTrue(asyncStarted.await(5, TimeUnit.SECONDS)) + assertEquals(SentryId.EMPTY_ID, sut.captureEvent(SentryEvent())) + + assertClientReport( + fixture.sentryOptions.clientReportRecorder, + listOf(DiscardedEvent(DiscardReason.QUEUE_OVERFLOW.reason, DataCategory.Error.category, 1)), + ) + + continueAsync.countDown() + sut.flush(5000) + } + @Test fun `when beforeSend is returns null, event is dropped`() { fixture.sentryOptions.setBeforeSend { _: SentryEvent, _: Any? -> null } @@ -2327,6 +2439,12 @@ class SentryClientTest { whenever(globalEventProcessorMock.process(any(), anyOrNull())).doAnswer { it.arguments.first() as SentryEvent } + whenever(scopedEventProcessorMock.processAsync(any(), anyOrNull())).doAnswer { + it.arguments.first() as SentryEvent + } + whenever(globalEventProcessorMock.processAsync(any(), anyOrNull())).doAnswer { + it.arguments.first() as SentryEvent + } whenever(beforeSendMock.execute(any(), anyOrNull())).doAnswer { it.arguments.first() as SentryEvent } @@ -2372,6 +2490,12 @@ class SentryClientTest { whenever(globalEventProcessorMock.process(any(), anyOrNull())).doAnswer { it.arguments.first() as SentryEvent } + whenever(scopedEventProcessorMock.processAsync(any(), anyOrNull())).doAnswer { + it.arguments.first() as SentryEvent + } + whenever(globalEventProcessorMock.processAsync(any(), anyOrNull())).doAnswer { + it.arguments.first() as SentryEvent + } whenever(beforeSendMock.execute(any(), anyOrNull())).thenReturn(null) val sut = diff --git a/sentry/src/test/java/io/sentry/SentryOptionsTest.kt b/sentry/src/test/java/io/sentry/SentryOptionsTest.kt index 75a24dd68d..b674d1e545 100644 --- a/sentry/src/test/java/io/sentry/SentryOptionsTest.kt +++ b/sentry/src/test/java/io/sentry/SentryOptionsTest.kt @@ -49,6 +49,18 @@ class SentryOptionsTest { assertFalse(SentryOptions().isDebug) } + @Test + fun `when options is initialized, async processing is false`() { + assertFalse(SentryOptions().isEnableAsyncProcessing) + } + + @Test + fun `when enableAsyncProcessing is set, overrides default`() { + val options = SentryOptions() + options.isEnableAsyncProcessing = true + assertTrue(options.isEnableAsyncProcessing) + } + @Test fun `when options is initialized, integrations contain UncaughtExceptionHandlerIntegration`() { assertTrue(SentryOptions().integrations.any { it is UncaughtExceptionHandlerIntegration }) @@ -399,6 +411,7 @@ class SentryOptionsTest { externalOptions.ignoredTransactions = listOf("transactionName1", "transaction-name-B") externalOptions.ignoredErrors = listOf("Some error", "Another .*") externalOptions.isEnableBackpressureHandling = false + externalOptions.isEnableAsyncProcessing = true externalOptions.isEnableDatabaseTransactionTracing = true externalOptions.isEnableCacheTracing = true externalOptions.maxRequestBodySize = SentryOptions.RequestSize.MEDIUM @@ -465,6 +478,7 @@ class SentryOptionsTest { options.ignoredErrors, ) assertFalse(options.isEnableBackpressureHandling) + assertTrue(options.isEnableAsyncProcessing) assertTrue(options.isEnableDatabaseTransactionTracing) assertTrue(options.isEnableCacheTracing) assertTrue(options.isForceInit)