The SegmentNotFound Exceptions in the AEM 6.3 Log

Statement:
Observed SegmentNotFound Exceptions in the logs. Due to some older issues in Oak or some inconsistencies in the repository, a segment can go missing and repository might be inconsistent.

For example:
[1]
*ERROR* [FelixStartLevel] org.apache.sling.event [org.apache.sling.event.impl.jobs.queues.QueueManager(1431)] The activate method has thrown an exception (org.apache.jackrabbit.oak.plugins.segment.SegmentNotFoundException: Segment da5bcb95-d00a-4c04-a9d9-0f10f2b14e5e not found)

[2]
*ERROR* [pool-6-thread-3] org.apache.sling.commons.scheduler.impl.QuartzScheduler Exception during job execution of org.apache.jackrabbit.oak.plugins.index.AsyncIndexUpdate@1dc173f9 : Segment e669f30b-e886-4b7a-b161-56432601ec6b not found
org.apache.jackrabbit.oak.plugins.segment.SegmentNotFoundException: Segment e669f30b-e886-4b7a-b161-56432601ec6b not found

Environment: AEM 6.0,6.1,6.2
Solution:
Run consistency checks on the repository and find the last good revision [healthy state] and revert back to it. Please do the following:
Step 1: Download a version of oak-run that matches your oak core version from https://mvnrepository.com/artifact/org.apache.jackrabbit/oak-run
§ To revert a corrupt segment store to its latest good state change into CQ's working directory (the one containing the crx-quickstart folder) and backup all files in ./crx-quickstart/repository/segmentstore/.
Step 2: Run the consistency check:
o java -Xmx6000m -jar oak-run-*.jar check -d1 -p /path/to/crx-quickstart/repository/segmentstore
o This searches backwards through the revisions until it finds a consistent one:
Look for message like below:
[main] INFO o.a.j.o.p.s.f.t.ConsistencyChecker - Found latest good revision afdb922d-ba53-4a1b-aa1b-1cb044b535cf:234880

Step 3: Revert the repository to this revision by editing ./crx-quickstart/repository/segmentstore/journal.log and deleting all lines after the line containing the latest good revision.
Step 4: Remove all ./crx-quickstart/repository/segmentstore/*.bak files.

Step 5: Run checkpoint clean-up to remove orphaned checkpoints:
§ java -Xmx6000m -jar oak-run-*.jar checkpoints /path/to/crx-quickstart/repository/segmentstore rm-unreferenced
Step6: Finally compact the repository:

§ java Xmx6000m -jar oak-run-*.jar compact /path/to/crx-quickstart/repository/segmentstore/