Reveal Help Center

1. Storybook Settings

Click the All storybooks link located under the STORYBOOK section in the left side bar to open the “All storybooks” page. The user can add a new storybook from this page or select an existing storybook from the list to add a new data source, export data and manage COSMIC models, clustering and document deletion functions. Use the Find feature to locate a storybook by entering any part of its Name.  

In the image below the All Storybooks page setting at the bottom right (echoing the page setting and controls at the top of the table) is set to 10 storybooks per page between the count summary and the paging controls.

604232bfc719e.png

See the following pages for details on each Storybook Administration item.

System and Tenant Admins may create a new analytical project.

The storybook dashboard gives a snapshot of running, stopped, errored, and completed tasks within the selected storybook.

The Data sources section allows System and Tenant Admins to upload, connect, and process documents & tags to your storybook.

D.  Export

Choose documents and data to export and customize output settings.

Define a custom entity model or type, or access and apply models from the Model Library.

This will route the user to the page that allows the user to create, modify and manage COSMIC groups and their settings.

Here are the tools to generate or regenerate AI vectors, near duplicates and cluster sets at any point after processing is complete. Clustering analyses textual documents and groups conceptually similar documents.

Document deletion function allows user to permanently delete processed documents from a storybook using a saved search.

The File storage page allows self-service of data loading and processing. You may upload data from multiple sources prior to processing, dragging & dropping files from your environment with no need to pre-load into SQL.

You can configure Insights to send alerts whenever new documents appear that meet defined criteria. Choose Insight notifications to view a list of recent alerts.

===================================================================================================

A. Add a new storybook

Click the Add storybook button on the “All storybooks” page to create a new storybook:

604232c224275.png

Choose a Storybook name, a Storybook database name, a Tenant this storybook belongs to, a destination database server (see Admin Guide Section 2.F.1 on how to create a SQL server connection) and select if you would like to Enable Insights.

Note

When Autogenerate is checked, the system will autogenerate the destination database name based on the storybook name indicated above.

B. Storybook dashboard

Clicking into a storybook from the All storybooks section will bring you to its dashboard. The storybook dashboard gives a snapshot of running, stopped, errored, and completed tasks within the selected storybook.

60638bec59199.png

Filter tasks by type, status and person initiating the task to find relevant information.

60638bee8a977.png
60638bf0399b3.png
60638bf1e3272.png
C. Data sources

The Data sources section allows you to upload, connect, and process documents & tags to your storybook.

60638bf3d34ae.png
  1. Import SQL data to your storybook

60638bf5d322d.png

Click the SQL server button to add a SQL connection.

Name your SQL server data source, select the server, database, table, and map fields.

60638bf7bc2cf.png
  • Naming

    • Name: Give your data source a name that you can use to reference in Reveal AI. We recommend naming it with descriptive information like server, database and table name so you can easily identify it.

    • SQL mapping: you can map all of your database fields from scratch by selecting “Custom” or save time by loading mapping selections used previously.

  • SQL connection

Select a source server you wish to read from. If your desired server is not available in the drop-down, go to Server connections under Tenant Settings to connect it.

Choose the database and table from which you wish to read your source data.

  • Database fields

Before processing begins, a SQL table with required metadata fields must be created. Only fields with compatible field types will be displayed as mapping options. Asterisked fields are required for processing.

ID

Field Name

Field Type

Field description

1

Id*

Number

Numeric identifier of the document. For Relativity, use ArtifactID

2

Control Number*

Text

Text identifier of the document

3

Custodian*

Text

Name of the custodian. If unavailable, processing will default empty values to “Empty_Custodian”

4

MD5 Hash*

Text

MD5Hash value, default to Control Number if empty

5

Group Identifier*

Text

Used to group email and attachments together; default to Control Number if empty. Notice for parent, the Group Identifier must be equal to the Control Number

6

Process Status*

Number

Indicate Reveal AI processing status

7

Extracted Text*

Text

Path to the text file exported, or text content itself

8

Email From

Text

Field only relevant to emails

9

Email To

Text

Field only relevant to emails

10

Email CC

Text

Field only relevant to emails

11

Email BCC

Text

Field only relevant to emails

12

Email Subject

Text

Field only relevant to emails

13

Primary Date Time

DateTime

Date and Time the email was sent, or last modified date time for attachments & e-files. It is important to include the “Time” part for this field.

14

Email Time zone

Text

The time zone with which the document should be processed. Most common values: EST/CST/PST/GMT

15

Email Attachment Count

Number

Field only relevant to emails. Number of attachments the email has.

16

eFile Author

Text

Last Author of the efile or attachment, if it exists

17

eFile Filename

Text

Filename of the efile or attachments

The Processing data settings page shows you the steps the system will take once processing begins. There are several settings associated with the computer-generated content detection, ingestion, natural language pipeline and pattern detection steps that you may modify. You may also modify the processing XML to include any custom settings that may be desired.

An Email notification option has been added, where if checked all data exports will automatically trigger a notification.

Note

If you are processing a Relativity workspace, the Reveal AI ADS package will automatically create a “NexLPProcessingStatus” field in the workspace. Before processing data, update Process Status field to 0 for all documents to be processed. For a list of all possible values for this field, refer to Appendix A. Make sure that the processing status field value for all documents that do not need to be processed has a Null value or a value greater than 5. If there are any records with processing status of 4, reset their processing status to 0. Process Data with any combination of entity models including the multiple entity Standard Model or any available from the Model library.

60638bf9eb912.png
  • Computer-generated Content Detection: click the Show options button to view and set options for computer-generated content detection:

    604232d12b574.png

Under Ingestion:

  • Default Time zone: The “Default time zone” option tells Reveal AI which time zone to use when processing email text if the Time Zone information is missing in the header lines.

  • Date Format: The system will prioritize processing using either UK or US date formats based on user selection for this option.

Under Natural language pipeline:

  • NER for eFiles: By default, Reveal AI only extracts Entities for emails. The “NER for eFiles” option allows user to extract Entities from loose files and attachments. Notice this could increase time needed for processing.

  • Entity models to run on data: Any models may appear by default according to your configuration. You may turn any model off by unchecking the associated checkbox. By pressing the Model library button you can view the list of additional available models as shown below. You may run any combination of models or none at all.

    60638bfd4d4a5.png

Under AI vector generation:

  • AI vector generation: By default you will build COSMIC vectors. You may optionally decide to build NexBERT vectors which is required if you are going to use NexBERT models.

    604232d4bfe93.png

Under Near duplicate document detection:

  • Check the Near duplicate detection box to run the process Near duplicate documents will be grouped and similarity scores calculated.

Under Pattern detection:

  • Patterns: By default, Reveal AI processes Patterns. If Pattern Detection is not required, uncheck the “Patterns” checkbox.

Before kicking off processing, the system will confirm the number of documents you will be processing and undergo final pre-flight checks to ensure the data is in good shape to process.

60638c0099586.png

If your data isn’t set up correctly you may get a warning in this screen.

Note

Processing will kick off regardless of any warnings.

604232d7dd042.png There are no documents ready for processing 604232d90bddf.png

This is the number of documents with a process status=0. If this is not the number you were expecting, please check your data source and update the process status field.

604232d7dd042.png 0 of X processing services are available 604232d90bddf.png

Your processing sevice is stopped. If you click Start your processing job will be added to the queue, but the processing service will need to be launched before this one begins.

604232d7dd042.png No service found with POS Tagger API configured 604232d90bddf.png

Your POS Tagger sevice is stopped. If you click Start, your processing job will be added to the queue, but the POS Tagger service will need to be launched before this one begins.

604232d7dd042.png No service found with NER configured 604232d90bddf.png

This data source was configured to run with entity extraction but no service was found with Named-Entity Recognition (NER) configured. It is recommended that this be configured to ensure actual results.

604232d7dd042.png No service found with DCI configured 604232d90bddf.png

This data source was configured to run with dataless classification but no service was found with DCI configured. It is recommended that this be configured to ensure actual results.

604232d7dd042.png Documents may not have attachments 604232d90bddf.png

Column “Groupidentifier” in Table “EDDSDBO.Document” of Database “EDDS1015555_Test” contains at least one blank or null value. Column “Groupidentifier” values will be replaced with values from Column “ControlNumber” for rows where this occurs. When this occurs, documents will not have any associated attachments.

604232d7dd042.png Documents may not have valid md5 value 604232d90bddf.png

Column “MD5Hash” in Table “EDDSDBO.Document” of Database “EDDS1015555_Test” contains at least one blank or null value. Column “MD5Hash” values will be replaced with values from Column “ControlNumber” for rows where this occurs.

604232d7dd042.png Documents may not have some necessary indexes 604232d90bddf.png

Table “EDDSDBO.Document” of Database “EDDS1015555_Test” does not have an index which contains the column Column””.

Create Missing Index

Once you click the Start button processing will kick off immediately. You will be able to check the status of processing in the Notifications center and/or storybook dashboard. There will be a square STOP button to the right of the Notification entry until processing is completed.

  • Notifications center:

60638c0c513d4.png
  • Storybook dashboard view

If you decide to click the Stop button while data is processing, your progress will be saved. You can resume by going to the Data sources page and clicking the triangle icon under the “Process” column as shown below.

60638c0e5db92.png
  1. Import from Exchange/Office 365

To import from Exchange or Office 365 choose the Exchange/Office 365 button:

60638c1035c49.png

And this window appears:

60638c1211c70.png

The fields for adding an Exchange or an Office 365 data source are:

  • Data source name (required): Choose an appropriate name.

  • Storybook (required): Automatically populated.

  • Control Number Prefix (required): Some descriptive title which will appear in front of the control number.

  • Server Connection: Choose from the dropdown list of connected data sources. The connected data sources choices can be configured as described in 2. Tenant Settings > F. Server Connections > 4. Add Exchange/Office 365 Server Connection.

  • Schedule: Either “On demand” which will initiate import when you complete configuration, or “Daily” according to whatever hour you specify.

  • Email notification: If checked all data exports will automatically trigger a notification.

When “Server Connection” is chosen, a checkbox list of mailboxes appears:

60638c142d535.png

Check any or all mailboxes from the list under Data and then choose Save and Continue.

60638c163d1c2.png

The following page appears:

60638c181892f.png
60638c1a257e0.png

The Processing data settings page shows you the steps the system will take once processing begins. There are several settings associated with the computer-generated content detection, ingestion, natural language pipeline, and pattern detection steps that you may modify. You may also modify the processing XML to include any custom settings that may be desired.

  • Computer-generated Content Detection: Click the Show options button to view and set options for computer-generated content detection:

    60638c1c210a3.png
  • Under Ingestion:

    • Default Time zone: The “Default time zone” option tells Reveal AI which time zone to use when processing email text if the Time Zone information is missing in the header lines.

    • Date Format: The system will prioritize processing using either UK or US date formats based on user selection for this option.

  • Under Natural language pipeline:

    • NER for eFiles: By default, Reveal AI only extracts Entities for emails. The “NER for eFiles” option allows user to extract Entities from loose files and attachments. Notice this could increase time needed for processing.

    • Entity models to run on data: Any models may appear by default according to your configuration. You may turn any model off by unchecking the associated checkbox. By pressing the Model library button you can view the list of additional available models as shown below. You may run any combination of models or none at all.

      60638c1e8a707.png
  • Under AI vector generation:

    • AI vector generation: By default you will build COSMIC vectors. You may optionally decide to build NexBERT vectors which is required if you are going to use NexBERT models.

      604232d4bfe93.png
  • Under Pattern detection:

    • Patterns: By default, Reveal AI processes Patterns. If Pattern Detection is not required, uncheck the “Patterns” checkbox.

Before kicking off processing the system will confirm the number of documents you will be processing, and undergo final pre-flight checks to ensure the data is in good shape to process.

60638c21cb122.png

If your data isn’t set up correctly you may get a warning in this screen.

Note

Processing will kick off regardless of any warnings.

604232d7dd042.png There are no documents ready for processing 604232d90bddf.png

This is the number of documents with a process status=0. If this is not the number you were expecting, please check your data source and update the process status field.

604232d7dd042.png 0 of X processing services are available 604232d90bddf.png

Your processing sevice is stopped. If you click Start your processing job will be added to the queue, but the processing service will need to be launched before this one begins.

604232d7dd042.png No service found with POS Tagger API configured 604232d90bddf.png

Your POS Tagger sevice is stopped. If you click Start, your processing job will be added to the queue, but the POS Tagger service will need to be launched before this one begins.

604232d7dd042.png No service found with NER configured 604232d90bddf.png

This data source was configured to run with entity extraction but no service was found with Named-Entity Recognition (NER) configured. It is recommended that this be configured to ensure actual results.

604232d7dd042.png No service found with DCI configured 604232d90bddf.png

This data source was configured to run with dataless classification but no service was found with DCI configured. It is recommended that this be configured to ensure actual results.

604232d7dd042.png Documents may not have attachments 604232d90bddf.png

Column “Groupidentifier” in Table “EDDSDBO.Document” of Database “EDDS1015555_Test” contains at least one blank or null value. Column “Groupidentifier” values will be replaced with values from Column “ControlNumber” for rows where this occurs. When this occurs, documents will not have any associated attachments.

604232d7dd042.png Documents may not have valid md5 value 604232d90bddf.png

Column “MD5Hash” in Table “EDDSDBO.Document” of Database “EDDS1015555_Test” contains at least one blank or null value. Column “MD5Hash” values will be replaced with values from Column “ControlNumber” for rows where this occurs.

604232d7dd042.png Documents may not have some necessary indexes 604232d90bddf.png

Table “EDDSDBO.Document” of Database “EDDS1015555_Test” does not have an index which contains the column Column””.

Create Missing Index

Once you click the start button, processing will kick off immediately. You will be able to check the status of processing in the notification center and/or storybook dashboard.

  • Notifications center:

60638c2c76aee.png
  • Storybook dashboard view

If you decide to click the Stop button while data is processing, your progress will be saved. You can resume by going to the Data sources page and clicking the triangle icon under the “Process” column as shown below.

60638c2ea74e0.png
  1. Import documents from Relativity

Please see Section 4. Integration with 3rd Parties > A. Relativity > 2. Import Documents from Relativityin this Admin Guide.

  1. Import tags from delimited files

Choose the Delimited file option to import user tags or COSMIC tags to your existing documents.

60638c30bb946.png

Enter a data source name and upload your file. Note that the file must be comma-delimited.

The file you upload must contain:

  • One column with control number.

  • Other columns with COSMIC or external tags.

    • COSMIC tags must be labeled Yes, No or Skip. This is the indicator at the document level.

    • External tags may have any value.

60638c32e17a1.png

Batch options: Batch options are by default 10,000 documents with a timeout of 60 seconds, and this should work for most exports. However, you can change the settings if you are having issues with delays during export.

If your file contains headers in the first row, check “First row contains headers.”

Once your file is successfully uploaded you will see the following:

60638c34f32fa.png

Map each field you wish to import using the field mapping tool.

60638c36da429.png

Select a Reveal AI field to which you wish to map your choices. If you would like to create a new field, click Create new field.

The following appears:

60638c38b7596.png

Choose a conflict resolution to tell the system what you want it to do if there are existing values that conflict with the newly uploaded values.

60638c3ab4487.png

Skip: Don’t import the value.

Merge: Keep both the existing and imported values.

Overwrite: Replace the existing value with the imported value.

Once you begin processing:

  • A “Process data” task will begin running in the Notifications Center.

    60638c3c9980b.png
  • You can track the progress in the Storybook/ Admin dashboard.

    60638c3e674ea.png
  • Your delimited file data source will appear in the Data Sources list with type “Delimited file”.

    60638c4013d7a.png

After processing has completed, you should expect the following:

EXTERNAL TAGS: Your tags will now be available in search:

6042331bd3f5a.png

Note

New external tags will not be available on the document level as a user tag.

60638c4299bf0.png

COSMIC TAGS: Correct tag (Yes / No / Skip) will be selected in the thread viewer.

Note

You cannot delete tags after they have been imported.

  1. Import tags from Relativity

Please see 4. Integration with 3rd Parties > A. Relativity > 3. Import Tags from Relativity in this Admin Guide.

  1. Relativity COSMIC Monitoring

Please see 4. Integration with 3rd Parties > A. Relativity > 6. COSMIC Monitoring in this Admin Guide.

  1. View data sources

After creation of a data source, new data sources will be available in the Connected data sources section.

Note

Data sources that appear in this list can be in all stages of processing.

  1. Clicking the 6042331edca40.png button under the “Process” column will:

    • Kick off processing if processing has not yet occurred.

    • Resume processing if it had been previously started and stopped.

    • Re-process if processing has already completed.

If you re-process, documents in your storybook will not be affected. Rather, any additional documents added to the data source will be processed.

  1. Clicking the 6042331ff2d9d.png button allows user to edit properties of the data source. User can re-map field, re-name data source by using the edit function. Edits will be picked up upon the next processing.

  1. Clicking the 60423321130af.png button allows user to delete the data source.

    Note

    Deleting the data source will not delete documents from the storybook. In order to delete documents permanently, follow the instructions under Section 1.H Document deletion.

If attempting to re-process, edit or delete a data source while processing is in progress, you will see a warning message:

6042332278d85.png

Click on a data source...

60638c472339d.png

...to see all completed, errored, stopped, or running processes.

60638c492419a.png

Expand details on each task to view & download logs.

  • Summary: provides processing results in detail.

  • Processing: includes processing exceptions.

  • Performance: includes performance metrics.

60638c4b07deb.png

The Error Report highighted above in red is described in Appendix E. New “Error Report” Function in Front-End/NPA.

If your memory usage exceeds 99% you will see a Low memory warning. It is recommended to stop processing, reduce the number of threads in the service configuration and restart processing.

60638c4cecb06.png
D. Export

Choose documents and data to export and customize output settings.  See User Guide Section 11 for more details.

  • Export to Relativity

Please see Section 4 - Integration with 3rd Parties > A. Relativity > 4. Export to Relativity in this Admin Guide.

E. Entity models

In the Admin Module, in the left column under STORYBOOK SETTINGS, choose Entity models, and the Entity models view appears:

60638c514db62.png

The Entity models view contains the following:

  • Create model: The Create model button allows you to select an existing custom entity type and put Reveal AI to work building a model based on that entity type. See Model creation below.

  • Model library: Provides access to the entity Model Library where all the published models are available for use. See Model library below for additional information on the storybook-specific view of the Model Library.

  • Entity model/Entity type: This button is used to choose between the Entity model and Entity type views. See Entity type view below.

  • Selection Column (marked by a checkbox): Use this column to choose any available existing model(s) to Run or Publish. Once a choice is made, Run and Publish buttons appear at the top:

    60638c53051a9.png
  • Model: The name and description of the model.

  • Entity Type: The entity type associated with the model.

  • F1: Broad information retrieval performance of the model – a golden mean average of recall and precision.

  • Recall: Evaluation of the model’s performance in finding all the relevant entities.

  • Precision: Evaluation of the model’s performance in minimizing false positives.

  • Status: Is the model Ready to be published or has it been Published.

  • Entity Examples: The number of entity examples are used to create the model.

  • Segment Examples: The number of segments containing entities.

  • Discarded Examples: The number of examples discarded by the AI model.

  • Build Source: Example set used to build model.

  • Build: Create the entity model based on existing examples.

  • Run: Run the entity model on the current data to generate new annotations.

  • Publish: Send the entity model to the library.

  • Action: Choosing the trash can icon deletes the model.

Entity type view

The Entity model view can be changed to the Entity type view by choosing the Entity model/Entity type button:

60638c54acab6.png

If you choose Entity type...

60423331bce2a.png

...you will see the following view:

60638c57c0dda.png

Which contains the following columns:

  • Visible: A checkmark here determines that this entity is visible to the user.

  • Entity type: The name of the entity type.

  • Action: When you choose the edit icon, the following pop-up appears. The pop-up allows you to modify the entity type name and determine whether it is visible in the storybook.

    60638c5995430.png
  • Models: The name of the entity model associated with the entity type.

Model Creation

In the Admin Module, in the left column under STORYBOOK SETTINGS, choose Entity models and then in the main view to the right under the heading “Entity models” choose Create model.

60638c5b62605.png

The Create Entity Model window appears:

60638c5d40fa9.png

Enter the following:

  • Name: Choose an appropriate name.

  • Entity Type: Select an existing Entity type from the dropdown list.

    Note

    Only Custom Entities will be available for model build.

  • Description: Enter additional information such as genesis or purpose.

Build and Run Model

Once you are happy with the annotations you may choose Build now. For more information on annotations see User Guide, Section 5, Custom Entity Types.

60638c5f1b303.png

When you select Build now, you will be prompted to choose which entity examples to use to build your model.

60638c60d09d7.png

You have two options:

  1. All examples. By choosing this option, all annotations associated with your custom entity will be used to build your model. These examples could be from earlier model results, entity search & extract results, and user annotations.

  2. Examples annotated by user. Only examples that are explicitly annotated by the user will be used to build your model. See User Guide, Section 5, Custom Entity Types.

When the model is built the above table will provide information retrieval measurements: F1Recall and Precision. You may now apply the model to your storybook by choosing Run. Go to the Notification icon for status. You may return to the EXPLORING tab to review results. (See User Guide Section 5C Building a Custom Entity Model.)

Publish Model

If you want to post the entity models to your library for use in other storybooks, choose Publish. When you publish an entity model you are sending a copy to the Entity model library. The library is a dynamic repository of entity models that includes the Standard Model (a collection of broadly applicable entity types) and whatever custom entity models have been provided, imported or created. 

Model Library

If you want to add a model from the library to your storybook, press the Model library button.

60638c62834ea.png

This storybook-specific view of the Model Library displays as a collection of catalogue cards that show what models are available to add to your storybook (there is an alternate tenant-wide view of the Entity model library described in Section 2 - Tenant Settings > E. Model library):

6042333e9f6c3.png

The full blue ADD buttons (outlined in green) are available to add. The faded blue ADD buttons (outlined in red) are unavailable because they are already in your storybook. The UPDATE button outlined in green is a revised version of a model from another storybook, an outdated version of which you have added to your current storybook. This marks that you have the option of updating this model in your current storybook. 

There are four filter buttons:

  • Find: Enter search text which will query entity model title, entity and notes.

  • Published by: Name of admin who published this model.

  • Order by: Either oldest to newest or newest to oldest.

  • Reset Filters: Clear the filters and view the entire library catalogue.

Entity Model Life Cycle

Once a model is built it can be made available to other storybooks by being published. When you view the library from within another storybook you may add the published model to the current storybook by choosing the ADD button (green outlined above). A published model cannot be added to the storybook whence it came, nor to any storybook to which it has already been added. This is prohibited (as red outlined above) by the ADD button being made unavailable and presented in a faded blue. 

If a model is altered through annotation, in the same storybook from which it originated, and then published, the outdated library version is replaced by the new version. In that case, when you view the library from within another storybook to which the earlier model version has been added, the new library version will appear with an UPDATE button (blue outlined above) instead of the ADD button. This will allow a user of an inherited model to check for updates prior to running or to consider re-running. 

If a model is altered through annotation in a storybook other than the one from which it originated, then when it is published to the library it is a distinct model - not a replacement for the existing one. This is exemplified above by the two models named AmerByKit and PhoneNumbers with different details underneath. 

By this means, the system keeps separate and distinct each model fed by annotations from different projects. 

For more information on managing the Entity model library see Section 2 - Tenant Settings > E. Model library.

F. COSMIC groups

This will route the user to the page that allows the user to create, modify and manage COSMIC groups and their settings.

60423340caaf9.png

Except for the Package option described below, this COSMIC group functionality is available to the non-admin user as well. See Reveal AI User Guide Section 9. COSMIC (Cognitive Machine Coding) for more details. 

A COSMIC Group - developed in a specific storybook - can be thought of as a deployable COSMIC Model. COSMIC Models are portable packages of knowledge a user can export at a storybook level. At the push of a button, Reveal AI will automatically package and encrypt your AI model from any storybook and allow for utilization of the AI Model in any storybook via the COSMIC Model Library.  

Package Model from Storybook

Before you can apply a model, it needs to be packaged from the source storybook. The Package feature is available under the Applied AI Flyout Menu item, which opens the COSMIC Group screen.

Flyout_Menu_-_Applied_AI.png
6042334378357.png

And COSMIC Mission Control appears:

6042334533c19.png

Expand the Package section and click the Package button to show the dialog box to package the model:

60423346b430e.png

Enter the Name and Notes for the model and click Publish to start the model generating process. Notice by default you need to have at least 4 positive samples and 1 negative sample in order to publish the model. 

Refresh the page and wait until the link becomes available. Click Download to save the model file in order to use the model in a different Reveal AI environment:

60423348a1b28.png

Things not packaged: the following metadata fields are not included in the reference models: Sender; Sender Domain; Recipient; Recipient Domain; Author; Organization; Dates.

G. Rebuild Data

You may need to generate or update certain data after initial processing. Processing updates for three items have been consolidated under Rebuild data.

The data which may be generated or rebuilt here are:

  1. AI Vectors

    AI vector generation allows you to build vectors at some point after processing is complete. A typical use for this might be that during processing you did not see any need for the creation of NexBERT vectors but have now changed your mind.

    60638c6a8b8e1.png

    Choose which of the two vector types you wish to build. The Notification Icon in the image below shows that COSMIC text vector creation has been completed.

    60638c6cd6001.png
  2. Near Duplicates

    Near duplicate processing to identify similar documents may be run from here either if skipped during initial ingestion or if linked updates are to be consolidated. Click Generate near duplicate documents and see the checkmark on a green Notifications icon at the upper right of the control bar.

    60638c6eef2b5.png
  3. Cluster Sets

    Clustering analyses textual documents and groups conceptually similar documents. In Reveal AI 1.14.04 and later versions, Cluster Queues are automatically created after each data processing is complete.

    To manually adjust the parameters and re-create clusters follow the instructions below:

    60638c70a7040.png
    • Enable: check this box to enable clustering.

    • Name: the name of the cluster you’d like to create.

    • Queue Sample Size: determines how large the sample size will be for the cluster queue.

    • Max Cluster Count: maximum number of clusters the system will create, recommend 2.

    Click Save to save the settings: the Run Cluster Set button will appear. Click Run Cluster Set button to kick off clustering.

    Notice the clustering processing is a backend service. To check if clustering is complete, open Viz and click “Treemap” button below:

    6042335018dcb.png

Also see Appendix D for a list of system generated clusters.

H. Document deletion

Document deletion function allows user to permanently delete processed documents from a storybook.

First, Create a Saved Search that includes documents to be removed from the storybook.

Select Document Deletion:

60638c73cb5cd.png

The Add Document Deletion screen appears.

60638c75e70cd.png
  1. Provide a Name for the document deletion job (required).

  2. Select a Saved Search from the dropdown list (required).

  3. Set a Query timeout value to return a list of documents selected for deletion (required).

  4. From the Schedule dropdown you may choose either On demand,Scheduled or Advanced.

    • If you choose Scheduled you enter the schedule dale and time from the pop-up date and time choices; you will click Save to set the scheduled job:

      60638c77c2c6a.png
    • If you choose On demand… the choices for the job are either Save or Save and identify.

      60638c7974036.png
    • The Advanced selection accepts a CRON expression in UTC timezone for Minute Hour Day-of-Month Month Day-of-Week; Save sets the job.

      60638c7b2574c.png
  5. You may determine a recipient for Email notification by checking the box to Send email with document deletion. Select the server from the dropdown list and enter the recipient(s) - both To and optionally BCC.

  6. When you Save an on-demand deletion, for example, the Document deletion screen will show the job and proceed to identify the documents and related elements to be removed.

  7. When identification is complete the results are reported and you are presented with this request for confirmation:

    60638c7ce9f31.png
  8. To proceed, choose Delete documents and data.

  9. You are prompted to confirm, with a warning that this cannot be undone.

    60638c7e5fe9c.png
  10. In Step 5 of 5, once the documents are deleted, a cleanup process begins automatically, reprocessing indexes, entities and other data elements affected by the document deletion.

    60638c801ff68.png

Reporting on both deletion and cleanup status can be viewed through the Storybook Dashboard as shown above. The Cleanup job, which opens once the Deletion job is complete, updates status.

60638c82466d2.png

...

60638c840dbc7.png

...

60638c8597148.png
I. File storage

The File storage page allows self-service of data loading and processing. You may upload data from multiple sources prior to processing. You drag & drop files from your environment with no need to pre-load into SQL.

To begin, choose File storage in the Storybook list and then the File icon.

60638c87b4d17.png

And the New File Upload page appears:

60638c89c002e.png

This requires the following entries:

  • Storybook*: The storybook name is entered automatically.

  • Upload name*: Provide a name for the incoming data file.

  • Notes: Optionally provide notes.

  • File Path*: Provide by dragging and dropping file into provided space.

60638c8b93f39.png

And choose Save.

60638c8d79b03.png

The new file appears in the file storage list.

You can also load from this location while in Section 1. Storybook > C. Data Sources above.

J. Insights notifications

See the User Guide Section 2. Reveal AI Exploring Page > C. Insights for a description of Insights purpose and functionality.

You can configure Insights to send alerts whenever new documents appear that meet defined criteria. Choose Insight notifications to view a list of recent alerts or to configure new alerts.

60638c8fa1366.png

The list includes:

  • The module name: Such as “Document Count”, “Inclusive Count”, etc.

  • The document count

  • The section

  • Date and Time of the notification

  • View Insights button: This button links to the referenced Insights User view.

To configure notifications choose the button as shown:

60638c91a33c3.png

The following page appears:

60638c9324db7.png

On this page you can designate who receives what notifications and when. You can add notifications to the notification list and set the time (UTC) of the notification with the dropdown toll at the bottom of the page.

Choose the Add notification button and the following appears:

60638c94eda0a.png

Make the following choices:

  • Section: Choose the relevant Section from the dropdown.

  • Module: Choose the relevant Module from the dropdown of Modules associated with the chosen section. The relevant criteria name follows the module name in parentheses.

  • Recipient type: Choose either Group or Email.

  • Group /Email: Either a Group or Email dropdown list will show depending on your previous choice. Choose one or more recipients.

For example:

60638c969ae99.png

Then hit Save. The new entries (one for each recipient) appear on the Notifications list. You may schedule the time of the daily alert.

60638c9840244.png

To set or change the Schedule, open the dropdown list of times (designated in UTC):

60638c9a11b3d.png

Choose the time for the alert and then choose Save.

The associated module in the Exploring view will now appear with an envelope icon indicating that notification is set up. See Document Count module in the illustration below.

60638c9bd10b3.png

Anyone configured for notification will receive trigger event email that looks roughly like this:

60638c9ddb6ce.jpg

Users who have been granted permission (by way of their group configuration - see Section 2 - Tenant Settings > B. Groups) may add notifications. A description of User utilization of Insights is available in the User Guide Section 2 - Reveal AI Exploring Page > C. Insights.