> For the complete documentation index, see [llms.txt](https://docs.borndigital.ai/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.borndigital.ai/digital-agent/tools-reference/text-and-language.md).

# Text & language

Matching, comparing, cleaning and translating text; language recognition.

Matching, comparing, cleaning and translating text; language recognition. How to add a tool and use its result is described in [Tools & smart functions](https://docs.borndigital.ai/digital-agent/tools-reference.md).

| Tool                     | Technical name          | What it does                                                                                                                                                                     | Languages                                      |
| ------------------------ | ----------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------- |
| Censor vulgarism         | `censor`                | Detect and censor vulgar expressions in text, with options to mask or list detected words.                                                                                       | sk, cs                                         |
| Cosine similarity        | `cosine_similarity`     | Compute cosine similarity between two strings or between a string and a list of strings.                                                                                         | any                                            |
| Detect vulgarism         | `bad_words`             | Sanitize user input by replacing vulgar words with placeholders or euphemistic substitutions.                                                                                    | cs                                             |
| Distance between strings | `get_edit_distance`     | Calculate linguistically informed edit distance of two string.                                                                                                                   | any                                            |
| Enum match               | `enum_match`            | Map synonyms in user input onto a fixed set of values using a shared dictionary file, matching whole tokens.                                                                     | any                                            |
| Keyword extractor        | `keywords`              | Return all keywords from the provided list or stem dictionary that are found in the user’s utterance.                                                                            | any                                            |
| Quick string matching    | `quick_string_matching` | Return the top string matches for a user’s input against a predefined entity list using TF-IDF fuzzy matching and cosine similarity.                                             | any                                            |
| Regex match              | `regex_match`           | Return `True` when the customer's message matches the regular expression (with optional flags) – or, with **Inverse** on, when it does not match. Otherwise the result is empty. | any                                            |
| Remove diacritics        | `remove_diacritics`     | Return the input text with all diacritical marks removed.                                                                                                                        | any                                            |
| Text anonymizer          | `anonymization`         | Anonymize personal and sensitive information in text by calling an external service.                                                                                             | any                                            |
| Text sanitizer           | `sanitize_text`         | Remove HTML and JavaScript code from the input text and return safe text                                                                                                         | any                                            |
| Translate text           | `translate`             | Translate input text to a target language using Azure Translate API                                                                                                              | cs, fr, hr, hu, nl, ro, ru, sk, en, pl, de, pt |
| Utterance language       | `utterance_language`    | Detect the language of a text and return its ISO language code                                                                                                                   | any                                            |

## Censor vulgarism

Technical name: `censor`

Detect and censor vulgar expressions in text, with options to mask or list detected words.

* Languages: sk, cs.

| Parameter    | Name           | Required | Default  | Description                                                                                                              |
| ------------ | -------------- | -------- | -------- | ------------------------------------------------------------------------------------------------------------------------ |
| Custom input | `custom_input` | no       |          | Provide text to sanitize instead of using the last user message.                                                         |
| Output type  | `output_type`  | no       | `censor` | Choose output mode: 'censor' replaces vulgar terms with '\*\*\*', 'detect' returns the count and list of detected words. |

**Example input:** This is a shitty solution, this sucks.

**Example output:**

```
'This is a *** solution, this ***.' (for 'censor') or {'vulgarismCount': 2, 'vulgarisms': ['shitty','sucks']} (for 'detect')
```

## Cosine similarity

Technical name: `cosine_similarity`

Compute cosine similarity between two strings or between a string and a list of strings.

| Parameter          | Name                 | Required | Default | Description                                                            |
| ------------------ | -------------------- | -------- | ------- | ---------------------------------------------------------------------- |
| First string       | `string_1`           | no       |         | The reference string to compare.                                       |
| Second string      | `string_2`           | no       |         | The target string or list of strings to compare against the reference. |
| Max features       | `max_features`       | no       | `3000`  | Maximum number of character ngrams to use as features.                 |
| Generic vectorizer | `generic_vectorizer` | no       | `False` | Use a pre-trained generic vectorizer instead of fitting a new one.     |

**Example input:** Compare 'Hurbanovo' with 'Hrbanovo'

**Example output:**

```
[['Telekom', 0.798], ['Google corp.', 0.012]]
```

## Detect vulgarism

Technical name: `bad_words`

Sanitize user input by replacing vulgar words with placeholders or euphemistic substitutions.

* Languages: cs.

| Parameter    | Name           | Required | Default | Description                                                      |
| ------------ | -------------- | -------- | ------- | ---------------------------------------------------------------- |
| Custom input | `custom_input` | no       |         | Provide text to sanitize instead of using the last user message. |

**Example input:** This is a shitty solution, this sucks.

**Example output:**

```
This is a * solution, this is nonsense*
```

## Distance between strings

Technical name: `get_edit_distance`

Calculate linguistically informed edit distance of two string.

| Parameter          | Name                 | Required | Default         | Description                                                                                                                                                                                                             |
| ------------------ | -------------------- | -------- | --------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| First string       | `string_1`           | yes      |                 | First string to compare.                                                                                                                                                                                                |
| Second string      | `string_2`           | yes      |                 | Second string to compare.                                                                                                                                                                                               |
| Edit distance mode | `edit_distance_mode` | no       | `edit_distance` | Mode of edit distance calculation. CER for Character Error Rate metrics as input, edit\_distance for edit distance as metric. By default 'edit\_distance', always using some sort of phonetic similarity consideration. |

**Example input:** string\_1: 'love', string\_2: '\[ 'loe', 'Love', 'lowe' ]'

**Example output:**

```
[['loe', 2.0], ['lowe', 0.75], ['Love', 0.25]]
```

## Enum match

Technical name: `enum_match`

Map synonyms in user input onto a fixed set of values using a shared dictionary file, matching whole tokens.

| Parameter      | Name             | Required | Default | Description                                                                                                      |
| -------------- | ---------------- | -------- | ------- | ---------------------------------------------------------------------------------------------------------------- |
| Entity type    | `entity_type`    | yes      |         | Name of the dictionary folder under namedentitysupport/dictionaries, holding `synonym,VALUE` lines per language. |
| Single answer  | `single_answer`  | no       | `False` | Return the first value instead of a list; the result is empty when nothing matches.                              |
| Allowed values | `allowed_values` | no       |         | Keep only these values, everything else is dropped.                                                              |
| Get indices    | `get_indices`    | no       | `False` | When on, return each value together with its character start/end indices.                                        |
| Custom input   | `custom_input`   | no       |         | Provide text to match against instead of the last user message.                                                  |

**Example input:** chcem si vymenit 200 eur v hotovosti

**Example output:**

```
['BRANCH']
```

## Keyword extractor

Technical name: `keywords`

Return all keywords from the provided list or stem dictionary that are found in the user’s utterance.

| Parameter     | Name            | Required | Default | Description                                                                  |
| ------------- | --------------- | -------- | ------- | ---------------------------------------------------------------------------- |
| Keywords      | `keywords`      | yes      | `[]`    | List of keywords or stems to search for in the user’s utterance.             |
| Ignore spaces | `ignore_spaces` | no       | `False` | Ignore spaces when matching keywords.                                        |
| Custom input  | `custom_input`  | no       |         | Override the user’s utterance with a custom string to extract keywords from. |

**Example input:** I am Erik and my friend is Adam.

**Example output:**

```
['erik', 'adam']
```

## Quick string matching

Technical name: `quick_string_matching`

Return the top string matches for a user’s input against a predefined entity list using TF-IDF fuzzy matching and cosine similarity.

| Parameter         | Name           | Required | Default | Description                                                              |
| ----------------- | -------------- | -------- | ------- | ------------------------------------------------------------------------ |
| Entity type       | `entity_type`  | yes      |         | Name of the predefined entity list to match against (e.g., 'companies'). |
| Remove vectorizer | `to_remove`    | no       | `False` | When on, clear the cached vectorizer so it is rebuilt on next match.     |
| Custom input      | `custom_input` | no       |         | Override the user’s utterance with a custom string for matching.         |

**Example input:** Acme Corporation

**Example output:**

```
[[1.0, 'Acme Corporation'], [0.93, 'Acme Corp.'], [0.89, 'Acme International']]
```

## Regex match

Technical name: `regex_match`

Return `True` when the customer's message matches the regular expression (with optional flags) – or, with **Inverse** on, when it does not match. Otherwise the result is empty.

| Parameter          | Name           | Required | Default | Description                                                                   |
| ------------------ | -------------- | -------- | ------- | ----------------------------------------------------------------------------- |
| Regular expression | `regex_str`    | yes      |         | Regular expression pattern to apply to the input utterance.                   |
| Flags              | `flags`        | no       |         | List of regex flags to apply (e.g., \['IGNORECASE', 'MULTILINE']).            |
| Inverse            | `inverse`      | no       | `False` | When on, the result is `True` when the pattern does **not** match.            |
| Custom input       | `custom_input` | no       |         | Override the user’s utterance with a custom string to test against the regex. |

**Example input:** I travel to Ireland

**Example output:**

```
True
```

## Remove diacritics

Technical name: `remove_diacritics`

Return the input text with all diacritical marks removed.

* Can also be used in expressions as `remove_diacritics()`.

| Parameter | Name   | Required | Default | Description                               |
| --------- | ------ | -------- | ------- | ----------------------------------------- |
| Text      | `text` | yes      |         | Text to normalize by removing diacritics. |

**Example input:** Ján Kováč

**Example output:**

```
Jan Kovac
```

## Text anonymizer

Technical name: `anonymization`

Anonymize personal and sensitive information in text by calling an external service.

| Parameter | Name   | Required | Default | Description                                                          |
| --------- | ------ | -------- | ------- | -------------------------------------------------------------------- |
| Text      | `text` | no       |         | Text to anonymize; if omitted, the latest user message will be used. |

**Example input:** I am John Smith and my phone number is 0901 123 456.

**Example output:**

```
I am [PERSON] and my phone number is [PHONE].
```

## Text sanitizer

Technical name: `sanitize_text`

Remove HTML and JavaScript code from the input text and return safe text

* Can also be used in expressions as `sanitize()`.

| Parameter    | Name           | Required | Default | Description                                        |
| ------------ | -------------- | -------- | ------- | -------------------------------------------------- |
| Custom input | `custom_input` | no       |         | Custom text to sanitize provided by the flow       |
| Tags         | `tags`         | no       |         | HTML tags to allow in the sanitized text           |
| Keep spaces  | `keep_spaces`  | no       | `False` | Whether to retain original spacing when sanitizing |

**Example input:**

Hello alert('X')World

**Example output:**

```
Hello World
```

## Translate text

Technical name: `translate`

Translate input text to a target language using Azure Translate API

* Languages: cs, fr, hr, hu, nl, ro, ru, sk, en, pl, de, pt.

| Parameter                | Name           | Required | Default | Description                       |
| ------------------------ | -------------- | -------- | ------- | --------------------------------- |
| Language for translation | `language_to`  | yes      |         | Target language to translate into |
| Input text               | `input_string` | yes      |         | Text to translate                 |
| Source language          | `from_lang`    | no       |         | Source language of the input text |

**Example input:** language\_to='cs', from\_lang='en', input\_string='Hello world'

**Example output:**

```
Ahoj světe
```

## Utterance language

Technical name: `utterance_language`

Detect the language of a text and return its ISO language code

| Parameter          | Name                   | Required | Default | Description                                                                      |
| ------------------ | ---------------------- | -------- | ------- | -------------------------------------------------------------------------------- |
| Require language   | `require_language`     | yes      |         | Expected language code to match against the detected language                    |
| Fail on match      | `fail_on_match`        | no       | `False` | Whether to fail when detected language matches the expected language             |
| Return ISO code    | `return_lang_iso_code` | no       | `False` | Whether to return the detected language ISO code instead of matching requirement |
| Possible languages | `possible_languages`   | no       |         | List of ISO codes to limit language detection                                    |
| Custom input       | `custom_input`         | no       |         | Custom text to detect language for instead of the last utterance                 |

**Example input:** utterance='ahoj svete' require\_language='cs' fail\_on\_match=False

**Example output:**

```
cs
```

{% hint style="info" %}
**Detecting the spoken language in calls** is not a tool: turn on **Enable language detection** in the Answer node (**Configuration** › **Listening**). With the next input, the variables `detected_language_code` (for example `de-DE`), `detected_language` (`de`), `detected_language_confidence` and `detected_language_utterance` are set. In the flow it is stored as the variable `stt_detect_language`.
{% endhint %}
