John Snow Labs Spark-NLP 3.2.2: Models Hub for the community by the community, new RoBERTa and XLM-RoBERTa Sentence Embeddings, 40 new models in 20 languages, bug fixes, and more!

[maziyarpanahi]

---

Overview

We are pleased to release Spark NLP 🚀 3.2.2! This release comes with accessible Models Hub to our community to host their models and pipelines for free, new RoBERTa and XLM-RoBERTa Sentence Embeddings, over 40 new models and pipelines in 20+ languages, bug fixes, and more

As always, we would like to thank our community for their feedback, questions, and feature requests.

---

New Features

• A new RoBertaSentenceEmbeddings annotator for sentence embeddings used in SentimentDL, ClassifierDL, and MultiClassifierDL annotators
• A new XlmRoBertaSentenceEmbeddings annotator for sentence embeddings used in SentimentDL, ClassifierDL, and MultiClassifierDL annotators
• Add support for AWS MFA via Spark NLP configuration
• Add new AWS configs to Spark NLP configuration when using a private S3 bucket to store logs for training models or access TF graphs needed in NerDLApproach
  • spark.jsl.settings.aws.credentials.access_key_id
  • spark.jsl.settings.aws.credentials.secret_access_key
  • spark.jsl.settings.aws.credentials.session_token
  • spark.jsl.settings.aws.s3_bucket
  • spark.jsl.settings.aws.region

---

Models Hub for the community, by the community

Serve Your Spark NLP Models for Free! You can host and share your Spark NLP models & pipelines publicly with everyone to reuse them with one line of code!

We are opening Models Hub to everyone to upload their models and pipelines, showcase their work, and share them with others.

Please visit the following page for more information: https://modelshub.johnsnowlabs.com/

---

Bug Fixes & Enhancements

• Improve loading merges file for RoBERTa tokenizer
• Remove batchSize param from broadcast in XlmRoBertaEmbeddings to be set after it is created
• Preserve previously generated metadata in BertSentenceEmbeddings annotator
• Set elmo as a default poolingLayer in ElmoEmbeddings
• Fix special tokens ids in XlmRoBertaEmbeddings annotator
• Fix distilbert_base_token_classifier_ontonotes model
• Fix distilbert_base_token_classifier_conll03 model
• Fix distilbert_base_token_classifier_few_nerd model
• Fix distilbert_token_classifier_persian_ner model
• Fix ner_conll_longformer_base_4096 model

---

Models and Pipelines

Spark NLP 3.2.2 comes with new Turkish text classifier pipelines, Expert BERT Word and Sentence embeddings such as wiki books and PubMed, new BERT model for 17 Indian languages, and Sentence Detection models for 15 new languages.

Pipelines

Name	Build	Lang
classifierdl_berturk_cyberbullying_pipeline	3.1.3	tr
classifierdl_bert_news_pipeline	3.1.3	de
classifierdl_electra_questionpair_pipeline	3.2.0	en
classifierdl_bert_news_pipeline	3.2.0	tr

Named Entity Recognition

Model	Name	Build	Lang
NerDLModel	ner_conll_elmo	3.2.2	en
NerDLModel	ner_conll_albert_base_uncased	3.2.2	en
NerDLModel	ner_conll_albert_large_uncased	3.2.2	en
NerDLModel	ner_conll_xlnet_base_cased	3.2.2	en

BERT Embeddings

Model	Name	Build	Lang
BertEmbeddings	bert_muril	3.2.0	xx
BertEmbeddings	bert_wiki_books_sst2	3.2.0	en
BertEmbeddings	bert_wiki_books_squad2	3.2.0	en
BertEmbeddings	bert_wiki_books_qqp	3.2.0	en
BertEmbeddings	bert_wiki_books_qnli	3.2.0	en
BertEmbeddings	bert_wiki_books_mnli	3.2.0	en
BertEmbeddings	bert_wiki_books	3.2.0	en
BertEmbeddings	bert_pubmed_squad2	3.2.0	en
BertEmbeddings	bert_pubmed	3.2.0	en
BertSentenceEmbeddings	sent_bert_wiki_books_sst2	3.2.0	en
BertSentenceEmbeddings	sent_bert_wiki_books_squad2	3.2.0	en
BertSentenceEmbeddings	sent_bert_wiki_books_qqp	3.2.0	en
BertSentenceEmbeddings	sent_bert_wiki_books_qnli	3.2.0	en
BertSentenceEmbeddings	sent_bert_wiki_books_mnli	3.2.0	en
BertSentenceEmbeddings	sent_bert_wiki_books	3.2.0	en
BertSentenceEmbeddings	sent_bert_pubmed_squad2	3.2.0	en
BertSentenceEmbeddings	sent_bert_pubmed	3.2.0	en
BertSentenceEmbeddings	sent_bert_muril	3.2.0	xx

Sentence Detection

Yiddish, Ukrainian, Telugu, Tamil, Somali, Sindhi, Russian, Punjabi, Nepali, Marathi, Malayalam, Kannada, Indonesian, Gujrati, Bosnian

Model	Name	Build	Lang
SentenceDetectorDLModel	sentence_detector_dl	3.2.0	yi
SentenceDetectorDLModel	sentence_detector_dl	3.2.0	uk
SentenceDetectorDLModel	sentence_detector_dl	3.2.0	te
SentenceDetectorDLModel	sentence_detector_dl	3.2.0	ta
SentenceDetectorDLModel	sentence_detector_dl	3.2.0	so
SentenceDetectorDLModel	sentence_detector_dl	3.2.0	sd
SentenceDetectorDLModel	sentence_detector_dl	3.2.0	ru
SentenceDetectorDLModel	sentence_detector_dl	3.2.0	pa
SentenceDetectorDLModel	sentence_detector_dl	3.2.0	ne
SentenceDetectorDLModel	sentence_detector_dl	3.2.0	mr
SentenceDetectorDLModel	sentence_detector_dl	3.2.0	ml
SentenceDetectorDLModel	sentence_detector_dl	3.2.0	kn
SentenceDetectorDLModel	sentence_detector_dl	3.2.0	id
SentenceDetectorDLModel	sentence_detector_dl	3.2.0	gu
SentenceDetectorDLModel	sentence_detector_dl	3.2.0	bs

The complete list of all 3700+ models & pipelines in 200+ languages is available on Models Hub.

---

Documentation

• TF Hub & HuggingFace to Spark NLP
• Models Hub with new models
• Spark NLP in Action
• Spark NLP documentation
• Spark NLP Scala APIs
• Spark NLP Python APIs
• Spark NLP Workshop notebooks
• Spark NLP publications
• Spark NLP training certification notebooks for Google Colab and Databricks
• Spark NLP Display for visualization of different types of annotations
• Discussions Engage with other community members, share ideas, and show off how you use Spark NLP!

---

Installation

Python

#PyPI

pip install spark-nlp==3.2.2

Spark Packages

spark-nlp on Apache Spark 3.0.x and 3.1.x (Scala 2.12 only):

spark-shell --packages com.johnsnowlabs.nlp:spark-nlp_2.12:3.2.2

pyspark --packages com.johnsnowlabs.nlp:spark-nlp_2.12:3.2.2

GPU

spark-shell --packages com.johnsnowlabs.nlp:spark-nlp-gpu_2.12:3.2.2

pyspark --packages com.johnsnowlabs.nlp:spark-nlp-gpu_2.12:3.2.2

spark-nlp on Apache Spark 2.4.x (Scala 2.11 only):

spark-shell --packages com.johnsnowlabs.nlp:spark-nlp-spark24_2.11:3.2.2

pyspark --packages com.johnsnowlabs.nlp:spark-nlp-spark24_2.11:3.2.2

GPU

spark-shell --packages com.johnsnowlabs.nlp:spark-nlp-gpu-spark24_2.11:3.2.2

pyspark --packages com.johnsnowlabs.nlp:spark-nlp-gpu-spark24_2.11:3.2.2

spark-nlp on Apache Spark 2.3.x (Scala 2.11 only):

spark-shell --packages com.johnsnowlabs.nlp:spark-nlp-spark23_2.11:3.2.2

pyspark --packages com.johnsnowlabs.nlp:spark-nlp-spark23_2.11:3.2.2

GPU

spark-shell --packages com.johnsnowlabs.nlp:spark-nlp-spark23-gpu_2.11:3.2.2

pyspark --packages com.johnsnowlabs.nlp:spark-nlp-gpu-spark23_2.11:3.2.2

Maven

spark-nlp on Apache Spark 3.0.x and 3.1.x:

<dependency>
    <groupId>com.johnsnowlabs.nlp</groupId>
    <artifactId>spark-nlp_2.12</artifactId>
    <version>3.2.2</version>
</dependency>

spark-nlp-gpu:

<dependency>
    <groupId>com.johnsnowlabs.nlp</groupId>
    <artifactId>spark-nlp-gpu_2.12</artifactId>
    <version>3.2.2</version>
</dependency>

spark-nlp on Apache Spark 2.4.x:

<dependency>
    <groupId>com.johnsnowlabs.nlp</groupId>
    <artifactId>spark-nlp-spark24_2.11</artifactId>
    <version>3.2.2</version>
</dependency>

spark-nlp-gpu:

<dependency>
    <groupId>com.johnsnowlabs.nlp</groupId>
    <artifactId>spark-nlp-gpu-spark24_2.11</artifactId>
    <version>3.2.2</version>
</dependency>

spark-nlp on Apache Spark 2.3.x:

<dependency>
    <groupId>com.johnsnowlabs.nlp</groupId>
    <artifactId>spark-nlp-spark23_2.11</artifactId>
    <version>3.2.2</version>
</dependency>

spark-nlp-gpu:

<dependency>
    <groupId>com.johnsnowlabs.nlp</groupId>
    <artifactId>spark-nlp-gpu-spark23_2.11</artifactId>
    <version>3.2.2</version>
</dependency>

FAT JARs

• CPU on Apache Spark 3.x: https://s3.amazonaws.com/auxdata.johnsnowlabs.com/public/jars/spark-nlp-assembly-3.2.2.jar

• GPU on Apache Spark 3.x: https://s3.amazonaws.com/auxdata.johnsnowlabs.com/public/jars/spark-nlp-gpu-assembly-3.2.2.jar

• CPU on Apache Spark 2.4.x: https://s3.amazonaws.com/auxdata.johnsnowlabs.com/public/jars/spark-nlp-spark24-assembly-3.2.2.jar

• GPU on Apache Spark 2.4.x: https://s3.amazonaws.com/auxdata.johnsnowlabs.com/public/jars/spark-nlp-gpu-spark24-assembly-3.2.2.jar

• CPU on Apache Spark 2.3.x: https://s3.amazonaws.com/auxdata.johnsnowlabs.com/public/jars/spark-nlp-spark23-assembly-3.2.2.jar

• GPU on Apache Spark 2.3.x: https://s3.amazonaws.com/auxdata.johnsnowlabs.com/public/jars/spark-nlp-gpu-spark23-assembly-3.2.2.jar

---

This discussion was created from the release John Snow Labs Spark-NLP 3.2.2: Models Hub for the community by the community, new RoBERTa and XLM-RoBERTa Sentence Embeddings, 40 new models in 20 languages, bug fixes, and more!.