Spark NLP

最先进的自然语言处理技术。(State of the Art Natural Language Processing)

Github stars Tracking Chart

Spark NLP:最先进的自然语言处理

Spark NLP 是一个建立在 Apache Spark ML 之上的自然语言处理库。它为机器学习管线提供了简单、高性能和准确的 NLP 注释,可在分布式环境中轻松扩展。Spark NLP 带有 330 多个预训练的管道和模型,使用超过 46 种语言。它支持最先进的变换器,如 BERT、XLNet、ELMO、ALBERT 和通用句子编码器(Universal Sentence Encoder),可以在集群中无缝使用。此外,它还提供了时序化、语音部分标记、命名实体识别、依赖性解析、拼写检查、多类文本分类、多类情感分析等更多 NLP 任务。

项目网站

查看我们的官方 Spark NLP 页面:http://nlp.johnsnowlabs.com/,以获取用户文档和示例

社区支持

  • Slack -- 与Spark NLP社区和团队进行实时讨论
  • GitHub -- Bug 报告,功能请求和贡献
  • Medium -- Spark NLP 文章
  • YouTube -- Spark NLP 视频教程

特性

  • 分词(Tokenization)
  • 停止单词删除(Stop Words Removal)
  • 归一化(Normalizer)
  • 词干(Stemmer)
  • 词形还原工具(Lemmatizer)
  • n元语法(NGrams)
  • 正则表达式匹配(Regex Matching)
  • 文字匹配(Text Matching)
  • 组块(Chunking)
  • 日期匹配器(Date Matcher)
  • 句子检测器(Sentence Detector)
  • 词性标记(Part-of-speech tagging)
  • 情绪检测(ML 模型)
  • 拼写检查器(ML 和 DL 模型)
  • 词嵌入(GloVe和Word2Vec)
  • BERT 嵌入(TF Hub 模型)
  • ELMO嵌入(TF Hub 模型)
  • ALBERT嵌入(TF Hub 模型)
  • XLNet嵌入
  • 通用语句编码器(TF Hub 模型)
  • 句子嵌入
  • 块嵌入
  • 语言检测/识别
  • 多类别情感分析(深度学习)
  • 多类文本分类(深度学习)
  • 命名实体识别(深度学习)
  • 依赖项解析(标记/未标记)
  • 轻松整合 TensorFlow
  • GPU 支持
  • 与 Spark ML 功能完全集成
  • +90种预训练的模型(21种语言)!
  • 提供10种语言的+67条预训练管道!
  • 多语言 NER 模型:荷兰语,英语,法语,德语,意大利语,挪威语,波兰语,葡萄牙语,俄语,西班牙语

要求

为了使用 Spark NLP,您需要满足以下要求:

  • Java 8
  • Apache Spark 2.4.x (or Apache Spark 2.3.x)

快速开始

这是一个如何在 Python 和 PySpark 中使用 Spark NLP 预训练管道的快速示例:

$ java -version
# should be Java 8 (Oracle or OpenJDK)
$ conda create -n sparknlp python=3.6 -y
$ conda activate sparknlp
$ pip install spark-nlp==2.5.4 pyspark==2.4.4

在 Python 控制台或 Jupyter Python3 内核中:

# Import Spark NLP
from sparknlp.base import *
from sparknlp.annotator import *
from sparknlp.pretrained import PretrainedPipeline
import sparknlp
# Start Spark Session with Spark NLP
# start() functions has two parameters: gpu and spark23
# sparknlp.start(gpu=True) will start the session with GPU support
# sparknlp.start(sparrk23=True) is when you have Apache Spark 2.3.x installed
spark = sparknlp.start()
# Download a pre-trained pipeline
pipeline = PretrainedPipeline('explain_document_dl', lang='en')
# Your testing dataset
text = """
The Mona Lisa is a 16th century oil painting created by Leonardo.
It's held at the Louvre in Paris.
"""
# Annotate your testing dataset
result = pipeline.annotate(text)
# What's in the pipeline
list(result.keys())
Output: ['entities', 'stem', 'checked', 'lemma', 'document',
'pos', 'token', 'ner', 'embeddings', 'sentence']
# Check the results
result['entities']
Output: ['Mona Lisa', 'Leonardo', 'Louvre', 'Paris']

有关更多示例,您可以访问我们的专用存储库以展示所有 Spark NLP 用例!

Apache Spark 支持

Spark NLP 2.5.4 已基于 Apache Spark 2.4.x 构建,并完全支持 Apache Spark 2.3.x:

Spark NLP Apache Spark 2.3.x Apache Spark 2.4.x
2.5.4 YES YES
2.4.x Partially YES
1.8.x Partially YES
1.7.x YES NO
1.6.x YES NO
1.5.x YES NO

注意:从 2.5.4 版本开始,我们同时支持 Apache Spark 2.4.x 和 Apache Spark2.3.x。

从我们的发布说明中查找有关 Spark NLP 版本的更多信息。

Databricks 支持

Spark NLP 2.5.4 已通过测试,并与以下运行时兼容:

  • 6.2
  • 6.2 ML
  • 6.3
  • 6.3 ML
  • 6.4
  • 6.4 ML
  • 6.5
  • 6.5 ML

EMR 支持

  • 5.26.0
  • 5.27.0

完整的 EMR 发布清单。

用法

(恕删略。请参考自述文件。)

示例

需要更多示例吗?请查看我们的专用存储库,以展示所有 Spark NLP 用例!

所有示例:spark-nlp-workshop

常问问题

在此处查看我们的文章和常见问题解答页面

致谢

特殊社区致谢

总体而言,感谢社区最近报告了重要问题并提出了错误修正请求。社区一直是最新版本中的关键,它在各种基于 Spark 的环境中提供反馈。

这里有一些重复提及的反馈和闲散的参与

  • @maziyarpanahi -- 为测试和有价值的反馈做出贡献。
  • @easimadi -- 提供文档和有价值的反馈。

贡献

我们感谢任何形式的贡献:

  • 创意
  • 反馈
  • 文件资料
  • 错误报告
  • NLP 培训和测试语料库
  • 开发和测试

克隆仓库并提交您的拉取请求! 或直接在此仓库中创建问题。

联系

nlp@johnsnowlabs.com

John Snow Labs

http://johnsnowlabs.com


(The first version translated by vz on 2020.07.19)

Overview

Name With OwnerJohnSnowLabs/spark-nlp
Primary LanguageScala
Program languageScala (Language Count: 6)
PlatformDocker, Linux, Mac, Windows
License:Apache License 2.0
Release Count143
Last Release Name5.3.2 (Posted on )
First Release Namebenchmark-0.0.1 (Posted on )
Created At2017-09-24 19:36:44
Pushed At2024-03-20 21:47:17
Last Commit At2024-03-20 20:18:06
Stargazers Count3.6k
Watchers Count100
Fork Count696
Commits Count8.5k
Has Issues Enabled
Issues Count856
Issue Open Count39
Pull Requests Count12018
Pull Requests Open Count11
Pull Requests Close Count1134
Has Wiki Enabled
Is Archived
Is Fork
Is Locked
Is Mirror
Is Private

Spark NLP: State of the Art Natural Language Processing

Build Status Maven Central PyPI version Anaconda-Cloud License

Spark NLP is a Natural Language Processing library built on top of Apache Spark ML. It provides simple, performant & accurate NLP annotations for machine learning pipelines that scale easily in a distributed environment. Spark NLP comes with 200+ pretrained pipelines and models in more than 45+ languages. It supports state-of-the-art transformers such as BERT, XLNet, ELMO, ALBERT, and Universal Sentence Encoder that can be used seamlessly in a cluster. It also offers Tokenization, Part-of-Speech Tagging, Named Entity Recognition, Dependency Parsing, Spell Checking, Multi-class text classification, Multi-class sentiment analysis, and many more NLP tasks.

Project's website

Take a look at our official Spark NLP page: http://nlp.johnsnowlabs.com/ for user documentation and examples

Community support

  • Slack For live discussion with the Spark NLP community and the team
  • GitHub Bug reports, feature requests, and contributions
  • Medium Spark NLP articles
  • YouTube Spark NLP video tutorials

Table of contents

Features

  • Tokenization
  • Stop Words Removal
  • Normalizer
  • Stemmer
  • Lemmatizer
  • NGrams
  • Regex Matching
  • Text Matching
  • Chunking
  • Date Matcher
  • Sentence Detector
  • Part-of-speech tagging
  • Sentiment Detection (ML models)
  • Spell Checker (ML and DL models)
  • Word Embeddings (GloVe and Word2Vec)
  • BERT Embeddings (TF Hub models)
  • ELMO Embeddings (TF Hub models)
  • ALBERT Embeddings (TF Hub models)
  • XLNet Embeddings
  • Universal Sentence Encoder (TF Hub models)
  • Sentence Embeddings
  • Chunk Embeddings
  • Language Detection / Identification
  • Multi-class Sentiment analysis (Deep learning)
  • Multi-class Text Classification (Deep learning)
  • Named entity recognition (Deep learning)
  • Dependency parsing (Labeled/unlabled)
  • Easy TensorFlow integration
  • GPU Support
  • Full integration with Spark ML functions
  • +90 pre-trained models in 21 languages!
  • +67 pre-trained pipelines in 10 languages!
  • Multi-lingual NER models: Dutch, English, French, German, Italian, Norwegian, Polish, Portuguese, Russian, Spanish

Requirements

In order to use Spark NLP you need the following requirements:

  • Java 8
  • Apache Spark 2.4.x (or Apache Spark 2.3.x)

Quick Start

This is a quick example of how to use Spark NLP pre-trained pipeline in Python and PySpark:

$ java -version
# should be Java 8 (Oracle or OpenJDK)
$ conda create -n sparknlp python=3.6 -y
$ conda activate sparknlp
$ pip install spark-nlp==2.5.4 pyspark==2.4.4

In Python console or Jupyter Python3 kernel:

# Import Spark NLP
from sparknlp.base import *
from sparknlp.annotator import *
from sparknlp.pretrained import PretrainedPipeline
import sparknlp

# Start Spark Session with Spark NLP
# start() functions has two parameters: gpu and spark23
# sparknlp.start(gpu=True) will start the session with GPU support
# sparknlp.start(sparrk23=True) is when you have Apache Spark 2.3.x installed
spark = sparknlp.start()

# Download a pre-trained pipeline
pipeline = PretrainedPipeline('explain_document_dl', lang='en')

# Your testing dataset
text = """
The Mona Lisa is a 16th century oil painting created by Leonardo.
It's held at the Louvre in Paris.
"""

# Annotate your testing dataset
result = pipeline.annotate(text)

# What's in the pipeline
list(result.keys())
Output: ['entities', 'stem', 'checked', 'lemma', 'document',
'pos', 'token', 'ner', 'embeddings', 'sentence']

# Check the results
result['entities']
Output: ['Mona Lisa', 'Leonardo', 'Louvre', 'Paris']

For more examples, you can visit our dedicated repository to showcase all Spark NLP use cases!

Apache Spark Support

Spark NLP 2.5.4 has been built on top of Apache Spark 2.4.x and fully supports Apache Spark 2.3.x:

Spark NLP Apache Spark 2.3.x Apache Spark 2.4.x
2.5.4 YES YES
2.4.x Partially YES
1.8.x Partially YES
1.7.x YES NO
1.6.x YES NO
1.5.x YES NO

NOTE: Starting 2.5.4 release, we support both Apache Spark 2.4.x and Apache Spark 2.3.x at the same time.

Find out more about Spark NLP versions from our release notes.

Databricks Support

Spark NLP 2.5.4 has been tested and is compatible with the following runtimes:

  • 6.2
  • 6.2 ML
  • 6.3
  • 6.3 ML
  • 6.4
  • 6.4 ML
  • 6.5
  • 6.5 ML

EMR Support

Spark NLP 2.5.4 has been tested and is compatible with the following EMR releases:

  • 5.26.0
  • 5.27.0

Full list of EMR releases.

Usage

Spark Packages

Command line (requires internet connection)

This library has been uploaded to the spark-packages repository.

The benefit of spark-packages is that makes it available for both Scala-Java and Python

To use the most recent version on Apache Spark 2.4.x just add the --packages com.johnsnowlabs.nlp:spark-nlp_2.11:2.5.4 to you spark command:

spark-shell --packages com.johnsnowlabs.nlp:spark-nlp_2.11:2.5.4
pyspark --packages com.johnsnowlabs.nlp:spark-nlp_2.11:2.5.4
spark-submit --packages com.johnsnowlabs.nlp:spark-nlp_2.11:2.5.4

This can also be used to create a SparkSession manually by using the spark.jars.packages option in both Python and Scala.

NOTE: To use Spark NLP with GPU you can use the dedicated GPU package com.johnsnowlabs.nlp:spark-nlp-gpu_2.11:2.5.4

NOTE: To use Spark NLP on Apache Spark 2.3.x you should instead use the following packages:

  • CPU: com.johnsnowlabs.nlp:spark-nlp-spark23_2.11:2.5.4
  • GPU: com.johnsnowlabs.nlp:spark-nlp-gpu-spark23_2.11:2.5.4

Scala

Our package is deployed to maven central. To add this package as a dependency in your application:

Maven

spark-nlp on Apacahe Spark 2.4.x:

<!-- https://mvnrepository.com/artifact/com.johnsnowlabs.nlp/spark-nlp -->
<dependency>
    <groupId>com.johnsnowlabs.nlp</groupId>
    <artifactId>spark-nlp_2.11</artifactId>
    <version>2.5.4</version>
</dependency>

spark-nlp-gpu:

<!-- https://mvnrepository.com/artifact/com.johnsnowlabs.nlp/spark-nlp-gpu -->
<dependency>
    <groupId>com.johnsnowlabs.nlp</groupId>
    <artifactId>spark-nlp-gpu_2.11</artifactId>
    <version>2.5.4</version>
</dependency>

spark-nlp on Apacahe Spark 2.3.x:

<!-- https://mvnrepository.com/artifact/com.johnsnowlabs.nlp/spark-nlp-spark23 -->
<dependency>
    <groupId>com.johnsnowlabs.nlp</groupId>
    <artifactId>spark-nlp-spark23_2.11</artifactId>
    <version>2.5.4</version>
</dependency>

spark-nlp-gpu:

<!-- https://mvnrepository.com/artifact/com.johnsnowlabs.nlp/spark-nlp-gpu-spark23 -->
<dependency>
    <groupId>com.johnsnowlabs.nlp</groupId>
    <artifactId>spark-nlp-gpu-spark23_2.11</artifactId>
    <version>2.5.4</version>
</dependency>

SBT

spark-nlp on Apacahe Spark 2.4.x:

// https://mvnrepository.com/artifact/com.johnsnowlabs.nlp/spark-nlp
libraryDependencies += "com.johnsnowlabs.nlp" %% "spark-nlp" % "2.5.4"

spark-nlp-gpu:

// https://mvnrepository.com/artifact/com.johnsnowlabs.nlp/spark-nlp-gpu
libraryDependencies += "com.johnsnowlabs.nlp" %% "spark-nlp-gpu" % "2.5.4"

spark-nlp on Apacahe Spark 2.3.x:

// https://mvnrepository.com/artifact/com.johnsnowlabs.nlp/spark-nlp-spark23
libraryDependencies += "com.johnsnowlabs.nlp" %% "spark-nlp-spark23" % "2.5.4"

spark-nlp-gpu:

// https://mvnrepository.com/artifact/com.johnsnowlabs.nlp/spark-nlp-gpu-spark23
libraryDependencies += "com.johnsnowlabs.nlp" %% "spark-nlp-gpu-spark23" % "2.5.4"

Maven Central: https://mvnrepository.com/artifact/com.johnsnowlabs.nlp

Python

Python without explicit Pyspark installation

Pip/Conda

If you installed pyspark through pip/conda, you can install spark-nlp through the same channel.

Pip:

pip install spark-nlp==2.5.4

Conda:

conda install -c johnsnowlabs spark-nlp

PyPI spark-nlp package / Anaconda spark-nlp package

Then you'll have to create a SparkSession either from Spark NLP:

import sparknlp

spark = sparknlp.start()

or manually:

spark = SparkSession.builder \
    .appName("Spark NLP")\
    .master("local[4]")\
    .config("spark.driver.memory","16G")\
    .config("spark.driver.maxResultSize", "2G") \
    .config("spark.jars.packages", "com.johnsnowlabs.nlp:spark-nlp_2.11:2.5.4")\
    .config("spark.kryoserializer.buffer.max", "1000M")\
    .getOrCreate()

If using local jars, you can use spark.jars instead for a comma delimited jar files. For cluster setups, of course you'll have to put the jars in a reachable location for all driver and executor nodes.

Quick example:

import sparknlp
from sparknlp.pretrained import PretrainedPipeline

#create or get Spark Session

spark = sparknlp.start()

sparknlp.version()
spark.version

#download, load, and annotate a text by pre-trained pipeline

pipeline = PretrainedPipeline('recognize_entities_dl', 'en')
result = pipeline.annotate('The Mona Lisa is a 16th century oil painting created by Leonardo')

Compiled JARs

Build from source

spark-nlp

  • FAT-JAR for CPU on Apache Spark 2.4.x
sbt assembly
  • FAT-JAR for GPU on Apache Spark 2.4.x
sbt -Dis_gpu=true assembly
  • FAT-JAR for CPU on Apache Spark 2.3.x
sbt -Dis_spark23=true assembly
  • FAT-JAR for GPU on Apache Spark 2.3.x
sbt -Dis_gpu=true -Dis_spark23=true assembly

Using the jar manually

If for some reason you need to use the JAR, you can either download the Fat JARs provided here or download it from Maven Central.

To add JARs to spark programs use the --jars option:

spark-shell --jars spark-nlp.jar

The preferred way to use the library when running spark programs is using the --packages option as specified in the spark-packages section.

Apache Zeppelin

Use either one of the following options

  • Add the following Maven Coordinates to the interpreter's library list
com.johnsnowlabs.nlp:spark-nlp_2.11:2.5.4
  • Add path to pre-built jar from here in the interpreter's library list making sure the jar is available to driver path

Python in Zeppelin

Apart from previous step, install python module through pip

pip install spark-nlp==2.5.4

Or you can install spark-nlp from inside Zeppelin by using Conda:

python.conda install -c johnsnowlabs spark-nlp

Configure Zeppelin properly, use cells with %spark.pyspark or any interpreter name you chose.

Finally, in Zeppelin interpreter settings, make sure you set properly zeppelin.python to the python you want to use and install the pip library with (e.g. python3).

An alternative option would be to set SPARK_SUBMIT_OPTIONS (zeppelin-env.sh) and make sure --packages is there as shown earlier, since it includes both scala and python side installation.

Jupyter Notebook (Python)

The easiest way to get this done is by making Jupyter Notebook run using pyspark as follows:

export SPARK_HOME=/path/to/your/spark/folder
export PYSPARK_PYTHON=python3
export PYSPARK_DRIVER_PYTHON=jupyter
export PYSPARK_DRIVER_PYTHON_OPTS=notebook

pyspark --packages com.johnsnowlabs.nlp:spark-nlp_2.11:2.5.4

Alternatively, you can mix in using --jars option for pyspark + pip install spark-nlp

If not using pyspark at all, you'll have to run the instructions pointed here

Google Colab Notebook

Google Colab is perhaps the easiest way to get started with spark-nlp. It requires no installation or set up other than having a Google account.

Run the following code in Google Colab notebook and start using spark-nlp right away.

import os

# Install java
! apt-get install -y openjdk-8-jdk-headless -qq > /dev/null
os.environ["JAVA_HOME"] = "/usr/lib/jvm/java-8-openjdk-amd64"
os.environ["PATH"] = os.environ["JAVA_HOME"] + "/bin:" + os.environ["PATH"]
! java -version

# Install pyspark
! pip install --ignore-installed pyspark==2.4.4

# Install Spark NLP
! pip install --ignore-installed spark-nlp==2.5.4

# Quick SparkSession start
import sparknlp
spark = sparknlp.start()

print("Spark NLP version")
sparknlp.version()
print("Apache Spark version")
spark.version

Here is a live demo on Google Colab that performs sentiment analysis and NER using pretrained spark-nlp models.

Databricks Cluster

  1. Create a cluster if you don't have one already

  2. On a new cluster or existing one you need to add the following to the Advanced Options -> Spark tab:

spark.kryoserializer.buffer.max 1000M
spark.serializer org.apache.spark.serializer.KryoSerializer
  1. In Libraries tab inside your cluster you need to follow these steps:

    3.1. Insatll New -> PyPI -> spark-nlp -> Install

    3.2. Install New -> Maven -> Coordinates -> com.johnsnowlabs.nlp:spark-nlp_2.11:2.5.4 -> Install

  2. Now you can attach your notebook to the cluster and use Spark NLP!

S3 Cluster

With no Hadoop configuration

If your distributed storage is S3 and you don't have a standard Hadoop configuration (i.e. fs.defaultFS)
You need to specify where in the cluster distributed storage you want to store Spark NLP's tmp files.
First, decide where you want to put your application.conf file

import com.johnsnowlabs.util.ConfigLoader
ConfigLoader.setConfigPath("/somewhere/to/put/application.conf")

And then we need to put in such application.conf the following content

sparknlp {
  settings {
    cluster_tmp_dir = "somewhere in s3n:// path to some folder"
  }
}

Pipelines and Models

Pipelines

Spark NLP offers more than 70 pre-trained pipelines in 10 languages.

English pipelines:

Pipeline Name Build lang
Explain Document ML explain_document_ml 2.4.0 en
Explain Document DL explain_document_dl 2.4.3 en
Recognize Entities DL recognize_entities_dl 2.4.3 en
Recognize Entities DL recognize_entities_bert 2.4.3 en
OntoNotes Entities Small onto_recognize_entities_sm 2.4.0 en
OntoNotes Entities Large onto_recognize_entities_lg 2.4.0 en
Match Datetime match_datetime 2.4.0 en
Match Pattern match_pattern 2.4.0 en
Match Chunk match_chunks 2.4.0 en
Match Phrases match_phrases 2.4.0 en
Clean Stop clean_stop 2.4.0 en
Clean Pattern clean_pattern 2.4.0 en
Clean Slang clean_slang 2.4.0 en
Check Spelling check_spelling 2.4.0 en
Check Spelling DL check_spelling_dl 2.5.0 en
Analyze Sentiment analyze_sentiment 2.4.0 en
Analyze Sentiment DL analyze_sentimentdl_use_imdb 2.5.0 en
Analyze Sentiment DL analyze_sentimentdl_use_twitter 2.5.0 en
Dependency Parse dependency_parse 2.4.0 en

Quick example:

import com.johnsnowlabs.nlp.pretrained.PretrainedPipeline
import com.johnsnowlabs.nlp.SparkNLP

SparkNLP.version()

val testData = spark.createDataFrame(Seq(
(1, "Google has announced the release of a beta version of the popular TensorFlow machine learning library"),
(2, "Donald John Trump (born June 14, 1946) is the 45th and current president of the United States")
)).toDF("id", "text")

val pipeline = PretrainedPipeline("explain_document_dl", lang="en")

val annotation = pipeline.transform(testData)

annotation.show()
/*
import com.johnsnowlabs.nlp.pretrained.PretrainedPipeline
import com.johnsnowlabs.nlp.SparkNLP
2.5.0
testData: org.apache.spark.sql.DataFrame = [id: int, text: string]
pipeline: com.johnsnowlabs.nlp.pretrained.PretrainedPipeline = PretrainedPipeline(explain_document_dl,en,public/models)
annotation: org.apache.spark.sql.DataFrame = [id: int, text: string ... 10 more fields]
+---+--------------------+--------------------+--------------------+--------------------+--------------------+--------------------+--------------------+--------------------+--------------------+--------------------+--------------------+
| id|                text|            document|               token|            sentence|             checked|               lemma|                stem|                 pos|          embeddings|                 ner|            entities|
+---+--------------------+--------------------+--------------------+--------------------+--------------------+--------------------+--------------------+--------------------+--------------------+--------------------+--------------------+
|  1|Google has announ...|[[document, 0, 10...|[[token, 0, 5, Go...|[[document, 0, 10...|[[token, 0, 5, Go...|[[token, 0, 5, Go...|[[token, 0, 5, go...|[[pos, 0, 5, NNP,...|[[word_embeddings...|[[named_entity, 0...|[[chunk, 0, 5, Go...|
|  2|The Paris metro w...|[[document, 0, 11...|[[token, 0, 2, Th...|[[document, 0, 11...|[[token, 0, 2, Th...|[[token, 0, 2, Th...|[[token, 0, 2, th...|[[pos, 0, 2, DT, ...|[[word_embeddings...|[[named_entity, 0...|[[chunk, 4, 8, Pa...|
+---+--------------------+--------------------+--------------------+--------------------+--------------------+--------------------+--------------------+--------------------+--------------------+--------------------+--------------------+
*/

annotation.select("entities.result").show(false)

/*
+----------------------------------+
|result                            |
+----------------------------------+
|[Google, TensorFlow]              |
|[Donald John Trump, United States]|
+----------------------------------+
*/

Please check our dedicated repository for the full list of pre-trained pipelines

Models

Spark NLP offers more than 130 pre-trained models in 45 languages.

List of languages: Afrikaans, Arabic, Armenian, Basque, Bengali, Breton, Bulgarian, Catalan, Czech, Dutch, English, Esperanto, Finnish, French, Galician, German, Greek, Hausa, Hebrew, Hindi, Hungarian, Indonesian, Irish, Italian, Japanese, Latin, Latvian, Marathi, Norwegian, Persian, Polish, Portuguese, Romanian, Russian, Slovak, Slovenian, Somali, Southern Sotho, Spanish, Swahili, Swedish, Tswana, Turkish, Ukrainian, Zulu

English Models:

Model Name Build Lang
LemmatizerModel (Lemmatizer) lemma_antbnc 2.0.2 en
PerceptronModel (POS) pos_anc 2.0.2 en
PerceptronModel (POS UD) pos_ud_ewt 2.2.2 en
NerCrfModel (NER with GloVe) ner_crf 2.4.0 en
NerDLModel (NER with GloVe) ner_dl 2.4.3 en
NerDLModel (NER with BERT) ner_dl_bert 2.4.3 en
NerDLModel (OntoNotes with GloVe 100d) onto_100 2.4.0 en
NerDLModel (OntoNotes with GloVe 300d) onto_300 2.4.0 en
DeepSentenceDetector ner_dl_sentence 2.4.0 en
SymmetricDeleteModel (Spell Checker) spellcheck_sd 2.0.2 en
NorvigSweetingModel (Spell Checker) spellcheck_norvig 2.0.2 en
ViveknSentimentModel (Sentiment) sentiment_vivekn 2.0.2 en
DependencyParser (Dependency) dependency_conllu 2.0.8 en
TypedDependencyParser (Dependency) dependency_typed_conllu 2.0.8 en

Embeddings:

Model Name Build Lang Offline
WordEmbeddings (GloVe) glove_100d 2.4.0 en
BertEmbeddings bert_base_uncased 2.4.0 en
BertEmbeddings bert_base_cased 2.4.0 en
BertEmbeddings bert_large_uncased 2.4.0 en
BertEmbeddings bert_large_cased 2.4.0 en
ElmoEmbeddings elmo 2.4.0 en
UniversalSentenceEncoder (USE) tfhub_use 2.4.0 en
UniversalSentenceEncoder (USE) tfhub_use_lg 2.4.0 en
AlbertEmbeddings albert_base_uncased 2.5.0 en
AlbertEmbeddings albert_large_uncased 2.5.0 en
AlbertEmbeddings albert_xlarge_uncased 2.5.0 en
AlbertEmbeddings albert_xxlarge_uncased 2.5.0 en
XlnetEmbeddings xlnet_base_cased 2.5.0 en
XlnetEmbeddings xlnet_large_cased 2.5.0 en

Classification:

Model Name Build Lang Offline
ClassifierDL (with tfhub_use) classifierdl_use_trec6 2.5.0 en
ClassifierDL (with tfhub_use) classifierdl_use_trec50 2.5.0 en
SentimentDL (with tfhub_use) sentimentdl_use_imdb 2.5.0 en
SentimentDL (with tfhub_use) sentimentdl_use_twitter 2.5.0 en
SentimentDL (with glove_100d) sentimentdl_glove_imdb 2.5.0 en

Quick online example:

# load NER model trained by deep learning approach and GloVe word embeddings
ner_dl = NerDLModel.pretrained('ner_dl')
# load NER model trained by deep learning approach and BERT word embeddings
ner_bert = NerDLModel.pretrained('ner_dl_bert')
// load French POS tagger model trained by Universal Dependencies
val french_pos = PerceptronModel.pretrained("pos_ud_gsd", lang="fr")
// load Italain LemmatizerModel
val italian_lemma = LemmatizerModel.pretrained("lemma_dxc", lang="it")

Quick offline example:

  • Loading PerceptronModel annotator model inside Spark NLP Pipeline
val french_pos = PerceptronModel.load("/tmp/pos_ud_gsd_fr_2.0.2_2.4_1556531457346/")
      .setInputCols("document", "token")
      .setOutputCol("pos")

Please check our dedicated repository for the full list of pre-trained models

Examples

Need more examples? Check out our dedicated repository to showcase all Spark NLP use cases!

All examples: spark-nlp-workshop

FAQ

Check our Articles and FAQ page here

Acknowledgments

Special community acknowledgements

Thanks in general to the community who have been lately reporting important issues and pull request with bugfixes.
The community has been key in the last releases with feedback in various Spark-based environments.

Here a few specific mentions for recurring feedback and slack participation

  • @maziyarpanahi - For contributing with testing and valuable feedback
  • @easimadi - For contributing with documentation and valuable feedback

Contributing

We appreciate any sort of contributions:

  • ideas
  • feedback
  • documentation
  • bug reports
  • NLP training and testing corpora
  • development and testing

Clone the repo and submit your pull-requests! Or directly create issues in this repo.

Contact

nlp@johnsnowlabs.com

John Snow Labs

http://johnsnowlabs.com

To the top