Abstract

Explain the work content

Wrote a base pytorch Model deployment file framework , The framework can reconcile the contradiction between the rapid experiment of Algorithm Engineers and the deployment of large-scale service model

Innovation points

1. Provide simple and extensible interfaces for model components
2. utilize pytorch Export the model for import caff2 Model to infer
3. Build multiple single task models at the same time , Through the above modules , Determine the shared model parameters , Conduct joint research , The joint training model can be represented by one or more inputs for multiple tasks

Introduction

Working background

(1) When building the framework of machine learning model , Especially in the framework of deep learning model , There is a big contradiction between experiment and deployment
(2) During the experiment , Usually use pytorch and tensorflow, The advantage is fast modeling , Establish dynamic graph model , The disadvantage is that the inference process is delayed , Deployment is challenging （ In fact, I don't know much about inference , What is deployment ）
When the model is deployed , Use a declarative framework , use caffe2 and tensorflow frame , Establish static graph model , The advantage is that the production scenario is highly optimized , But it is difficult to use and the experimental life cycle is longer .
(3) The current contradiction is because most NLP The model is a dynamic model , It is not easy to represent... In a static diagram , It increases the challenge of opening various services on the basis of industrial scale .

Innovation points

1. Fast modeling
2. It becomes easy to use models on new data , Reduce additional workload
3. Researchers and workers on the same production line , Create with minimal cost 、 assessment 、 The whole process from model delivery to deployment .
4. Ensure high performance when deploying the model ( Low latency and high throughput )

Related work

Introduction to related research algorithms , Analyze the advantages and disadvantages .
Different model frameworks advantage shortcoming
CoreNLP Support research and production Deep learning neural networks are not supported
FLAIR Support research and deep learning models Production is not supported
AllenNLP Support research and deep learning models Production is not supported
Spacy 2.0 Support deep learning model and production Modeling is not supported

Discussion

Paper introduction

PyText It is a framework to help researchers or engineers quickly deploy machine learning systems , With custom processing raw data model 、 Reporting indicators 、 Training methods 、 Output training model method and other functions .
At this point, you can study the building blocks of various models , Conduct in-depth research .

Source structure

All components need to be maintained in the global registry , Defined by the parameters required to configure it
Component structure = Training (Data Handler/Model/Optimizer/Metric Reporter/Trainer)+ forecast (Predictor)+ export (Exporter), All models are exported as ONXX Format

Model structure

Single task training model
Token Embedding+Representation+Decoder+Output Layer
Multi task training model
Build multiple single task models at the same time , Through the above modules , Determine the shared model parameters , Conduct joint research , The joint training model can be represented by one or more inputs for multiple tasks

Related preconditions of multitasking training model

1. Set up different data sources , Different data sources correspond to different tasks
2. A single task scenario must be implemented , Make sure you reuse it in your multitasking setup .

An example of multitasking training model
Variables can be represented by shared words , Realization (1) Text classification and word marking model ,(2) Two text categories + Text matching model

The model can support tasks

1. Text classification
2. Word marker
3. Semantic analysis
4. Language model
5. Joint model

Experiments

The experimental steps

PyText Deployment tasks

1. Implementation model
2. Bound to the pytorch Reasoning services for , Real time Small-Scale Evaluation of real-time traffic samples
3. Export to caffe2 Model , But it does not necessarily support overly complex versions
4. If 3 Step no , Then use pytorch Of C++ API, Rewrite the model and encapsulate it into Caffe2 Format
5. Publish to Caffe2 In our forecasting service , Perform training operations

experimental condition
The experiment is on a computer with only cpu On your machine , There's... On the machine 48 individual 2.5GHz Of Intel Xeon E5-2680 processor ,251 GB RAM and CentOS 7.5.c++ Code using gcc -O3 compile

Model deployment
The model studies pytext Export to C++ and caffe2 in , Export as an attempt to identify the delay in the task

Result analysis

Challenges of model deployment
Text preprocessing ：

1. Text tokenization
2. Character conversion ID
3. string alignment

Vocabulary management methods

1. Maintain vocabulary in remote services
2. Export to Caffe2 after , Add vocabulary to the network

The second option is to synchronize / Control different versions of use cases and language models

Future research direction

1. Add tasks

(1) Add multilingual task modeling research
(2) Add question and answer system 、 Summarize the system 、 Research on reading comprehension tasks

2. Performance benchmarks and corresponding improvements

In order to improve high throughput , Reduce delay time
(1) Training ： Add distributed training
(2) forecast ： Benchmark your training , Tune the model deployment according to the predicted load pattern .

3. Model interpretability

Monitoring indicators , Debug the inside of the model , Add more tool support
(1) tensorboard+visdom
(2) Explore different model interpretation methods

4. Model robustness

In order to resist the antagonistic input of disturbance 、 noise 、 The influence of grammatical differences on the accuracy of the model , Add countermeasure training and data enhancement technology .

5. Mobile model deployment

Training lightweight models , Deploy models on mobile devices