jina的小样本文本分类

https://jina.ai/news/jina-classifier-for-high-performance-zero-shot-and-few-shot-classification/ 

目前精度可以做到90%+，在意图分类中可以做到较为稳定的95%+精度，离商用还有一点差距。

 

Zero-Shot Classification

The Classifier API offers powerful zero-shot classification capabilities, allowing you to categorize text or image without pre-training on labeled data. Every classifier starts with zero-shot capabilities, which can later be enhanced with additional training data or updates - a topic we'll explore in the next section.

Example 1: Route LLM Requests

Here's an example using the classifier API for LLM query routing:

curl https://api.jina.ai/v1/classify \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer YOUR_API_KEY_HERE" \
  -d '{
    "model": "jina-embeddings-v3",
    "labels": [
      "Simple task",
      "Complex reasoning",
      "Creative writing"
    ],
    "input": [
      "Calculate the compound interest on a principal of $10,000 invested for 5 years at an annual rate of 5%, compounded quarterly.",
      "分析使用CRISPR基因编辑技术在人类胚胎中的伦理影响。考虑潜在的医疗益处和长期社会后果。",
      "AIが自意識を持つディストピアの未来を舞台にした短編小説を書いてください。人間とAIの関係や意識の本質をテーマに探求してください。",
      "Erklären Sie die Unterschiede zwischen Merge-Sort und Quicksort-Algorithmen in Bezug auf Zeitkomplexität, Platzkomplexität und Leistung in der Praxis.",
      "Write a poem about the beauty of nature and its healing power on the human soul.",
      "Translate the following sentence into French: The quick brown fox jumps over the lazy dog."
    ]
  }'

This example demonstrates the use of jina-embeddings-v3 to route user queries in multiple languages (English, Chinese, Japanese, and German) into three categories, which correspond to three different sizes of LLMs. The API response format is as follows:

{
  "usage": {"total_tokens": 256, "prompt_tokens": 256},
  "data": [
    {"object": "classification", "index": 0, "prediction": "Simple task", "score": 0.35216382145881653},
    {"object": "classification", "index": 1, "prediction": "Complex reasoning", "score": 0.34310275316238403},
    {"object": "classification", "index": 2, "prediction": "Creative writing", "score": 0.3487184941768646},
    {"object": "classification", "index": 3, "prediction": "Complex reasoning", "score": 0.35207709670066833},
    {"object": "classification", "index": 4, "prediction": "Creative writing", "score": 0.3638903796672821},
    {"object": "classification", "index": 5, "prediction": "Simple task", "score": 0.3561534285545349}
  ]
}

The response includes:

• usage: Information about token usage.
• data: An array of classification results, one for each input. Each result contains the predicted label (prediction) and a confidence score (score). The score for each class is computed via softmax normalization - for zero-shot it's based on cosine similarities between input and label embeddings under classification task-LoRA; while for few-shot it's based on learned linear transformations of the input embedding for each class - resulting in probabilities that sum to 1 across all classes.The index corresponds to the position of the input in the original request.

 

Few-Shot Classification

Few-shot classification offers an easy approach to creating and updating classifiers with minimal labeled data. This method provides two primary endpoints: train and classify.

The train endpoint lets you create or update a classifier with a small set of examples. Your first call to train will return a classifier_id, which you can use for subsequent training whenever you have new data, notice changes in data distribution, or need to add new classes. This flexible approach allows your classifier to evolve over time, adapting to new patterns and categories without starting from scratch.

Similar to zero-shot classification, you'll use the classify endpoint for making predictions. The main difference is that you'll need to include your classifier_id in the request, but you won't need to provide candidate labels since they're already part of your trained model.

tagExample: Train a Support Ticket Assigner

Let's explore these features through an example of classifying customer support tickets for assignment to different teams in a rapidly growing tech startup.

Initial Training

curl -X 'POST' \
  'https://api.jina.ai/v1/train' \
  -H 'accept: application/json' \
  -H 'Authorization: Bearer YOUR_API_KEY_HERE' \
  -H 'Content-Type: application/json' \
  -d '{
  "model": "jina-embeddings-v3",
  "access": "private",
  "input": [
    {
      "text": "I cant log into my account after the latest app update.",
      "label": "team1"
    },
    {
      "text": "My subscription renewal failed due to an expired credit card.",
      "label": "team2"
    },
    {
      "text": "How do I export my data from the platform?",
      "label": "team3"
    }
  ],
  "num_iters": 10
}'

Note that in few-shot learning, we are free to use team1 team2 as the class labels even though they don't have inherent semantic meaning. In the response, you will get a classifier_id that represents this newly created classifier.

{
  "classifier_id": "918c0846-d6ae-4f34-810d-c0c7a59aee14",
  "num_samples": 3,
}

 

Note the classifier_id down, you will need it to refer to this classifier later.

Updating Classifier to Adapt Team Restructuring

As the example company grows, new types of issues emerge, and the team structure also changes. The beauty of few-shot classification lies in its ability to quickly adapt to these changes. We can easily update the classifier giving classifier_id and new examples, introducing new team categories (e.g. team4) or reassigning existing issue types to different teams as the organization evolves.

curl -X 'POST' \
  'https://api.jina.ai/v1/train' \
  -H 'accept: application/json' \
  -H 'Authorization: Bearer YOUR_API_KEY_HERE' \
  -H 'Content-Type: application/json' \
  -d '{
  "classifier_id": "b36b7b23-a56c-4b52-a7ad-e89e8f5439b6",
  "input": [
    {
      "text": "Im getting a 404 error when trying to access the new AI chatbot feature.",
      "label": "team4"
    },
    {
      "text": "The latest security patch is conflicting with my company firewall.",
      "label": "team1"
    },
    {
      "text": "I need help setting up SSO for my organization account.",
      "label": "team5"
    }
  ],
  "num_iters": 10
}'

Using a Trained Classifier

During inference, you only need to provide the input text and the classifier_id. The API handles the mapping between your input and the previously trained classes, returning the most appropriate label based on the classifier's current state.

curl -X 'POST' \
  'https://api.jina.ai/v1/classify' \
  -H 'accept: application/json' \
  -H 'Authorization: Bearer YOUR_API_KEY_HERE' \
  -H 'Content-Type: application/json' \
  -d '{
  "classifier_id": "b36b7b23-a56c-4b52-a7ad-e89e8f5439b6",
  "input": [
    {
      "text": "The new feature is causing my dashboard to load slowly."
    },
    {
      "text": "I need to update my billing information for tax purposes."
    }
  ]
}'

The few-shot mode has two unique parameters.

Parameter num_iters

The num_iters parameter tunes how intensively the classifier learns from your training examples. While the default value of 10 works well for most cases, you can strategically adjust this value based on your confidence in the training data. For high-quality examples that are crucial for classification, increase num_iters to reinforce their importance. Conversely, for less reliable examples, lower num_iters to minimize their impact on the classifier's performance. This parameter can also be used to implement time-aware learning, where more recent examples get higher iteration counts to adapt to evolving patterns while maintaining historical knowledge.

Parameter access

The access parameter lets you control who can use your classifier. By default, classifiers are private and only accessible to you. Setting access to "public" allows anyone with your classifier_id to use it with their own API key and token quota. This enables sharing classifiers while maintaining privacy - users can't see your training data or configuration, and you can't see their classification requests. This parameter is only relevant for few-shot classification, as zero-shot classifiers are stateless. There's no need to share zero-shot classifiers since identical requests will always yield the same responses regardless of who makes them.

Remarks on Few-Shot Learning

Few-shot classification in our API has some unique characteristics worth noting. Unlike traditional machine learning models, our implementation uses one-pass online learning - training examples are processed to update the classifier's weights but aren't stored afterward. This means you can't retrieve historical training data, but it ensures better privacy and resource efficiency.

While few-shot learning is powerful, it does require a warm-up period to outperform zero-shot classification. Our benchmarks show that 200-400 training examples typically provide enough data to see superior performance. However, you don't need to provide examples for all classes upfront - the classifier can scale to accommodate new classes over time. Just be aware that newly added classes may experience a brief cold-start period or class imbalance until sufficient examples are provided.

Benchmark

For our benchmark analysis, we evaluated zero-shot and few-shot approaches across diverse datasets, including text classification tasks like emotion detection (6 classes) and spam detection (2 classes), as well as image classification tasks like CIFAR10 (10 classes). The evaluation framework used standard train-test splits, with zero-shot requiring no training data and few-shot using portions of the training set. We tracked key metrics like train size and target class count, allowing for controlled comparisons. To ensure robustness, particularly for few-shot learning, each input went through multiple training iterations. We compared these modern approaches against traditional baselines like Linear SVM and RBF SVM to provide context for their performance.

F1 scores are plotted. For the full benchmark settings, please check out this Google spreadsheet.

The F1 plots reveal interesting patterns across three tasks. Not surprisingly, zero-shot classification shows constant performance from the start, regardless of training data size. In contrast, few-shot learning demonstrates a rapid learning curve, initially starting lower but quickly surpassing zero-shot performance as training data increases. Both methods ultimately achieve comparable accuracy around the 400-sample mark, with few-shot maintaining a slight edge. This pattern holds true for both multi-class and image classification scenarios, suggesting that few-shot learning can be particularly advantageous when some training data is available, while zero-shot offers reliable performance even without any training examples. The table below summarizes the difference between zero-shot and few-shot classification from the API user point of view.

Feature	Zero-shot	Few-shot
Primary Use Case	Default solution for general classification	For data outside v3/clip-v1's domain or time-sensitive data
Training Data Required	No	Yes
Labels Required in /train	N/A	Yes
Labels Required in /classify	Yes	No
Classifier ID Required	No	Yes
Semantic Labels Required	Yes	No
State Management	Stateless	Stateful
Continuous Model Updates	No	Yes
Access Control	No	Yes
Maximum Classes	256	16
Maximum Classifiers	N/A	16
Maximum Inputs per Request	1,024	1,024
Maximum Token Length per Input	8,192 tokens	8,192 tokens

Summary

The Classifier API offers powerful zero-shot and few-shot classification for both text and image content, powered by advanced embedding models like jina-embeddings-v3 and jina-clip-v1. Our benchmarks show that zero-shot classification provides reliable performance without training data, making it an excellent starting point for most tasks with support for up to 256 classes. While few-shot learning can achieve slightly better accuracy with training data, we recommend beginning with zero-shot classification for its immediate results and flexibility.

The API's versatility supports various applications, from routing LLM queries to detecting website accessibility and categorizing multilingual content. Whether you're starting with zero-shot or transitioning to few-shot learning for specialized cases, the API maintains a consistent interface for seamless integration into your pipeline. We're particularly excited to see how developers will leverage this API in their applications, and we'll be rolling out support for new embedding models like jina-clip-v2 in the future.