Load text: https://tensorflow.google.cn/tutorials/load_data/text
This tutorial demonstrates two ways to load and preprocess text.
- First, you will use
Kerasutilities and preprocessing layers. These include:
tf.keras.utils.text_dataset_from_directoryto turn data into atf.data.Datasetandtf.keras.layers.TextVectorizationfor data standardization, tokenization, and vectorization. If you are new to TensorFlow, you should start with these. - Then, you will use lower-level utilities like
tf.data.TextLineDatasetto load text files,tf.lookupfor custom in-model lookup tables, andTensorFlow TextAPIs, such astext.UnicodeScriptTokenizerandtext.case_fold_utf8, to preprocess the data for finer-grain control.

浙公网安备 33010602011771号