大型数据集的 TFIDF
问题描述
我有一个包含大约 800 万篇新闻文章的语料库，我需要将它们的 TFIDF 表示为稀疏矩阵.对于相对较少数量的样本，我已经能够使用 scikit-learn 做到这一点，但我相信它不能用于如此庞大的数据集，因为它首先将输入矩阵加载到内存中，这是一个昂贵的过程.
I have a corpus which has around 8 million news articles, I need to get the TFIDF representation of them as a sparse matrix. I have been able to do that using scikit-learn for relatively lower number of samples, but I believe it can't be used for such a huge dataset as it loads the input matrix into memory first and that's an expensive process.
有谁知道，为大型数据集提取 TFIDF 向量的最佳方法是什么?
Does anyone know, what would be the best way to extract out the TFIDF vectors for large datasets?
解决方案
Gensim 有一个高效的 tf-idf 模型 并且不需要一次将所有内容都保存在内存中.
Gensim has an efficient tf-idf model and does not need to have everything in memory at once.
您的语料库只需要是一个可迭代的，因此它不需要一次将整个语料库保存在内存中.
Your corpus simply needs to be an iterable, so it does not need to have the whole corpus in memory at a time.
make_wiki 脚本在 Wikipedia 上运行大约根据评论，50m 在笔记本电脑上.
The make_wiki script runs over Wikipedia in about 50m on a laptop according to the comments.
相关文章