RewriteFilterPriority HIGH
添加到IirfGlobal.ini。
FieldCache是啥？
我们知道，如果对每一个文档号都用reader 读取域的值会影响速度，所以Lucene 引入了FieldCache 来进行缓存，而FieldCache 并非在存储域中读取，而是在索引域中读取，从而
不必构造Document 对象，然而要求此索引域是不分词的，有且只有一个Token。
讲到这里，我们似乎知道了一些东西，那就是，索引进LUCENE中数据至少会在两个地方有相关的存储，一个就是存储域，一个就是索引域。
即然FieldCache说它是从索引域中读取，速度相对从存储域读取肯定快，为什么不用它呢？
说干就干，经过一番折腾，得到下面的测试结果：
环境：对100W条数据量，取出其中要分组的字段，当然这个字段是不分词索引进去的
第一种方式：从Term区域读取
View Code
Dictionary<int, object> dictionary = new Dictionary<int, object>();
Term startTerm = new Term("CompanyID");
TermEnum te = indexReader.Terms(startTerm);
if (te != null)
{
Term currTerm = te.Term();
while ((currTerm != null) && (currTerm.Field() == startTerm.Field())) //term fieldnames are interned
{
//if (te.DocFreq() > 1)
//{
TermDocs td = indexReader.TermDocs(currTerm);
while (td.Next())
{
dictionary.Add(td.Doc(), currTerm.Text());
}
//}
if (!te.Next())
{
break;
}
currTerm = te.Term();
}
}
取出100W个CompanyID耗时：共耗时20.9291356秒
第二种方式：从FieldCache中读取
View Code
StringIndex stringIndex = FieldCache_Fields.DEFAULT.GetStringIndex(indexReader, "CompanyID");
取出100W个CompanyID耗时：共耗时2.8249935秒
补充：StringIndex有两个属性：lookup、order
string[] lookup：按照字典顺序排列的所有term
int[] order： 其中位置表中文档号，order[i]表示第i个document某个field包含的term在lookup中的位置
获取docid对应的term的值：termValue = lookup[order[doc]]
两者性能一目了然，我也不多说了，快去试试吧！
最后当然也不忘了说一句，如果你的数据量是千万级别或上亿了，那你必须得考虑分布式计算、并行计算这一类的计术了，呵呵。
[by:http://www.cnblogs.com/zengen/archive/2011/04/19/2020681.html]
public Expression<Func<Job, bool>> ToLambda()
{
Type type = typeof (Job);
ParameterExpression parameterExpression = Expression.Parameter(type, "job");
Expression body = Expression.Equal(Expression.Property(parameterExpression, "MemberId"), Expression.Constant(MemberId));
if (Key.IsNotNullAndWhiteSpace())
{
MemberExpression keyMember = Expression.Property(parameterExpression, "JobName");
Expression value = Expression.Constant(Key);
MethodCallExpression keyExpression = Expression.Call(keyMember,
typeof (string).GetMethod("Contains"),
value);
body = Expression.And(body, keyExpression);
}
if(IsOpen.HasValue)
{
MemberExpression isOpenExpression = Expression.Property(parameterExpression, "IsOpen");
Expression openExpression = Expression.Equal(isOpenExpression, Expression.Constant(IsOpen));
body = Expression.And(body, openExpression);
}
return Expression.Lambda<Func<Job, bool>>(body, parameterExpression);
}
Using the Tika Java Library In Your .Net Application With IKVM
This may sound scary and heretical but did you know it is possible to leverage Java libraries from .Net applications with no TCP sockets or web services getting caught in the crossfire? Let me introduce you to IKVM, which is frankly magic:
IKVM.NET is an implementation of Java for Mono and the Microsoft .NET Framework. It includes the following components:
- A Java Virtual Machine implemented in .NET
- A .NET implementation of the Java class libraries
- Tools that enable Java and .NET interoperability
Using IKVM we have been able to successfully integrate our Dovetail Seeker search application with the Tika text extraction library implemented in Java. With Tika we can easily pull text out of rich documents from many supported formats. Why Tika? Because there is nothing comparable in the .Net world as Tika.
This post will review how we integrated with Tika. If you like code you can find this example in a repo up on Github.
Compiling a Jar Into An Assembly
First thing, we need to get our hands on the latest version of Tika. I downloaded and built the Tika source using Maven as instructed. The result of this was a few jar files. The one we are interested in is tika-app-x.x.jar which has everything we need bundled into one useful container.
Next up we need to convert this jar we’ve built to a .Net assembly. Do this using ikvmc.exe.
tika\build>ikvmc.exe -target:library tika-app-0.7.jar
Unfortunately, you will see tons of troublesome looking warnings but the end result is a .Net assembly wrapping the Java jar which you can reference in your projects.
Using Tika From .Net
IKVM is pretty transparent. You simply reference the the Tika app assembly and your .Net code is talking to Java types. It is a bit weird at first as you have Java versions of types and .Net versions. Next you’ll want to make sure that all the dependent IKVM runtime assemblies are included with your project. Using Reflector I found that the Tika app assembly referenced a lot of IKVM assemblies which do not appear to be used. I had to figure out through trial and error which assemblies where not being touched by the rich document extractions being done. If need be you could simple include all of the referenced IKVM assemblies with your application. Below I have done the work for you and eliminated all references to all the IKVM assemblies which appear to be in play.
16 assemblies down to 5. A much smaller deployment.
Using Tika
To do some text extraction we’ll ask Tika, very nicely, to parse the files we throw at it. For my purposes this involved having Tika automatically determine how to parse the stream and extract the text and metadata about the document.
public TextExtractionResult Extract(string filePath) { var parser = new AutoDetectParser(); var metadata = new Metadata(); var parseContext = new ParseContext(); java.lang.Class parserClass = parser.GetType(); parseContext.set(parserClass, parser); try { var file = new File(filePath); var url = file.toURI().toURL(); using (var inputStream = MetadataHelper.getInputStream(url, metadata)) { parser.parse(inputStream, getTransformerHandler(), metadata, parseContext); inputStream.close(); } return assembleExtractionResult(_outputWriter.toString(), metadata); } catch (Exception ex) { throw new ApplicationException("Extraction of text from the file '{0}' failed.".ToFormat(filePath), ex); } } |
One Important Cavet
Java has a concept called a ClassLoader which has something to do with how Java types are found and loaded. There is probably a better way around this but for some reason if you do not implement a custom ClassLoader and also set an application setting cueing the IKVM runtime about which .Net type to use as the ClassLoader.
public class MySystemClassLoader : ClassLoader { public MySystemClassLoader(ClassLoader parent) : base(new AppDomainAssemblyClassLoader(typeof(MySystemClassLoader).Assembly)) { } } |
Here is an example app.config telling IKVM where the ClassLoader is found.
<?xml version="1.0" encoding="utf-8" ?> <configuration> <appSettings> <add key="ikvm:java.system.class.loader" value="TikaOnDotNet.MySystemClassLoader, TikaOnDotNet" /> </appSettings> </configuration> |
This step is very important. If IKVM cannot find a class loader, for some horrible reason, Tika will work fine but extract only empty documents with no metadata. The main reason this is troubling is that no exception is raised. For this reason we actually have a validation step in our application ensuring that the app setting is present and that it resolves to a valid type.
Demo
Here is a test demonstrating an extraction and the result.
[Test] public void should_extract_from_pdf() { var textExtractionResult = new TextExtractor().Extract("Tika.pdf"); textExtractionResult.Text.ShouldContain("pack of pickled almonds"); Console.WriteLine(textExtractionResult); } |
Put simply rich documents like this go in.
And a TextExtractionResult comes out:
public class TextExtractionResult { public string Text { get; set; } public string ContentType { get; set; } public IDictionary<string, string> Metadata { get; set; } //toString() override } |
Here is the raw output from Tika:
Conclusion
I hope this helps boost your confidence that you can use Java libraries in your .Net code and I hope my example repo will be of assistance if you need to do some work with Tika on the .Net platform. Enjoy.
常见问题：无法更改关系，因为一个或多个外键属性不可以为 null。对关系作出更改后，会将相关的外键属性设置为 null 值。如果外键不支持 null 值，则必须定义新的关系，必须向外键属性分配另一个非 null 值，或必须删除无关的对象。
解决方法：
例如OrderItem和Product是一对多的关系
OrderItem.ProductId是关系的外键
你要先删除对product的引用，比如表OrderItem里有一个到Product.Id的外键的话，需要将其设为空值对应。所以在删除之前要将OrderItem.ProductId设置为0或null
